From 42c06a1877c2bff723ba50b343b27f7f7f5ecefb Mon Sep 17 00:00:00 2001 From: djstrong Date: Wed, 17 Sep 2025 16:25:50 +0200 Subject: [PATCH 01/11] Enhance documentation for label sets and versions across various files, adding references to the glossary for clarity. Update environment variable descriptions in `.env.local.example` files and improve comments in scripts to ensure consistent understanding of label set concepts. --- apps/ensindexer/.env.local.example | 3 ++- apps/ensrainbow/.env.local.example | 16 ++-------------- .../scripts/download-ensrainbow-files.sh | 4 ++-- .../scripts/download-prebuilt-database.sh | 4 ++-- .../docs/ensrainbow/concepts/glossary.mdx | 4 ++-- .../concepts/label-sets-and-versioning.mdx | 2 +- .../concepts/typescript-interfaces.mdx | 4 ++-- .../docs/ensrainbow/contributing/index.mdx | 4 ++-- .../docs/ensrainbow/usage/configuration.mdx | 4 ++-- packages/ensnode-sdk/src/ensrainbow/types.ts | 5 +---- 10 files changed, 18 insertions(+), 32 deletions(-) diff --git a/apps/ensindexer/.env.local.example b/apps/ensindexer/.env.local.example index 73439e54b..cef0a6dbd 100644 --- a/apps/ensindexer/.env.local.example +++ b/apps/ensindexer/.env.local.example @@ -213,7 +213,7 @@ ENSRAINBOW_URL=http://localhost:3223 # https://ensnode.io/ensrainbow/usage/available-label-sets/ # # LABEL_SET_ID: The label set identifier that will be used for label healing requests sent to ENSRainbow. -# Each label set id references a collection of rainbow records. +# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". # This must match the label set ID configured in your ENSRainbow server. # # For full subgraph backwards compatibility, LABEL_SET_ID must be set to "subgraph" @@ -221,6 +221,7 @@ ENSRAINBOW_URL=http://localhost:3223 LABEL_SET_ID=subgraph # LABEL_SET_VERSION: A non-negative integer representing the version of the label set to request from ENSRainbow. +# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". # This "fully pins" ENSIndexer to a deterministic set of ENSRainbow label healing responses across time, # even if the connected ENSRainbow later ingests additional records into the same label set. # This must be less than or equal to the label set version configured in your ENSRainbow server. diff --git a/apps/ensrainbow/.env.local.example b/apps/ensrainbow/.env.local.example index dbc7cb067..4c9bf6386 100644 --- a/apps/ensrainbow/.env.local.example +++ b/apps/ensrainbow/.env.local.example @@ -27,13 +27,7 @@ DB_SCHEMA_VERSION=3 # --- Label Set ID --- # -# Definition: An identifier that categorizes the type of ENSRainbow data (e.g., a -# full production set vs. a minimal test set). -# -# Goal: To provide different "namespaces" or categories of ENSRainbow data. This allows -# ENSRainbow to serve different datasets for different use cases from the -# same Docker image, such as a tiny dataset for testing versus a -# comprehensive one for production. +# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". # # How to configure: Choose the Label Set ID that matches your needs. # - `subgraph`: The current production dataset. Use this for current production applications. @@ -44,13 +38,7 @@ LABEL_SET_ID=ens-test-env # --- Label Set Version --- # -# Definition: A non-negative integer representing an incremental update to a -# dataset within a specific `LABEL_SET_ID`. -# -# Goal: To support the deterministic evolution of datasets over time. As new -# ENS names are discovered, they are added in new, incremental versions. -# This allows services like ENSIndexer to achieve reproducible results -# by targeting a specific, immutable version of the data. +# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". # # How to configure: # - To get the most up-to-date data, use the highest available version number diff --git a/apps/ensrainbow/scripts/download-ensrainbow-files.sh b/apps/ensrainbow/scripts/download-ensrainbow-files.sh index 00d894399..ac84faa49 100755 --- a/apps/ensrainbow/scripts/download-ensrainbow-files.sh +++ b/apps/ensrainbow/scripts/download-ensrainbow-files.sh @@ -8,8 +8,8 @@ set -euo pipefail # This script downloads a specific ENSRainbow labelset file. # # The labelsets are identified by: -# - LABEL_SET_ID: The identifier for a label set, which is a collection of ENS labelhash-to-label mappings from a specific source. -# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set. +# - LABEL_SET_ID: The identifier for a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). +# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). # # This script requires these two identifiers as command-line arguments to # download the correct labelset file (.ensrainbow), its checksum, and a diff --git a/apps/ensrainbow/scripts/download-prebuilt-database.sh b/apps/ensrainbow/scripts/download-prebuilt-database.sh index e2f7a99ce..c2abdd948 100644 --- a/apps/ensrainbow/scripts/download-prebuilt-database.sh +++ b/apps/ensrainbow/scripts/download-prebuilt-database.sh @@ -13,8 +13,8 @@ set -euo pipefail # # The database is versioned using a three-part system: # - DB_SCHEMA_VERSION: The physical layout/structure of the database. -# - LABEL_SET_ID: The identifier for a label set, which is a collection of ENS labelhash-to-label mappings from a specific source. -# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set. +# - LABEL_SET_ID: The identifier for a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). +# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). # # This script requires these three identifiers as command-line arguments to # download the correct pre-built database archive (.tgz), its checksum, and a diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index 46b487837..888222fe1 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -15,9 +15,9 @@ This page defines the **core terminology** used throughout the ENSRainbow codeba | **Labelhash** | `keccak256` hash of the UTF-8 bytes of a label (no pre-normalisation), represented as a **0x-prefixed 64-digit lowercase hex** string (32 bytes). | `0xaf2caa…03cc` | | **Heal** | The act of converting a _labelhash_ back to its original _label_ via a rainbow table lookup. | `heal('0xaf2c…') → 'vitalik'` | | **Rainbow Record** | An entry mapping a `labelhash` ➜ `label`. Persisted as a LevelDB key (labelhash bytes) and UTF-8 value (_see Data Model_). | – | -| **Label Set** | A logical collection of rainbow records that share a common **source** and **versioning** scheme (e.g. subgraph v0). Identified by `labelSetId` & `labelSetVersion`. | id: `subgraph`, version: `0` | +| **Label Set** | A logical collection of rainbow records (ENS labelhash-to-label mappings) that share a common **source** and **versioning** scheme. Each label set represents a dataset snapshot that enables deterministic healing across time. Label sets are identified by a `labelSetId` and `labelSetVersion`. | id: `subgraph`, version: `0` | | **Label Set ID** | String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. | `subgraph`, `discovery-a` | -| **Label Set Version** | Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Enables deterministic healing across time. | `0`, `1`, `2` | +| **Label Set Version** | Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Version `0` is always the initial dataset. Enables deterministic healing across time by allowing clients to pin to specific versions for reproducible results. | `0`, `1`, `2` | | **Healable Count** | Total number of labels that can currently be healed by the running server. Exposed via `/count`. | `7 892 001` | | **Status Code** | High-level outcome of an API call – either `success` or `error`. | – | | **Error Code** | HTTP-style numeric code describing the error (`400`, `404`, `500`). | – | diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx index cc71e1517..4135009b6 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx @@ -9,7 +9,7 @@ keywords: [ensrainbow, versioning, label sets, deterministic] ## Why Label Sets & Versions? -A **label set** is analogous to a _dataset snapshot_. Every time the upstream data (e.g. an on-chain subgraph export) changes, we mint a new **label set version** so that: +A **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) is analogous to a _dataset snapshot_. Every time the upstream data (e.g. an on-chain subgraph export) changes, we mint a new **label set version** so that: ### 1. Deterministic Results Clients that pin _version `N`_ are guaranteed to get the _exact same_ heal response today, tomorrow, and two years from now. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx index adc32470a..333ebcd0f 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx @@ -7,7 +7,7 @@ sidebar: keywords: [ensrainbow, typescript, interfaces, types] --- -ENSRainbow's TypeScript APIs expose two companion interfaces that describe **which label sets are available (server-side) or requested (client-side)**. +ENSRainbow's TypeScript APIs expose two companion interfaces that describe **which label sets** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) **are available (server-side) or requested (client-side)**. ## Server Label Set @@ -24,7 +24,7 @@ interface EnsRainbowServerLabelSet { #### Fields -- **`labelSetId`** identifies **which label set** the server is currently serving. +- **`labelSetId`** identifies **which label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) the server is currently serving. - **`highestLabelSetVersion`** is the **highest version** available through the server for the label set id. The server will not return labels from a version _greater_ than this value (unless it ingests another incremental label set). ## Client Label Set diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx index 401a0f986..216fd9ad2 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx @@ -185,11 +185,11 @@ When using ENSRainbow with Docker, the following environment variables control w - **Goal**: Ensures compatibility between the ENSRainbow software and the structure of downloaded database files that are prebuilt for startup-time optimizations. - **Configuration**: It is strongly recommended to use the latest available schema version unless you have specific compatibility requirements. -- **`LABEL_SET_ID`**: The identifier for a **Label Set**, which is a collection of ENS labelhash-to-label mappings from a specific source. +- **`LABEL_SET_ID`**: The identifier for a **Label Set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). - **Goal**: To enable the extensible definition of new label sets (e.g., subgraph vs. production vs. test). - **Configuration**: See the [Available Label Sets](/ensrainbow/usage/available-label-sets) page for a complete list of currently available label set IDs and their descriptions. -- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **Label Set**. +- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **Label Set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). - **Goal**: To support the deterministic evolution of datasets over time, allowing services to achieve reproducible results. - **Configuration**: Use the highest available version number for the most up-to-date data. Versions are sequential and incremental: - `0` - The initial/base version of the **Label Set**. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx index daed73df7..1e773b9e0 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx @@ -33,12 +33,12 @@ These variables are **only used by shell scripts** for downloading and setting u - **Configuration**: It is strongly recommended to use the latest available schema version unless you have specific compatibility requirements. - **Used by**: Download scripts to fetch the correct database format. -- **`LABEL_SET_ID`**: The identifier for a **label set**, which is a collection of ENS labelhash-to-label mappings from a specific source. +- **`LABEL_SET_ID`**: The identifier for a **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). - **Goal**: To enable the extensible definition of new label sets (e.g., subgraph vs. production vs. test). - **Configuration**: See the [Available Label Sets](/ensrainbow/usage/available-label-sets) page for a complete list of currently available label set IDs and their descriptions. - **Used by**: Download scripts to fetch the correct label set. -- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **label set**. +- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). - **Goal**: To support the deterministic evolution of datasets over time, allowing services to achieve reproducible results. - **Configuration**: Use the highest available version number for the most up-to-date data. Versions are sequential and incremental: - `0` - The initial/base version of the **label set**. diff --git a/packages/ensnode-sdk/src/ensrainbow/types.ts b/packages/ensnode-sdk/src/ensrainbow/types.ts index e2a93019b..d59c1deb9 100644 --- a/packages/ensnode-sdk/src/ensrainbow/types.ts +++ b/packages/ensnode-sdk/src/ensrainbow/types.ts @@ -1,8 +1,5 @@ /** - * A label set ID identifies a set of labels that can be used for deterministic healing. - * A label set allows clients to deterministically heal their state against a server, - * ensuring that both are operating on the same version of data. - * + * A label set ID identifies a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). * It is guaranteed to be 1 to 50 characters long and contain only lowercase letters (a-z) * and hyphens (-). */ From 73376a7398b61eab941137798f51ec09793d9596 Mon Sep 17 00:00:00 2001 From: djstrong Date: Mon, 22 Sep 2025 16:11:38 +0200 Subject: [PATCH 02/11] Update environment variable documentation and improve comments in scripts to enhance clarity on label sets and versions. Add glossary references for better understanding across multiple files. --- apps/ensindexer/.env.local.example | 18 +--- apps/ensrainbow/.env.local.example | 32 +------ .../scripts/download-ensrainbow-files.sh | 6 +- .../scripts/download-prebuilt-database.sh | 8 +- .../docs/ensrainbow/concepts/data-model.mdx | 2 +- .../docs/ensrainbow/concepts/glossary.mdx | 89 +++++++++++++++---- .../concepts/label-sets-and-versioning.mdx | 4 +- .../concepts/typescript-interfaces.mdx | 4 +- .../docs/ensrainbow/contributing/index.mdx | 4 +- .../docs/ensrainbow/usage/configuration.mdx | 4 +- packages/ensnode-sdk/src/ensrainbow/types.ts | 2 +- 11 files changed, 99 insertions(+), 74 deletions(-) diff --git a/apps/ensindexer/.env.local.example b/apps/ensindexer/.env.local.example index cef0a6dbd..0f56f8a95 100644 --- a/apps/ensindexer/.env.local.example +++ b/apps/ensindexer/.env.local.example @@ -206,28 +206,18 @@ PLUGINS=subgraph,basenames,lineanames,threedns,protocol-acceleration,registrars, ENSRAINBOW_URL=http://localhost:3223 # Pinned Label Set Configuration for requests to ENSRainbow +# ENSRainbow label set configuration (see https://ensnode.io/ensrainbow/usage/configuration for details) # Required. ENSIndexer must be pinned to a specific label set ID and version to ensure deterministic # indexing results across time. # # For a list of available label sets and their configurations, visit: # https://ensnode.io/ensrainbow/usage/available-label-sets/ # -# LABEL_SET_ID: The label set identifier that will be used for label healing requests sent to ENSRainbow. -# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". -# This must match the label set ID configured in your ENSRainbow server. -# -# For full subgraph backwards compatibility, LABEL_SET_ID must be set to "subgraph" -# and LABEL_SET_VERSION must be set to 0. +# LABEL_SET_ID: see https://ensnode.io/ensrainbow/concepts/glossary#label-set-id. LABEL_SET_ID=subgraph -# LABEL_SET_VERSION: A non-negative integer representing the version of the label set to request from ENSRainbow. -# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". -# This "fully pins" ENSIndexer to a deterministic set of ENSRainbow label healing responses across time, -# even if the connected ENSRainbow later ingests additional records into the same label set. -# This must be less than or equal to the label set version configured in your ENSRainbow server. -# -# For full subgraph backwards compatibility, LABEL_SET_ID must be set to "subgraph" -# and LABEL_SET_VERSION must be set to 0. +# LABEL_SET_VERSION: see https://ensnode.io/ensrainbow/concepts/ +glossary#label-set-version. LABEL_SET_VERSION=0 # The "primary" ENSIndexer service URL diff --git a/apps/ensrainbow/.env.local.example b/apps/ensrainbow/.env.local.example index 4c9bf6386..7eb88231b 100644 --- a/apps/ensrainbow/.env.local.example +++ b/apps/ensrainbow/.env.local.example @@ -12,37 +12,11 @@ LOG_LEVEL=info # container will download on its first startup. This approach keeps the Docker # image small and allows data to be updated independently of the application code. -# --- Database Schema Version --- -# -# Definition: Specifies the version of the physical database layout. It is not -# related to the application's API version. -# -# Goal: Ensures the application's code is compatible with the on-disk database -# format. Think of it like a migration version for a traditional database. -# -# How to configure: You should almost always use the latest available version -# (e.g., `3`) to run the most recent and efficient database structure. Only use -# an older version if you have a specific, legacy system requirement. +# Database schema version (see https://ensnode.io/ensrainbow/usage/configuration for details) DB_SCHEMA_VERSION=3 -# --- Label Set ID --- -# -# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". -# -# How to configure: Choose the Label Set ID that matches your needs. -# - `subgraph`: The current production dataset. Use this for current production applications. -# - `ens-test-env`: A very small, lightweight dataset used for testing with the ens-test-env. -# - `searchlight`: (Future) An extended dataset with additional label discoveries. -# - `discovery-a`: (Future) A dataset for dynamically discovered labels. +# Label set ID (see https://ensnode.io/ensrainbow/concepts/glossary#label-set-id) LABEL_SET_ID=ens-test-env -# --- Label Set Version --- -# -# See https://ensnode.io/ensrainbow/concepts/glossary/ for definition of "label set". -# -# How to configure: -# - To get the most up-to-date data, use the highest available version number -# for your chosen `LABEL_SET_ID`. Versions start at `0`. -# - If you need to reproduce a result from an earlier point in time, you would -# use a specific, older version number. +# Label set version (see https://ensnode.io/ensrainbow/concepts/glossary#label-set-version) LABEL_SET_VERSION=0 diff --git a/apps/ensrainbow/scripts/download-ensrainbow-files.sh b/apps/ensrainbow/scripts/download-ensrainbow-files.sh index ac84faa49..bf0c2245f 100755 --- a/apps/ensrainbow/scripts/download-ensrainbow-files.sh +++ b/apps/ensrainbow/scripts/download-ensrainbow-files.sh @@ -8,8 +8,10 @@ set -euo pipefail # This script downloads a specific ENSRainbow labelset file. # # The labelsets are identified by: -# - LABEL_SET_ID: The identifier for a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). -# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). +# - LABEL_SET_ID +# - LABEL_SET_VERSION +# +# See https://ensnode.io/ensrainbow/concepts/glossary/ for details. # # This script requires these two identifiers as command-line arguments to # download the correct labelset file (.ensrainbow), its checksum, and a diff --git a/apps/ensrainbow/scripts/download-prebuilt-database.sh b/apps/ensrainbow/scripts/download-prebuilt-database.sh index c2abdd948..af49d08e2 100644 --- a/apps/ensrainbow/scripts/download-prebuilt-database.sh +++ b/apps/ensrainbow/scripts/download-prebuilt-database.sh @@ -12,9 +12,11 @@ set -euo pipefail # saving the end-user from a slow and resource-intensive data ingestion process. # # The database is versioned using a three-part system: -# - DB_SCHEMA_VERSION: The physical layout/structure of the database. -# - LABEL_SET_ID: The identifier for a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). -# - LABEL_SET_VERSION: A non-negative integer representing the version of a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). +# - DB_SCHEMA_VERSION +# - LABEL_SET_ID +# - LABEL_SET_VERSION +# +# See https://ensnode.io/ensrainbow/concepts/glossary/ for details. # # This script requires these three identifiers as command-line arguments to # download the correct pre-built database archive (.tgz), its checksum, and a diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx index e1df686d0..51eb7aabb 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx @@ -136,6 +136,6 @@ graph LR; ## Related Documentation -- **[Glossary](/ensrainbow/concepts/glossary)** - Key terms like System Key, Ingestion, etc. +- **[Glossary](/ensrainbow/concepts/glossary)** - Key terms like [System Key](/ensrainbow/concepts/glossary#system-key), [Ingestion](/ensrainbow/concepts/glossary#ingestion), etc. - **[Label Sets & Versioning](/ensrainbow/concepts/label-sets-and-versioning)** - Understanding the versioning system - **[TypeScript Interfaces](/ensrainbow/concepts/typescript-interfaces)** - Type definitions for working with the data diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index 888222fe1..a18c46570 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -9,22 +9,79 @@ keywords: [ensrainbow, glossary, terminology, definitions] This page defines the **core terminology** used throughout the ENSRainbow codebase and documentation. If you notice an unfamiliar word elsewhere in the docs, come back to this page – it is probably defined here. -| Term | Definition | Example | -|------|------------|---------| -| **Label** | A single component of an ENS name (characters between two dots). Can contain **any** valid UTF-8 string – it may or may not be ENS-normalised. | `vitalik`, `😺`, `example.eth` has labels `example` & `eth` | -| **Labelhash** | `keccak256` hash of the UTF-8 bytes of a label (no pre-normalisation), represented as a **0x-prefixed 64-digit lowercase hex** string (32 bytes). | `0xaf2caa…03cc` | -| **Heal** | The act of converting a _labelhash_ back to its original _label_ via a rainbow table lookup. | `heal('0xaf2c…') → 'vitalik'` | -| **Rainbow Record** | An entry mapping a `labelhash` ➜ `label`. Persisted as a LevelDB key (labelhash bytes) and UTF-8 value (_see Data Model_). | – | -| **Label Set** | A logical collection of rainbow records (ENS labelhash-to-label mappings) that share a common **source** and **versioning** scheme. Each label set represents a dataset snapshot that enables deterministic healing across time. Label sets are identified by a `labelSetId` and `labelSetVersion`. | id: `subgraph`, version: `0` | -| **Label Set ID** | String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. | `subgraph`, `discovery-a` | -| **Label Set Version** | Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Version `0` is always the initial dataset. Enables deterministic healing across time by allowing clients to pin to specific versions for reproducible results. | `0`, `1`, `2` | -| **Healable Count** | Total number of labels that can currently be healed by the running server. Exposed via `/count`. | `7 892 001` | -| **Status Code** | High-level outcome of an API call – either `success` or `error`. | – | -| **Error Code** | HTTP-style numeric code describing the error (`400`, `404`, `500`). | – | -| **Rainbow Table** | A pre-computed set of `labelhash → label` pairs used for healing. | – | -| **Ingestion** | One-off process that streams a `.ensrainbow` snapshot into LevelDB. | `pnpm run ingest subgraph_0.ensrainbow` | -| **System Key** | Special LevelDB key (length ≠ 32 bytes) storing metadata such as schema version, label set id, etc. | `0xff 0xff 0xff 0xfd` | -| **ENS Normalization** | The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even if not normalised. | – | +## Label + +A single component of an ENS name (characters between two dots). Can contain **any** valid UTF-8 string – it may or may not be ENS-normalised. + +**Example:** `vitalik`, `😺`, `example.eth` has labels `example` & `eth` + +## Labelhash + +`keccak256` hash of the UTF-8 bytes of a label (no pre-normalisation), represented as a **0x-prefixed 64-digit lowercase hex** string (32 bytes). + +**Example:** `0xaf2caa…03cc` + +## Heal + +The act of converting a _labelhash_ back to its original _label_ via a rainbow table lookup. + +**Example:** `heal('0xaf2c…') → 'vitalik'` + +## Rainbow Record + +An entry mapping a `labelhash` ➜ `label`. Persisted as a LevelDB key (labelhash bytes) and UTF-8 value (_see Data Model_). + +## Label Set + +A logical collection of [rainbow records](#rainbow-record) that share a common **source** and **versioning** scheme. Each label set represents a dataset snapshot that enables deterministic healing across time. Label sets are identified by a `labelSetId` and `labelSetVersion`. + +**Example:** id: `subgraph`, version: `0` + +## Label Set ID + +String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. + +**Example:** `subgraph`, `discovery-a` + +## Label Set Version + +Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Version `0` is always the initial dataset. Enables deterministic healing across time by allowing clients to pin to specific versions for reproducible results. + +**Example:** `0`, `1`, `2` + +## Healable Count + +Total number of labels that can currently be healed by the running server. Exposed via `/count`. + +**Example:** `7 892 001` + +## Status Code + +High-level outcome of an API call – either `success` or `error`. + +## Error Code + +HTTP-style numeric code describing the error (`400`, `404`, `500`). + +## Rainbow Table + +A pre-computed set of `labelhash → label` pairs used for healing. + +## Ingestion + +One-off process that streams a `.ensrainbow` snapshot into LevelDB. + +**Example:** `pnpm run ingest subgraph_0.ensrainbow` + +## System Key + +Special LevelDB key (length ≠ 32 bytes) storing metadata such as schema version, label set id, etc. + +**Example:** `0xff 0xff 0xff 0xfd` + +## ENS Normalization + +The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even if not normalised. ## Related Documentation diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx index 4135009b6..5c05f3a53 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/label-sets-and-versioning.mdx @@ -9,7 +9,7 @@ keywords: [ensrainbow, versioning, label sets, deterministic] ## Why Label Sets & Versions? -A **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) is analogous to a _dataset snapshot_. Every time the upstream data (e.g. an on-chain subgraph export) changes, we mint a new **label set version** so that: +A **[label set](/ensrainbow/concepts/glossary#label-set)** is analogous to a _dataset snapshot_. Every time the upstream data (e.g. an on-chain subgraph export) changes, we mint a new **[label set version](/ensrainbow/concepts/glossary#label-set-version)** so that: ### 1. Deterministic Results Clients that pin _version `N`_ are guaranteed to get the _exact same_ heal response today, tomorrow, and two years from now. @@ -51,6 +51,6 @@ This flexibility ensures applications can choose between **staying current** wit ## Related Documentation -- **[Glossary](/ensrainbow/concepts/glossary)** - Key terminology including version-related terms +- **[Glossary](/ensrainbow/concepts/glossary)** - Key terminology including [label set](/ensrainbow/concepts/glossary#label-set) and [label set version](/ensrainbow/concepts/glossary#label-set-version) terms - **[TypeScript Interfaces](/ensrainbow/concepts/typescript-interfaces)** - Type definitions for ENSRainbow's server and client - **[Data Model](/ensrainbow/concepts/data-model)** - How versions are stored in the database diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx index 333ebcd0f..ac7e4b95e 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/typescript-interfaces.mdx @@ -7,7 +7,7 @@ sidebar: keywords: [ensrainbow, typescript, interfaces, types] --- -ENSRainbow's TypeScript APIs expose two companion interfaces that describe **which label sets** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) **are available (server-side) or requested (client-side)**. +ENSRainbow's TypeScript APIs expose two companion interfaces that describe **which [label sets](/ensrainbow/concepts/glossary#label-set)** **are available (server-side) or requested (client-side)**. ## Server Label Set @@ -24,7 +24,7 @@ interface EnsRainbowServerLabelSet { #### Fields -- **`labelSetId`** identifies **which label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition) the server is currently serving. +- **`labelSetId`** identifies **which [label set](/ensrainbow/concepts/glossary#label-set)** the server is currently serving. - **`highestLabelSetVersion`** is the **highest version** available through the server for the label set id. The server will not return labels from a version _greater_ than this value (unless it ingests another incremental label set). ## Client Label Set diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx index 216fd9ad2..5151600cb 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/contributing/index.mdx @@ -185,11 +185,11 @@ When using ENSRainbow with Docker, the following environment variables control w - **Goal**: Ensures compatibility between the ENSRainbow software and the structure of downloaded database files that are prebuilt for startup-time optimizations. - **Configuration**: It is strongly recommended to use the latest available schema version unless you have specific compatibility requirements. -- **`LABEL_SET_ID`**: The identifier for a **Label Set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). +- **`LABEL_SET_ID`**: See **[Label Set ID](/ensrainbow/concepts/glossary#label-set-id)**. - **Goal**: To enable the extensible definition of new label sets (e.g., subgraph vs. production vs. test). - **Configuration**: See the [Available Label Sets](/ensrainbow/usage/available-label-sets) page for a complete list of currently available label set IDs and their descriptions. -- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **Label Set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). +- **`LABEL_SET_VERSION`**: See **[Label Set Version](/ensrainbow/concepts/glossary#label-set-version)**. - **Goal**: To support the deterministic evolution of datasets over time, allowing services to achieve reproducible results. - **Configuration**: Use the highest available version number for the most up-to-date data. Versions are sequential and incremental: - `0` - The initial/base version of the **Label Set**. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx index 1e773b9e0..a69743a00 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/usage/configuration.mdx @@ -33,12 +33,12 @@ These variables are **only used by shell scripts** for downloading and setting u - **Configuration**: It is strongly recommended to use the latest available schema version unless you have specific compatibility requirements. - **Used by**: Download scripts to fetch the correct database format. -- **`LABEL_SET_ID`**: The identifier for a **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). +- **`LABEL_SET_ID`**: See **[Label Set ID](/ensrainbow/concepts/glossary#label-set-id)**. - **Goal**: To enable the extensible definition of new label sets (e.g., subgraph vs. production vs. test). - **Configuration**: See the [Available Label Sets](/ensrainbow/usage/available-label-sets) page for a complete list of currently available label set IDs and their descriptions. - **Used by**: Download scripts to fetch the correct label set. -- **`LABEL_SET_VERSION`**: A non-negative integer representing the version of a **label set** (see [Glossary](/ensrainbow/concepts/glossary/) for definition). +- **`LABEL_SET_VERSION`**: See **[Label Set Version](/ensrainbow/concepts/glossary#label-set-version)**. - **Goal**: To support the deterministic evolution of datasets over time, allowing services to achieve reproducible results. - **Configuration**: Use the highest available version number for the most up-to-date data. Versions are sequential and incremental: - `0` - The initial/base version of the **label set**. diff --git a/packages/ensnode-sdk/src/ensrainbow/types.ts b/packages/ensnode-sdk/src/ensrainbow/types.ts index d59c1deb9..d7d8ca32b 100644 --- a/packages/ensnode-sdk/src/ensrainbow/types.ts +++ b/packages/ensnode-sdk/src/ensrainbow/types.ts @@ -1,5 +1,5 @@ /** - * A label set ID identifies a label set (see https://ensnode.io/ensrainbow/concepts/glossary/ for definition). + * A label set ID identifies a label set (see https://ensnode.io/ensrainbow/concepts/glossary#label-set for definition). * It is guaranteed to be 1 to 50 characters long and contain only lowercase letters (a-z) * and hyphens (-). */ From 62d87a54b40f884f2e57b0b206674fa81a85a56a Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 22:11:24 +0200 Subject: [PATCH 03/11] Update glossary with environment variable definitions and enhance descriptions in Terraform variables for clarity on label sets and versions. Add references to the glossary for improved understanding. --- .../docs/ensrainbow/concepts/glossary.mdx | 36 +++++++++++++++---- terraform/variables.tf | 8 ++--- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index a18c46570..f4fe120ed 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -9,41 +9,41 @@ keywords: [ensrainbow, glossary, terminology, definitions] This page defines the **core terminology** used throughout the ENSRainbow codebase and documentation. If you notice an unfamiliar word elsewhere in the docs, come back to this page – it is probably defined here. -## Label +## Label {#label} A single component of an ENS name (characters between two dots). Can contain **any** valid UTF-8 string – it may or may not be ENS-normalised. **Example:** `vitalik`, `😺`, `example.eth` has labels `example` & `eth` -## Labelhash +## Labelhash {#labelhash} `keccak256` hash of the UTF-8 bytes of a label (no pre-normalisation), represented as a **0x-prefixed 64-digit lowercase hex** string (32 bytes). **Example:** `0xaf2caa…03cc` -## Heal +## Heal {#heal} The act of converting a _labelhash_ back to its original _label_ via a rainbow table lookup. **Example:** `heal('0xaf2c…') → 'vitalik'` -## Rainbow Record +## Rainbow Record {#rainbow-record} An entry mapping a `labelhash` ➜ `label`. Persisted as a LevelDB key (labelhash bytes) and UTF-8 value (_see Data Model_). -## Label Set +## Label Set {#label-set} A logical collection of [rainbow records](#rainbow-record) that share a common **source** and **versioning** scheme. Each label set represents a dataset snapshot that enables deterministic healing across time. Label sets are identified by a `labelSetId` and `labelSetVersion`. **Example:** id: `subgraph`, version: `0` -## Label Set ID +## Label Set ID {#label-set-id} String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. **Example:** `subgraph`, `discovery-a` -## Label Set Version +## Label Set Version {#label-set-version} Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Version `0` is always the initial dataset. Enables deterministic healing across time by allowing clients to pin to specific versions for reproducible results. @@ -83,6 +83,28 @@ Special LevelDB key (length ≠ 32 bytes) storing metadata such as schema versio The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even if not normalised. +## Environment Variables + +### LABEL_SET_ID {#label-set-id-env} + +Environment variable that specifies the identifier for a [label set](#label-set). See [Label Set ID](#label-set-id) for the definition of this identifier. + +**Used by:** Download scripts and Docker entrypoint to fetch the correct label set + +### LABEL_SET_VERSION {#label-set-version-env} + +Environment variable that specifies the version of a [label set](#label-set). See [Label Set Version](#label-set-version) for the definition of this version number. + +**Used by:** Download scripts and Docker entrypoint to fetch the correct label set version + +### LOG_LEVEL {#log-level-env} + +Environment variable that controls the verbosity of logging output. + +**Format:** String +**Valid Values:** `fatal`, `error`, `warn`, `info`, `debug`, `trace`, `silent` +**Default:** `info` + ## Related Documentation - **[Label Sets & Versioning](/ensrainbow/concepts/label-sets-and-versioning)** - Understanding the versioning system diff --git a/terraform/variables.tf b/terraform/variables.tf index c3bb1bb6b..29fb78603 100644 --- a/terraform/variables.tf +++ b/terraform/variables.tf @@ -43,21 +43,21 @@ variable "quicknode_endpoint_name" { # The "fully pinned" label set reference that ENSIndexer will request ENSRainbow use for deterministic label healing across time. This label set reference is "fully pinned" as it requires both the labelSetId and labelSetVersion fields to be defined. variable "ensindexer_label_set_id" { type = string - description = "The label set ID that ENSIndexer will request from ENSRainbow for deterministic label healing (e.g., 'subgraph', 'ens-test-env')" + description = "The label set ID that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-id-env for definition." } variable "ensindexer_label_set_version" { type = string - description = "The label set version that ENSIndexer will request from ENSRainbow for deterministic label healing (e.g., '0', '1')" + description = "The label set version that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-version-env for definition." } # Label set that ENSRainbow will offer to its clients variable "ensrainbow_label_set_id" { type = string - description = "The label set ID that ENSRainbow will offer to its clients (e.g., 'subgraph', 'ens-test-env')" + description = "The label set ID that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-id-env for definition." } variable "ensrainbow_label_set_version" { type = string - description = "The highest label set version that ENSRainbow will offer to its clients (e.g., '0', '1')" + description = "The highest label set version that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-version-env for definition." } From e954fa188eebcbc2bdf6099560ce3da18852fd01 Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 22:21:59 +0200 Subject: [PATCH 04/11] Update terminology documentation to clarify the definition of LabelHash and add references to the ENSRainbow Glossary for comprehensive understanding. --- docs/ensnode.io/src/content/docs/docs/reference/terminology.mdx | 2 +- packages/ensnode-sdk/src/ens/types.ts | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/ensnode.io/src/content/docs/docs/reference/terminology.mdx b/docs/ensnode.io/src/content/docs/docs/reference/terminology.mdx index 72ad23859..62508cdf0 100644 --- a/docs/ensnode.io/src/content/docs/docs/reference/terminology.mdx +++ b/docs/ensnode.io/src/content/docs/docs/reference/terminology.mdx @@ -89,7 +89,7 @@ When rendering a **name**, one must take care to differentiate between an _unkno ### LabelHash, labelhash function -In this terminology reference, we say that the **LabelHash** of a **Label** is the 32-byte hashed result of calling the **`labelhash` function** with that **Label** as input. +In this terminology reference, we say that the **LabelHash** of a **Label** is the result of calling the **`labelhash` function** with that **Label** as input. For the complete technical definition, see the [ENSRainbow Glossary](/docs/ensrainbow/concepts/glossary#labelhash). That is, `0xaf2caa1c2ca1d027f1ac823b529d0a67cd144264b2789fa2ea4d63a67c7103cc` is the **LabelHash** of `vitalik`, which is the result of calling the **`labelhash` function** like so: diff --git a/packages/ensnode-sdk/src/ens/types.ts b/packages/ensnode-sdk/src/ens/types.ts index 6321f2bc5..1ea876da2 100644 --- a/packages/ensnode-sdk/src/ens/types.ts +++ b/packages/ensnode-sdk/src/ens/types.ts @@ -47,6 +47,7 @@ export type NormalizedName = Name & { __brand: "NormalizedName" }; * * @see https://docs.ens.domains/terminology#labelhash * @see https://ensnode.io/docs/reference/terminology#labels-labelhashes-labelhash-function + * @see https://docs.ensnode.io/docs/ensrainbow/concepts/glossary#labelhash */ export type LabelHash = Hex; From b58612b1b5d89b42c15612fbb9909106cc9bf5a0 Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 22:22:49 +0200 Subject: [PATCH 05/11] Enhance documentation by adding glossary references for the term "heal" in the ENSRainbow overview and API client comments, improving clarity and user understanding. --- docs/ensnode.io/src/content/docs/ensrainbow/index.mdx | 6 +++--- packages/ensrainbow-sdk/src/client.ts | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/index.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/index.mdx index 7b13badcf..a6cfb44f1 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/index.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/index.mdx @@ -1,6 +1,6 @@ --- title: What is ENSRainbow? -description: ENSRainbow is an open-source service that heals unknown ENS names by converting encoded labelhashes back to human-readable labels. +description: ENSRainbow is an open-source service that [heals](/docs/ensrainbow/concepts/glossary#heal) unknown ENS names by converting encoded labelhashes back to human-readable labels. sidebar: label: Overview order: 1 @@ -18,11 +18,11 @@ The ENS Registry allows subnames to be created onchain without revealing onchain ## How ENSRainbow Helps -ENSRainbow significantly improves "name healing" coverage compared to relying solely on services like the ENS Subgraph. Its goal is to heal as many ENS names as possible, minimizing the probability that end-users encounter unknown labels. +ENSRainbow significantly improves "name [healing](/docs/ensrainbow/concepts/glossary#heal)" coverage compared to relying solely on services like the ENS Subgraph. Its goal is to heal as many ENS names as possible, minimizing the probability that end-users encounter unknown labels. Key aspects of ENSRainbow include: -* **Resolving Encoded Labelhashes:** It translates cryptic labelhashes into human-readable labels. +* **Resolving Encoded Labelhashes:** It translates cryptic labelhashes into human-readable labels via [healing](/docs/ensrainbow/concepts/glossary#heal). * **Sidecar to ENSNode:** It integrates with ENSNode to provide deterministic name healing across time. * **Improved Healing Coverage:** Aims to minimize the probability that end-users encounter unknown labels, far exceeding previous capabilities. * **Extensible and Decentralized Data Management:** Uses a label set ID and label set version system for enabling any number of different label sets (collections of rainbow records) to support incremental updates across time. diff --git a/packages/ensrainbow-sdk/src/client.ts b/packages/ensrainbow-sdk/src/client.ts index 7d599b71c..9c6d3d9aa 100644 --- a/packages/ensrainbow-sdk/src/client.ts +++ b/packages/ensrainbow-sdk/src/client.ts @@ -242,7 +242,7 @@ export class EnsRainbowApiClient implements EnsRainbow.ApiClient { } /** - * Attempt to heal a labelHash to its original label. + * Attempt to [heal](/docs/ensrainbow/concepts/glossary#heal) a labelHash to its original label. * * Note on returned labels: ENSRainbow returns labels exactly as they are * represented in source rainbow table data. This means: From f540f70cc72af5c2245041913d9488a2b1b9ccf1 Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 22:33:29 +0200 Subject: [PATCH 06/11] Enhance documentation by adding glossary references for "rainbow record" in multiple files, improving clarity and user understanding of the data model and related concepts. --- apps/ensrainbow/src/lib/database.ts | 2 ++ apps/ensrainbow/src/lib/rainbow-record.ts | 2 +- apps/ensrainbow/src/utils/rainbow-record.ts | 3 +++ .../src/content/docs/ensrainbow/concepts/data-model.mdx | 3 +++ 4 files changed, 9 insertions(+), 1 deletion(-) diff --git a/apps/ensrainbow/src/lib/database.ts b/apps/ensrainbow/src/lib/database.ts index fcd154063..ec2195db6 100644 --- a/apps/ensrainbow/src/lib/database.ts +++ b/apps/ensrainbow/src/lib/database.ts @@ -84,6 +84,8 @@ export function isRainbowRecordKey(key: ByteArray): boolean { /** * Type representing the ENSRainbow LevelDB database. * + * For user-facing documentation, see the [Data Model documentation](/docs/ensrainbow/concepts/data-model). + * * Schema: * - Keys are binary encoded and represent: * - For rainbow records: The raw bytes of the ENS labelHash. Always a byte length of 32. diff --git a/apps/ensrainbow/src/lib/rainbow-record.ts b/apps/ensrainbow/src/lib/rainbow-record.ts index bbdaf1740..d68fdee06 100644 --- a/apps/ensrainbow/src/lib/rainbow-record.ts +++ b/apps/ensrainbow/src/lib/rainbow-record.ts @@ -3,7 +3,7 @@ import { buildLabelSetVersion, type Label, type LabelSetVersion } from "@ensnode import { getErrorMessage } from "@/utils/error-utils"; /** - * A versioned rainbow record. + * A versioned [rainbow record](/docs/ensrainbow/concepts/glossary#rainbow-record). */ export interface VersionedRainbowRecord { /** The original label string */ diff --git a/apps/ensrainbow/src/utils/rainbow-record.ts b/apps/ensrainbow/src/utils/rainbow-record.ts index 6f8d7e270..4031b73a6 100644 --- a/apps/ensrainbow/src/utils/rainbow-record.ts +++ b/apps/ensrainbow/src/utils/rainbow-record.ts @@ -3,6 +3,9 @@ import type { ByteArray } from "viem"; import type { LabelHash } from "@ensnode/ensnode-sdk"; import { labelHashToBytes } from "@ensnode/ensnode-sdk"; +/** + * A [rainbow record](/docs/ensrainbow/concepts/glossary#rainbow-record) interface. + */ export interface RainbowRecord { labelHash: ByteArray; label: string; diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx index 51eb7aabb..6250bdd96 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/data-model.mdx @@ -19,6 +19,8 @@ ENSRainbow stores its rainbow table in a **LevelDB** database. The schema is int ## Versioned Rainbow Record +A [rainbow record](/docs/ensrainbow/concepts/glossary#rainbow-record) with versioning information: + ```ts interface VersionedRainbowRecord { label: string; // original label @@ -58,6 +60,7 @@ Protocol Buffers is Google's language-neutral, platform-neutral extensible mecha The `.ensrainbow` file format uses two main message types: ```protobuf +// Protobuf serialization format for [rainbow records](/docs/ensrainbow/concepts/glossary#rainbow-record) message RainbowRecord { bytes labelhash = 1; // 32-byte labelhash string label = 2; // original label string From 6946cc1e53c5c52ec7073fb503972330e42e7b40 Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 23:26:31 +0200 Subject: [PATCH 07/11] Refine glossary documentation by correcting formatting for LABEL_SET_VERSION references and enhancing clarity in environment variable definitions. Update examples for consistency and improve overall readability. --- apps/ensindexer/.env.local.example | 3 +-- .../docs/ensrainbow/concepts/glossary.mdx | 22 ++++++++++--------- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/apps/ensindexer/.env.local.example b/apps/ensindexer/.env.local.example index 0f56f8a95..e6a4eb99f 100644 --- a/apps/ensindexer/.env.local.example +++ b/apps/ensindexer/.env.local.example @@ -216,8 +216,7 @@ ENSRAINBOW_URL=http://localhost:3223 # LABEL_SET_ID: see https://ensnode.io/ensrainbow/concepts/glossary#label-set-id. LABEL_SET_ID=subgraph -# LABEL_SET_VERSION: see https://ensnode.io/ensrainbow/concepts/ -glossary#label-set-version. +# LABEL_SET_VERSION: see https://ensnode.io/ensrainbow/concepts/glossary#label-set-version. LABEL_SET_VERSION=0 # The "primary" ENSIndexer service URL diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index f4fe120ed..d188c0cd1 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -9,41 +9,41 @@ keywords: [ensrainbow, glossary, terminology, definitions] This page defines the **core terminology** used throughout the ENSRainbow codebase and documentation. If you notice an unfamiliar word elsewhere in the docs, come back to this page – it is probably defined here. -## Label {#label} +## Label A single component of an ENS name (characters between two dots). Can contain **any** valid UTF-8 string – it may or may not be ENS-normalised. **Example:** `vitalik`, `😺`, `example.eth` has labels `example` & `eth` -## Labelhash {#labelhash} +## Labelhash `keccak256` hash of the UTF-8 bytes of a label (no pre-normalisation), represented as a **0x-prefixed 64-digit lowercase hex** string (32 bytes). **Example:** `0xaf2caa…03cc` -## Heal {#heal} +## Heal The act of converting a _labelhash_ back to its original _label_ via a rainbow table lookup. **Example:** `heal('0xaf2c…') → 'vitalik'` -## Rainbow Record {#rainbow-record} +## Rainbow Record An entry mapping a `labelhash` ➜ `label`. Persisted as a LevelDB key (labelhash bytes) and UTF-8 value (_see Data Model_). -## Label Set {#label-set} +## Label Set A logical collection of [rainbow records](#rainbow-record) that share a common **source** and **versioning** scheme. Each label set represents a dataset snapshot that enables deterministic healing across time. Label sets are identified by a `labelSetId` and `labelSetVersion`. **Example:** id: `subgraph`, version: `0` -## Label Set ID {#label-set-id} +## Label Set ID String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. **Example:** `subgraph`, `discovery-a` -## Label Set Version {#label-set-version} +## Label Set Version Non-negative integer that monotonically increases when new labelhash-to-label mappings are added to a label set. Each version contains incremental additions since the previous version. Version `0` is always the initial dataset. Enables deterministic healing across time by allowing clients to pin to specific versions for reproducible results. @@ -85,19 +85,21 @@ The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even ## Environment Variables -### LABEL_SET_ID {#label-set-id-env} + +### LABEL_SET_ID Environment variable that specifies the identifier for a [label set](#label-set). See [Label Set ID](#label-set-id) for the definition of this identifier. **Used by:** Download scripts and Docker entrypoint to fetch the correct label set -### LABEL_SET_VERSION {#label-set-version-env} + +### LABEL_SET_VERSION Environment variable that specifies the version of a [label set](#label-set). See [Label Set Version](#label-set-version) for the definition of this version number. **Used by:** Download scripts and Docker entrypoint to fetch the correct label set version -### LOG_LEVEL {#log-level-env} +### LOG_LEVEL Environment variable that controls the verbosity of logging output. From 25e03d9690ffcff0ea38ef870c14e5d8b14709c3 Mon Sep 17 00:00:00 2001 From: djstrong Date: Sat, 27 Sep 2025 23:37:37 +0200 Subject: [PATCH 08/11] Update glossary and Terraform variable descriptions to correct label set reference links, ensuring consistency and clarity in documentation. Enhance user understanding of environment variables related to label sets. --- .../src/content/docs/ensrainbow/concepts/glossary.mdx | 2 -- terraform/variables.tf | 8 ++++---- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index d188c0cd1..c5208b832 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -85,14 +85,12 @@ The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even ## Environment Variables - ### LABEL_SET_ID Environment variable that specifies the identifier for a [label set](#label-set). See [Label Set ID](#label-set-id) for the definition of this identifier. **Used by:** Download scripts and Docker entrypoint to fetch the correct label set - ### LABEL_SET_VERSION Environment variable that specifies the version of a [label set](#label-set). See [Label Set Version](#label-set-version) for the definition of this version number. diff --git a/terraform/variables.tf b/terraform/variables.tf index 29fb78603..bf0c19ad9 100644 --- a/terraform/variables.tf +++ b/terraform/variables.tf @@ -43,21 +43,21 @@ variable "quicknode_endpoint_name" { # The "fully pinned" label set reference that ENSIndexer will request ENSRainbow use for deterministic label healing across time. This label set reference is "fully pinned" as it requires both the labelSetId and labelSetVersion fields to be defined. variable "ensindexer_label_set_id" { type = string - description = "The label set ID that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-id-env for definition." + description = "The label set ID that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label_set_id for definition." } variable "ensindexer_label_set_version" { type = string - description = "The label set version that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-version-env for definition." + description = "The label set version that ENSIndexer will request from ENSRainbow for deterministic label healing. See https://ensnode.io/ensrainbow/concepts/glossary/#label_set_version for definition." } # Label set that ENSRainbow will offer to its clients variable "ensrainbow_label_set_id" { type = string - description = "The label set ID that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-id-env for definition." + description = "The label set ID that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label_set_id for definition." } variable "ensrainbow_label_set_version" { type = string - description = "The highest label set version that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label-set-version-env for definition." + description = "The highest label set version that ENSRainbow will offer to its clients. See https://ensnode.io/ensrainbow/concepts/glossary/#label_set_version for definition." } From ebb03ff736bf8ffe32c6593fc641a36e18d1ca0d Mon Sep 17 00:00:00 2001 From: "kwrobel.eth" Date: Sun, 28 Sep 2025 00:07:04 +0200 Subject: [PATCH 09/11] Fix labelhash glossary URL in types.ts Updated the URL for the labelhash glossary reference. --- packages/ensnode-sdk/src/ens/types.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/ensnode-sdk/src/ens/types.ts b/packages/ensnode-sdk/src/ens/types.ts index 1ea876da2..c8d244a6d 100644 --- a/packages/ensnode-sdk/src/ens/types.ts +++ b/packages/ensnode-sdk/src/ens/types.ts @@ -47,7 +47,7 @@ export type NormalizedName = Name & { __brand: "NormalizedName" }; * * @see https://docs.ens.domains/terminology#labelhash * @see https://ensnode.io/docs/reference/terminology#labels-labelhashes-labelhash-function - * @see https://docs.ensnode.io/docs/ensrainbow/concepts/glossary#labelhash + * @see https://ensnode.io/ensrainbow/concepts/glossary#labelhash */ export type LabelHash = Hex; From 57836edd1b893e7018f5f194125c7e78f91ac6e2 Mon Sep 17 00:00:00 2001 From: "kwrobel.eth" Date: Mon, 29 Sep 2025 17:54:11 +0200 Subject: [PATCH 10/11] Create fifty-spies-call.md --- .changeset/fifty-spies-call.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .changeset/fifty-spies-call.md diff --git a/.changeset/fifty-spies-call.md b/.changeset/fifty-spies-call.md new file mode 100644 index 000000000..757b6846f --- /dev/null +++ b/.changeset/fifty-spies-call.md @@ -0,0 +1,9 @@ +--- +"ensindexer": patch +"ensrainbow": patch +"@docs/ensnode": patch +"@ensnode/ensnode-sdk": patch +"@ensnode/ensrainbow-sdk": patch +--- + +Refine ENSRainbow Docs From 9d7b12b522fe96b48b07fd7999195ccabc34a0e5 Mon Sep 17 00:00:00 2001 From: djstrong Date: Wed, 17 Dec 2025 12:00:22 +0100 Subject: [PATCH 11/11] Enhance ENSRainbow documentation by adding detailed instructions for creating `.ensrainbow` files from both SQL and CSV sources. Update the description of the `searchlight` label set to reflect its availability and improved dataset features. Revise performance metrics for the `searchlight` dataset and include usage examples for downloading the extended discovery dataset. --- .../docs/ensrainbow/concepts/glossary.mdx | 4 ++- .../docs/ensrainbow/concepts/performance.mdx | 2 +- .../src/content/docs/ensrainbow/faq.mdx | 33 +++++++++++++++---- .../ensrainbow/usage/available-label-sets.mdx | 15 ++++++--- 4 files changed, 40 insertions(+), 14 deletions(-) diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx index c5208b832..f8fdebc85 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/glossary.mdx @@ -41,7 +41,7 @@ A logical collection of [rainbow records](#rainbow-record) that share a common * String (1-50 chars) consisting of lowercase ASCII letters and hyphens that names a label set. -**Example:** `subgraph`, `discovery-a` +**Example:** `subgraph`, `discovery-a`, `searchlight` ## Label Set Version @@ -85,6 +85,8 @@ The ENSIP-15 canonicalisation process; ENSRainbow stores labels **as-is**, even ## Environment Variables +These environment variables are typically set in Docker containers, shell scripts, or system configuration files. See the [Configuration](/ensrainbow/usage/configuration) guide for complete setup instructions. + ### LABEL_SET_ID Environment variable that specifies the identifier for a [label set](#label-set). See [Label Set ID](#label-set-id) for the definition of this identifier. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/performance.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/performance.mdx index c883fd44d..25444ba80 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/concepts/performance.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/concepts/performance.mdx @@ -10,6 +10,6 @@ keywords: [ensrainbow, performance] |---------|-------------------|--------------|--------------| | `ens-test-env / 0` | 1 MB | 5 MB | < 30 s | | `subgraph / 0` | 3 GB | 7 GB | ~20 min | -| `searchlight / latest` | 8 GB | 18 GB | ~50 min | +| `searchlight / 0` | 13 GB | 29 GB | ~30 min | `*` Times measured on a 4-core CPU & NVMe SSD. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/faq.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/faq.mdx index fa0d5704b..5be779975 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/faq.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/faq.mdx @@ -16,26 +16,45 @@ New label-set versions are generated manually for now: - **`subgraph`**: data from the ENS Subgraph rainbow tables, no plans to update - **`discovery-a`**: may be updated periodically as new labels are dynamically discovered - **`ens-test-env`**: Static test dataset, no plans to update -- **`searchlight`**: Enhanced discovery dataset (coming soon) +- **`searchlight`**: Enhanced discovery dataset with additional label discoveries beyond the subgraph To stay informed about new versions, monitor the [Available Label Sets](/ensrainbow/usage/available-label-sets/) documentation page. ## How can I create my own .ensrainbow file? -Currently, the `convert` command is the **only way** to create new `.ensrainbow` files from scratch. This command converts PostgreSQL rainbow table dumps (`.sql.gz` format) into the binary protobuf format that ENSRainbow uses. +ENSRainbow provides two methods for creating `.ensrainbow` files from different data sources: -**To create a custom .ensrainbow file:** +### Method 1: SQL Conversion +The `convert` command converts PostgreSQL rainbow table dumps (`.sql.gz` format) into the binary protobuf format that ENSRainbow uses. + +**To create a custom .ensrainbow file from SQL:** 1. **Prepare your data** as a PostgreSQL dump file (`.sql.gz`) with ENS labels and labelhashes 2. **Run the convert command:** ```bash - pnpm run convert --input-file your_data.sql.gz --output-file custom.ensrainbow + pnpm run convert \ + --input-file your_data.sql.gz \ + --output-file custom.ensrainbow \ + --label-set-id custom \ + --label-set-version 0 ``` -3. **Specify the label set details** using `--label-set-id` and `--label-set-version` flags -**Note:** You can download existing `.ensrainbow` files using the download scripts, but for creating entirely new files with your own data, the `convert` command is currently the only option available. +### Method 2: CSV Conversion +The `convert-csv` command converts CSV files (with 1 or 2 columns) into `.ensrainbow` format. This is ideal for custom datasets, test data, or external sources. + +**To create a custom .ensrainbow file from CSV:** + +1. **Prepare your data** as a CSV file with labels (and optionally labelhashes) +2. **Run the convert-csv command:** + ```bash + pnpm run convert-csv \ + --input-file your_labels.csv \ + --output-file custom.ensrainbow \ + --label-set-id custom \ + --label-set-version 0 + ``` -See the [CLI Reference](/ensrainbow/contributing/cli-reference/) for detailed command usage. +For complete instructions, examples, and workflow guidance, see the [Creating ENSRainbow Files](/ensrainbow/concepts/creating-files) guide. See the [CLI Reference](/ensrainbow/contributing/cli-reference/) for detailed command usage. ## Does ENSRainbow normalise labels? No. It returns labels exactly as stored. Your client should perform ENS Normalisation if required. diff --git a/docs/ensnode.io/src/content/docs/ensrainbow/usage/available-label-sets.mdx b/docs/ensnode.io/src/content/docs/ensrainbow/usage/available-label-sets.mdx index 78f808b75..4539b316d 100644 --- a/docs/ensnode.io/src/content/docs/ensrainbow/usage/available-label-sets.mdx +++ b/docs/ensnode.io/src/content/docs/ensrainbow/usage/available-label-sets.mdx @@ -20,6 +20,7 @@ Use these **currently available** identifiers with the [download script](/ensrai ./scripts/download-ensrainbow-files.sh subgraph 0 # Production dataset ./scripts/download-ensrainbow-files.sh ens-test-env 0 # Test dataset ./scripts/download-ensrainbow-files.sh discovery-a 0 # Discovery dataset +./scripts/download-ensrainbow-files.sh searchlight 0 # Extended discovery dataset ``` ## Available Label Sets @@ -54,15 +55,13 @@ The information below reflects actual availability as of the last check. |---------|--------|-------------| | `0` | ✅ Available | Initial empty dataset for dynamic discoveries | -### Planned/Coming Soon - -#### `searchlight` +#### `searchlight` **Source**: Extended discovery mechanisms -**Description**: Enhanced dataset with additional label discoveries beyond the subgraph. +**Description**: Enhanced dataset with additional label discoveries beyond the subgraph, providing maximum healing coverage. | Version | Status | Description | |---------|--------|-------------| -| `0` | 🚧 Coming Soon | Extended dataset with additional discoveries | +| `0` | ✅ Available | Extended dataset with additional discoveries | ## Usage Examples @@ -84,6 +83,12 @@ cd apps/ensrainbow ./scripts/download-ensrainbow-files.sh discovery-a 0 ``` +### For Maximum Coverage +```bash title="Download extended discovery dataset" +cd apps/ensrainbow +./scripts/download-ensrainbow-files.sh searchlight 0 +``` + ### For API Configuration ```bash title="Environment variables" export LABEL_SET_ID=subgraph