Hash modulo meta

doc/manual/rl-next/derivation-meta.md

@@ -0,0 +1,43 @@
+---
+synopsis: "Derivation metadata without affecting output paths"
+prs: []
+issues: []
+---
+
+Packages in the expression language often have valuable metadata that is intentionally omitted from the derivations so as not to cause unnecessary rebuilds.
+Unfortunately, this omission made it very hard to access this information.
+
+The new [`derivation-meta`](@docroot@/development/experimental-features.md#xp-feature-derivation-meta) feature solves this problem, and make it easy for package management systems and tooling to associate metadata with builds without disruption or loss of cache effectiveness.
+
+Derivations using [structured attributes](@docroot@/store/derivation/index.md#structured-attrs) can now include a [`__meta`](@docroot@/language/advanced-attributes.md#adv-attr-meta) attribute for metadata like package descriptions, licenses, and maintainer information, without affecting output paths or triggering rebuilds.
+
+To use this feature, derivations must:
+
+1. Set [`__structuredAttrs`](@docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs) = true
+2. Include `"derivation-meta"` in [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.md#adv-attr-requiredSystemFeatures)
+3. Enable the [`derivation-meta`](@docroot@/development/experimental-features.md#xp-feature-derivation-meta) experimental feature in [configuration](@docroot@/command-ref/conf-file.md#conf-experimental-features)
+
+Example:
+
+```nix
+derivation {
+  name = "example";
+  __structuredAttrs = true;
+  requiredSystemFeatures = [ "derivation-meta" ];
+  __meta = {
+    description = "Example package";
+    license = "MIT";
+  };
+  # ... other attributes ...
+}
+```
+
+The [`__meta`](@docroot@/language/advanced-attributes.md#adv-attr-meta) attribute and `derivation-meta` system feature are filtered from output path computation using [hash modulo](@docroot@/store/derivation/outputs/input-address.md#hash-quotient-drv). This means:
+
+- Changing metadata does not invalidate binary caches
+- Enabling or disabling the feature does not affect output paths
+- Derivation paths (`.drv` files) still change when metadata changes, preserving the full derivation record
+- As usual, changes to meta attributes translate to derivation changes when they are explicitly used for the generation of derivations; usually in the dependents of a package.
+
+Care has been taken so that derivations that do or do not use this feature resolve to the same output hashes.
+While this can be thwarted using the last point or by including a *`drvPath`* as opposed to a (normal) *output path*, we observe in practice that both configurations (`requiredSystemFeatures = ["derivation-meta"];` or without) are binary cache compatible, sharing the same build.

doc/manual/source/language/advanced-attributes.md

@@ -67,6 +67,61 @@ Derivations can declare some infrequently used optional attributes.
     [`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites), maxSize, and maxClosureSize.
     will have no effect.
 
+  - [`__meta`]{#adv-attr-meta}\
+    > **Warning**
+    >
+    > This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
+    >
+    > To use this attribute, you must enable the
+    > [`derivation-meta`][xp-feature-derivation-meta] experimental feature.
+    > For example, in [nix.conf](../command-ref/conf-file.md) you could add:
+    >
+    > ```
+    > extra-experimental-features = derivation-meta
+    > ```
+
+    When using [structured attributes](#adv-attr-structuredAttrs), the `__meta` attribute
+    allows you to attach arbitrary metadata to a derivation without affecting its output paths
+    or triggering rebuilds when the metadata changes.
+
+    This is useful for tooling that needs to associate information with derivations, such as:
+    - Package descriptions and maintainer information
+    - License and homepage metadata
+    - Documentation generation data
+
+    To opt in with your derivations (typically mediated by Nixpkgs rather than written directly):
+
+    1. Set [`__structuredAttrs`](#adv-attr-structuredAttrs) = true
+    2. Add `"derivation-meta"` to [`requiredSystemFeatures`](#adv-attr-requiredSystemFeatures)
+
+    Example:
+
+    ```nix
+    derivation {
+      name = "example";
+      # ... other attributes ...
+      __structuredAttrs = true;
+      requiredSystemFeatures = [ "derivation-meta" ];
+      __meta = {
+        description = "A useful tool";
+        version = "1.0.0";
+        maintainer = "alice@example.com";
+      };
+    }
+    ```
+
+    When these requirements are met, both `__meta` and the `derivation-meta` system feature
+    are filtered from output path computation (see [Hash Modulo](@docroot@/store/derivation/outputs/input-address.md#hash-quotient-drv)).
+    This means you can enable or disable the feature, or change metadata content, without affecting
+    output paths or invalidating binary caches.
+
+    > **Note**
+    >
+    > While output paths of the derivation itself remain stable when `__meta` changes, the
+    > derivation path (`.drv` file) will differ, as it contains the full derivation including metadata.
+    > Additionally, if expressions reference values from `__meta` (such as `pkg.meta.mainProgram`),
+    > those dependent derivations will, as usual, be affected if those *otherwise "meta"* values change.
+
 ## Output checks
 
 See the [corresponding section in the derivation output page](@docroot@/store/derivation/outputs/index.md).
@@ -370,5 +425,7 @@ Here is more information on the `output*` attributes, and what values they may b
 [fixed-output derivation]: @docroot@/glossary.md#gloss-fixed-output-derivation
 [file system object]: @docroot@/store/file-system-object.md
 [store object]: @docroot@/store/store-object.md
+[xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
+[xp-feature-derivation-meta]: @docroot@/development/experimental-features.md#xp-feature-derivation-meta
 [xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
 [xp-feature-git-hashing]: @docroot@/development/experimental-features.md#xp-feature-git-hashing

doc/manual/source/protocols/json/schema/derivation-options-v1.yaml

@@ -26,6 +26,8 @@ required:
   - requiredSystemFeatures
   - preferLocalBuild
   - allowSubstitutes
+  # Note: `meta` is not in the required list to maintain backward compatibility
+  # with existing JSON that predates the derivation-meta feature
 properties:
   outputChecks:
     type: object
@@ -139,6 +141,19 @@ properties:
     description: |
       Whether substituting from other stores should be allowed for this derivation's outputs.
 
+  meta:
+    oneOf:
+      - type: object
+      - type: "null"
+    title: Metadata
+    description: |
+      Metadata excluded from hash computation when using structured attributes with the `derivation-meta` system feature.
+
+      Must be a JSON object containing arbitrary metadata about the derivation (description, license, maintainers, etc.).
+      This field is only populated when both `__meta` is present in structured attributes and `derivation-meta`
+      is listed in `requiredSystemFeatures`. Otherwise, `__meta` remains as a regular attribute in structured attributes
+      and is included in the derivation hash.
+
 additionalProperties: false
 
 $defs:

doc/manual/source/store/derivation/outputs/input-address.md

@@ -16,6 +16,10 @@ Concretely, this would cause a "mass rebuild" whenever any fetching detail chang
 To solve this problem, we compute output hashes differently, so that certain output hashes become identical.
 We call this concept quotient hashing, in reference to quotient types or sets.
 
+Similarly, the [derivation-meta][xp-feature-derivation-meta] experimental feature uses quotient hashing to allow metadata changes without affecting output paths.
+When a derivation requires the `derivation-meta` [system feature](@docroot@/command-ref/conf-file.md#conf-system-features) and uses [structured attributes](@docroot@/store/derivation/index.md#structured-attrs), both the [`__meta`](@docroot@/language/advanced-attributes.md#adv-attr-meta) attribute and `derivation-meta` from [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.md#adv-attr-requiredSystemFeatures) are filtered from hash computation.
+This enables tooling to attach arbitrary metadata to derivations without invalidating binary caches or triggering unnecessary rebuilds.
+
 So how do we compute the hash part of the output paths of an input-addressed derivation?
 This is done by the function `hashQuotientDerivation`, shown below.
 
@@ -222,5 +226,6 @@ where \\(\\mathrm{caInputs}(d)\\) returns the content-addressed inputs of \\(d\\
 > It also relates derivations whose `inputDrvOutputs` have been rewritten into `inputSrcs`.
 
 [deriving-path]: @docroot@/store/derivation/index.md#deriving-path
-[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
 [xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
+[xp-feature-derivation-meta]: @docroot@/development/experimental-features.md#xp-feature-derivation-meta
+[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations

src/libstore-tests/data/derivation/ca/all_set.json

@@ -19,6 +19,7 @@
   "impureHostDeps": [
     "/usr/bin/ditto"
   ],
+  "meta": null,
   "noChroot": true,
   "outputChecks": {
     "forAllOutputs": {

src/libstore-tests/data/derivation/ca/structuredAttrs_all_set.json

@@ -19,6 +19,7 @@
   "impureHostDeps": [
     "/usr/bin/ditto"
   ],
+  "meta": null,
   "noChroot": true,
   "outputChecks": {
     "perOutput": {

src/libstore-tests/data/derivation/ia/all_set.json

@@ -16,6 +16,7 @@
   "impureHostDeps": [
     "/usr/bin/ditto"
   ],
+  "meta": null,
   "noChroot": true,
   "outputChecks": {
     "forAllOutputs": {

src/libstore-tests/data/derivation/ia/defaults.json

@@ -5,6 +5,7 @@
   "exportReferencesGraph": {},
   "impureEnvVars": [],
   "impureHostDeps": [],
+  "meta": null,
   "noChroot": false,
   "outputChecks": {
     "forAllOutputs": {

src/libstore-tests/data/derivation/ia/structuredAttrs_all_set.json

@@ -16,6 +16,7 @@
   "impureHostDeps": [
     "/usr/bin/ditto"
   ],
+  "meta": null,
   "noChroot": true,
   "outputChecks": {
     "perOutput": {

src/libstore-tests/data/derivation/ia/structuredAttrs_defaults.json

@@ -5,6 +5,7 @@
   "exportReferencesGraph": {},
   "impureEnvVars": [],
   "impureHostDeps": [],
+  "meta": null,
   "noChroot": false,
   "outputChecks": {
     "perOutput": {}

src/libstore-tests/data/derivation/meta-derivation.drv

@@ -0,0 +1 @@
+Derive([("out","/nix/store/c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-meta-derivation","","")],[],["/nix/store/c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-dep1"],"x86_64-linux","/bin/sh",["-c","echo hello > $out"],[("__json","{\"__meta\":{\"description\":\"A test derivation\",\"maintainer\":\"test@example.com\",\"version\":\"1.0\"},\"requiredSystemFeatures\":[\"derivation-meta\"]}"),("out","/nix/store/c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-meta-derivation")])

src/libstore-tests/data/derivation/meta-derivation.json

@@ -0,0 +1,34 @@
+{
+  "args": [
+    "-c",
+    "echo hello > $out"
+  ],
+  "builder": "/bin/sh",
+  "env": {
+    "out": "/nix/store/c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-meta-derivation"
+  },
+  "inputs": {
+    "drvs": {},
+    "srcs": [
+      "c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-dep1"
+    ]
+  },
+  "meta": {
+    "description": "A test derivation",
+    "maintainer": "test@example.com",
+    "version": "1.0"
+  },
+  "name": "meta-derivation",
+  "outputs": {
+    "out": {
+      "path": "c015dhfh5l0lp6wxyvdn7bmwhbbr6hr9-meta-derivation"
+    }
+  },
+  "structuredAttrs": {
+    "requiredSystemFeatures": [
+      "derivation-meta"
+    ]
+  },
+  "system": "x86_64-linux",
+  "version": 4
+}

src/libstore-tests/derivation-advanced-attrs.cc

@@ -180,6 +180,7 @@ static const DerivationOptions<SingleDerivedPath> advancedAttributes_defaults =
     .requiredSystemFeatures = {},
     .preferLocalBuild = false,
     .allowSubstitutes = true,
+    .meta = std::nullopt,
 };
 
 TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_defaults)
@@ -272,6 +273,7 @@ DerivationOptions<SingleDerivedPath> advancedAttributes_ia = {
     .requiredSystemFeatures = {"rainbow", "uid-range"},
     .preferLocalBuild = true,
     .allowSubstitutes = false,
+    .meta = std::nullopt,
 };
 
 TEST_F(DerivationAdvancedAttrsTest, advancedAttributes_ia)
@@ -302,6 +304,7 @@ DerivationOptions<SingleDerivedPath> advancedAttributes_ca = {
     .requiredSystemFeatures = {"rainbow", "uid-range"},
     .preferLocalBuild = true,
     .allowSubstitutes = false,
+    .meta = std::nullopt,
 };
 
 TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes)
@@ -322,6 +325,7 @@ DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_defaults
     .requiredSystemFeatures = {},
     .preferLocalBuild = false,
     .allowSubstitutes = true,
+    .meta = std::nullopt,
 };
 
 TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs_defaults)
@@ -444,6 +448,7 @@ DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_ia = {
     .requiredSystemFeatures = {"rainbow", "uid-range"},
     .preferLocalBuild = true,
     .allowSubstitutes = false,
+    .meta = std::nullopt,
 };
 
 TEST_F(DerivationAdvancedAttrsTest, advancedAttributes_structuredAttrs)
@@ -486,6 +491,7 @@ DerivationOptions<SingleDerivedPath> advancedAttributes_structuredAttrs_ca = {
     .requiredSystemFeatures = {"rainbow", "uid-range"},
     .preferLocalBuild = true,
     .allowSubstitutes = false,
+    .meta = std::nullopt,
 };
 
 TEST_F(CaDerivationAdvancedAttrsTest, advancedAttributes_structuredAttrs)
@@ -517,4 +523,17 @@ TEST_JSON_OPTIONS(CaDerivationAdvancedAttrsTest, structuredAttrs_all_set, struct
 
 #undef TEST_JSON_OPTIONS
 
+// Test backward compatibility: JSON without 'meta' field should be ingestible
+TEST_F(DerivationAdvancedAttrsTest, DerivationOptions_backward_compat_no_meta)
+{
+    // Read existing JSON and remove the 'meta' field to simulate old format
+    this->readTest("defaults.json", [&](auto encoded) {
+        auto j = json::parse(encoded);
+        j.erase("meta"); // Remove meta field to simulate old JSON
+        auto got = j.template get<DerivationOptions<SingleDerivedPath>>();
+        // Should successfully deserialize with meta = std::nullopt
+        EXPECT_EQ(got.meta, std::nullopt);
+    });
+}
+
 } // namespace nix