Hash modulo meta

[roberth]

Motivation

meta is lost at the derivation level, but it doesn't need to be that way, thanks to quotient hashing aka hash modulo.

This change makes use of the existing "discrepancy" between derivation hashes and output hashes to allow more information to be stored in derivations without causing rebuilds.

The core of the change is fairly simple; see the hashDerivationModulo hunk.

Changes

• Add system feature derivation-meta improving hash quotient logic to support storing meta
• Adjust JSON formats to make meta a first class top level attr instead of the ATerm embedding

Notes

• Must be opted in to by derivations/expressions, as usual, like ca derivations, structuredAttrs
• Not supported in non-structuredAttrs. Didn't seem worth the added complexity. Users should move to structuredAttrs instead and afaik Nixpkgs is on track to do so.

Example

$ nix run github:roberth/nix/hash-modulo-meta -- \
  build --store ~/stores/tmp \
    --extra-experimental-features derivation-meta \
    --impure --expr \
    'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
...
<progress bar, substituting from cache.nixos.org>
...

$ nix run github:roberth/nix/hash-modulo-meta -- derivation show --store ~/stores/tmp --extra-experimental-features derivation-meta --impure --expr 'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
{
  "vcllzyc6d87wxlncjcwxlgy82lna4h4v-python3-3.13.9.drv": {
    "args": ...
    ...
    "meta": {
      "available": true,
      "broken": false,
      "changelog": "https://docs.python.org/release/3.13.9/whatsnew/changelog.html",
...
  }
}

Context

• Step towards correctly implementing meta.timeout
• Implements, closes Have our cake and eat it too derivation metadata #10780
• Incremental change on topic of New derivation-making builtin #9774
• Closes perhaps Provide SBOM information in an "queryable" way #13564

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[edolstra]

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

It's not entirely obvious how derivation meta data would be used in practice, especially since derivers are kind of useless (e.g. cache.nixos.org paths do have a deriver, but you have no way to get to them, so you wouldn't be able to get to the metadata). Perhaps in conjunction with #11749, the metadata could be propagated into the provenance records of store paths (which are already JSON).

[roberth]

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

Correct, and this is now also covered by a functional test.

not entirely obvious how derivation meta data would be used in practice

You can instantiate your stuff and inspect the metadata.
If you don't have the expression sources for what you've deployed, you probably should(!), but if you don't provenance could connect that back, either by eval reproduction, by reference or by value.

• eval reproduction: provenance has a pinned flakeref, and you can instantiate that to acquire derivations
• by reference: maybe you actually do have derivations in a store
• by value: meta could be duplicated into the provenance structures, and it will benefit from the infrastructure that's set up around this:
  • having a reliable interface between Nix and the expression world (ie derivation { __meta })
    (meta attribute is not reliable because not the whole derivation closure occurs as packages that are seen by the CLI)
  • that interface getting used, pkgs: Add config.derivationMeta nixpkgs#466932

So this is a good step towards making #11749 more effective.

• Store path provenance tracking #11749 -> improvements for when expression sources are not immediately known
• Hash modulo meta #14686 -> access the metadata for whole closures