Skip to content

Conversation

@Popax21
Copy link

@Popax21 Popax21 commented Dec 6, 2025

Motivation

Adds a minimum age gate option to the nix-collect-garbage / nix store gc commands, which enables users to restrict the GC to deleting paths whose most recent usage was more than a certain period of time in the past.
"Usage" is intentionally defined ambiguously, however in general all operations which produce/require the
presence of a given store path count as "usage"; this is tracked within a new local store DB column.

Fixes: #2793, #7572

Context

Most of the complexity related to this feature stems from properly tracking the "last usage time" of any given store object - implementing the actual age gate itself is rather trivial (which means that if this PR were to be rejected, a more simple MVP implementing just #2793 could be considered instead).
After some consideration, I have decided to implement this in the form of a new database field which is added to existing DBs using the scheme migration mechanism. This requires "bumping" the DB field whenever the store path is used, however other than checking the store path's atime (which does not produce meaningful results on most modern Linux installations) I was unable to come up with any suitable alternative designs.

This DB field is bumped whenever store path is produced / required by some piece of code, which most notably includes (cached) derivation builds - this means that e.g. executing nix run nixpkgs#hello will bump the usage time every time, even though no new derivations were built / substituted. Bumping is implemented using a new bumpLastUsageTime store API method, which is invoked from various places scattered all around the code base. This sadly makes this change rather invasive vertically, as in it affecting a lot of files, however every individual made change on its own is not too invasive.

The new lastUsageTime field is also exposed as part of the publicly documented store path info, and synced across the worker protocol using a new fine-grained protocol feature (as added in commit 3be7c00). This requires some more general plumbing to pass down the set of negotiated features to the serialization code, but allows the field to be accessed from non-root users using the Nix daemon.

For easier reviewing, this PR is split into three commits:

  • one commit adds the lastUsageTime field / associated bumpLastUsageTime time method, as well as the corresponding local store DB logic. It also adds various scattered bumpLastUsageTime calls all across the codebase. This commit does not yet implement synchronization of the field using the worker protocol though.
  • one commit adds the path-last-usage-time worker protocol feature, which when negotiated enables serialization of the lastUsageTime field across worker protocol connections. This commit has been split off from the first commit since it necessitated modifying some baseline plumbing code to expose the set of negotiated features to the serialization code, and the concrete implementation featured here may be suboptimal / swapped out for a different method by the core Nix team if desired.
  • one last commit adds the actual GC age gate feature + associated CLI options using the newly added lastUsageTime field as a base. This commit itself is rather minimal, and as such could be easily cherry-picked / adapted to use the path registration time instead to at least fix idea: nix-collect-garbage option to also respect creation date of a derivation #2793 if this PR were to be rejected.

Some final points which should be considered before this is merged:

  • Should the bumpLastUsageTime method be forwarded using a new worker protocol operation? If yes this might facilitate the creation of LRU-style remote Nix caches, however for the sake of simplicity this has been cut for now.
  • New tests should be added for all new behaviors. I tried to find existing tests for the --older-than age gate for profile GCs as a base for such tests, however I was unable to find any. Additionally, there is currently no common test support code for worker protocol serialization tests which depends on negotiated protocol features (from what I can tell). All in all, given the scope of the PR I have decided to first focus on finalizing the exact implementation of this feature in collaboration of the Nix team before writing tests which might become obsolete over the lifespan of the PR.

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added documentation new-cli Relating to the "nix" command store Issues and pull requests concerning the Nix store labels Dec 6, 2025
This attribute tracks the last time a given store path object was "used" in a new local store DB field.
The information collected this way may be used by new features like e.g. a "minimum age" GC gate.

"Usage" is intentionally defined ambiguously, however in general all operations which produce/require the presence of a given store path count as "usage".
In this vein the commit adds a bunch of `bumpLastUsageTime` calls all over the codebase in appropriate spots.

This commit does not yet make any changes to any network protocols like e.g. the worker protocol; this is done in a followup commit.
Serialize the lastUsageTime path info attribute as part of the worker protocol, making it accessible to e.g. non-root users using the nix daemon.

This is accomplished using a new fine-grained worker protocol feature, along with some additional plumbing to make the set of negotiated features accessible to the serializer code.
Use the recently added `lastUsageTime` store path attribute to allow users to constrain the GC to only deleting paths which have been last used n days ago.
This is useful for ensuring that e.g. periodic GCs aren't too aggressive.
@Popax21
Copy link
Author

Popax21 commented Dec 6, 2025

One note about the current implementation: running nix path-info /nix/store/<path> will currently bump the last usage timestamp, while e.g. nix path-info nixpkgs#hello will not. This seems to be because the former goes through the full installable building logic (which bumps the usage timestamp), while the latter does not (fully) do so (which can be best observed by running said command while nixpkgs#hello isn't already in the store, which will fail and not trigger a build; compare this with passing in an invalid store path, which will yield a "don't know how to build X" error). I'm unsure how to best address this since passing an already built store path to other operations taking an installable (like e.g. nix shell) should definitiviely still bump the usage timestamp.

Copy link
Contributor

@xokdvium xokdvium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the performance overhead look like from the evaluator with a remote store? Doesn't this introduce potentially much more daemon ops and "db churn"?

EDIT: Ah, I see. The bumping is only implemented for the local store case. This seems fine to me.

@Popax21
Copy link
Author

Popax21 commented Dec 6, 2025

The bumping is only implemented for the local store case.

Exactly; I've mentioned this as a point still requiring discussion in the PR description. Additionally, since most if not all bumpLastUsageTime calls immediately follow a isValidPath query in a "check for valid path, if valid bump then bail" pattern it should be possible to remove any protocol op churn concerns by introducing a new combined isValidPathAndBumpIfYes worker protocol op (and in fact this might on second thought even be required to ensure the last usage time is bumped properly in some local nix-daemon scenarios where a non-root user runs nix commands).

@xokdvium
Copy link
Contributor

xokdvium commented Dec 6, 2025

Makes sense. Will do a more in-depth review a bit later. Thanks! The idea sounds good in principle if db churn isn't a concern. Even just implementing this for the local store should already provide a concrete improvement, since usage would be bumped if a store object is used during a build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation new-cli Relating to the "nix" command store Issues and pull requests concerning the Nix store

Projects

None yet

Development

Successfully merging this pull request may close these issues.

idea: nix-collect-garbage option to also respect creation date of a derivation

2 participants