-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Add a minimum age gate option to the nix-collect-garbage / nix store gc commands
#14725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
This attribute tracks the last time a given store path object was "used" in a new local store DB field. The information collected this way may be used by new features like e.g. a "minimum age" GC gate. "Usage" is intentionally defined ambiguously, however in general all operations which produce/require the presence of a given store path count as "usage". In this vein the commit adds a bunch of `bumpLastUsageTime` calls all over the codebase in appropriate spots. This commit does not yet make any changes to any network protocols like e.g. the worker protocol; this is done in a followup commit.
Serialize the lastUsageTime path info attribute as part of the worker protocol, making it accessible to e.g. non-root users using the nix daemon. This is accomplished using a new fine-grained worker protocol feature, along with some additional plumbing to make the set of negotiated features accessible to the serializer code.
Use the recently added `lastUsageTime` store path attribute to allow users to constrain the GC to only deleting paths which have been last used n days ago. This is useful for ensuring that e.g. periodic GCs aren't too aggressive.
|
One note about the current implementation: running |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does the performance overhead look like from the evaluator with a remote store? Doesn't this introduce potentially much more daemon ops and "db churn"?
EDIT: Ah, I see. The bumping is only implemented for the local store case. This seems fine to me.
Exactly; I've mentioned this as a point still requiring discussion in the PR description. Additionally, since most if not all |
|
Makes sense. Will do a more in-depth review a bit later. Thanks! The idea sounds good in principle if db churn isn't a concern. Even just implementing this for the local store should already provide a concrete improvement, since usage would be bumped if a store object is used during a build. |
Motivation
Adds a minimum age gate option to the
nix-collect-garbage/nix store gccommands, which enables users to restrict the GC to deleting paths whose most recent usage was more than a certain period of time in the past."Usage" is intentionally defined ambiguously, however in general all operations which produce/require the
presence of a given store path count as "usage"; this is tracked within a new local store DB column.
Fixes: #2793, #7572
Context
Most of the complexity related to this feature stems from properly tracking the "last usage time" of any given store object - implementing the actual age gate itself is rather trivial (which means that if this PR were to be rejected, a more simple MVP implementing just #2793 could be considered instead).
After some consideration, I have decided to implement this in the form of a new database field which is added to existing DBs using the scheme migration mechanism. This requires "bumping" the DB field whenever the store path is used, however other than checking the store path's
atime(which does not produce meaningful results on most modern Linux installations) I was unable to come up with any suitable alternative designs.This DB field is bumped whenever store path is produced / required by some piece of code, which most notably includes (cached) derivation builds - this means that e.g. executing
nix run nixpkgs#hellowill bump the usage time every time, even though no new derivations were built / substituted. Bumping is implemented using a newbumpLastUsageTimestore API method, which is invoked from various places scattered all around the code base. This sadly makes this change rather invasive vertically, as in it affecting a lot of files, however every individual made change on its own is not too invasive.The new
lastUsageTimefield is also exposed as part of the publicly documented store path info, and synced across the worker protocol using a new fine-grained protocol feature (as added in commit 3be7c00). This requires some more general plumbing to pass down the set of negotiated features to the serialization code, but allows the field to be accessed from non-root users using the Nix daemon.For easier reviewing, this PR is split into three commits:
lastUsageTimefield / associatedbumpLastUsageTimetime method, as well as the corresponding local store DB logic. It also adds various scatteredbumpLastUsageTimecalls all across the codebase. This commit does not yet implement synchronization of the field using the worker protocol though.path-last-usage-timeworker protocol feature, which when negotiated enables serialization of thelastUsageTimefield across worker protocol connections. This commit has been split off from the first commit since it necessitated modifying some baseline plumbing code to expose the set of negotiated features to the serialization code, and the concrete implementation featured here may be suboptimal / swapped out for a different method by the core Nix team if desired.lastUsageTimefield as a base. This commit itself is rather minimal, and as such could be easily cherry-picked / adapted to use the path registration time instead to at least fix idea: nix-collect-garbage option to also respect creation date of a derivation #2793 if this PR were to be rejected.Some final points which should be considered before this is merged:
bumpLastUsageTimemethod be forwarded using a new worker protocol operation? If yes this might facilitate the creation of LRU-style remote Nix caches, however for the sake of simplicity this has been cut for now.--older-thanage gate for profile GCs as a base for such tests, however I was unable to find any. Additionally, there is currently no common test support code for worker protocol serialization tests which depends on negotiated protocol features (from what I can tell). All in all, given the scope of the PR I have decided to first focus on finalizing the exact implementation of this feature in collaboration of the Nix team before writing tests which might become obsolete over the lifespan of the PR.Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.