diff --git a/src/query.md b/src/query.md index 8377a7b2f..5df9de762 100644 --- a/src/query.md +++ b/src/query.md @@ -69,22 +69,24 @@ are cheaply cloneable; insert an `Rc` if necessary). If, however, the query is *not* in the cache, then the compiler will call the corresponding **provider** function. A provider is a function -implemented in a specific module and **manually registered** into the -[`Providers`][providers_struct] struct during compiler initialization. -The macro system generates the [`Providers`][providers_struct] struct, -which acts as a function table for all query implementations, where each +implemented in a specific module and **manually registered** into either +the [`Providers`][providers_struct] struct (for local crate queries) or +the [`ExternProviders`][extern_providers_struct] struct (for external crate queries) +during compiler initialization. The macro system generates both structs, +which act as function tables for all query implementations, where each field is a function pointer to the actual provider. -**Note:** The `Providers` struct is generated by macros and acts as a function table for all query implementations. -It is **not** a Rust trait, but a plain struct with function pointer fields. +**Note:** Both the `Providers` and `ExternProviders` structs are generated by macros and act as function tables for all query implementations. +They are **not** Rust traits, but plain structs with function pointer fields. **Providers are defined per-crate.** The compiler maintains, internally, a table of providers for every crate, at least -conceptually. Right now, there are really two sets: the providers for -queries about the **local crate** (that is, the one being compiled) -and providers for queries about **external crates** (that is, -dependencies of the local crate). Note that what determines the crate -that a query is targeting is not the *kind* of query, but the *key*. +conceptually. There are two sets of providers: +- The `Providers` struct for queries about the **local crate** (that is, the one being compiled) +- The `ExternProviders` struct for queries about **external crates** (that is, +dependencies of the local crate) + +Note that what determines the crate that a query is targeting is not the *kind* of query, but the *key*. For example, when you invoke `tcx.type_of(def_id)`, that could be a local query or an external query, depending on what crate the `def_id` is referring to (see the [`self::keys::Key`][Key] trait for more @@ -117,31 +119,31 @@ they define both a `provide` and a `provide_extern` function, through ### How providers are set up -When the tcx is created, it is given the providers by its creator using -the [`Providers`][providers_struct] struct. This struct is generated by -the macros here, but it is basically a big list of function pointers: - -[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html +When the tcx is created, it is given both the local and external providers by its creator using +the `Providers` struct from `rustc_middle::util`. This struct contains both the local and external providers: ```rust,ignore -struct Providers { - type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>, - // ... one field for each query +pub struct Providers { + pub queries: crate::query::Providers, // Local crate providers + pub extern_queries: crate::query::ExternProviders, // External crate providers + pub hooks: crate::hooks::Providers, } ``` +Each of these provider structs is generated by the macros and contains function pointers for their respective queries. + #### How are providers registered? -The `Providers` struct is filled in during compiler initialization, mainly by the `rustc_driver` crate. -But the actual provider functions are implemented in various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc). +The `util::Providers` struct is filled in during compiler initialization, by the `rustc_interface` crate from the `DEFAULT_QUERY_PROVIDERS` static. +The actual provider functions are defined across various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc). To register providers, each crate exposes a [`provide`][provide_fn] function that looks like this: [provide_fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/hir/fn.provide.html ```rust,ignore -pub fn provide(providers: &mut Providers) { - *providers = Providers { +pub fn provide(providers: &mut query::Providers) { + *providers = query::Providers { type_of, // ... add more providers here ..*providers @@ -149,27 +151,104 @@ pub fn provide(providers: &mut Providers) { } ``` -- This function takes a mutable reference to the `Providers` struct and sets the fields to point to the correct provider functions. -- You can also assign fields individually, e.g. `providers.type_of = type_of;`. +Note that this function accepts `query::Providers` not `util::Providers`. It is exceedingly rare to need a `provide` function that doesn't just accept `query::Providers`. If more than the `queries` field of `util::Providers` is being updated then `util::Providers` can be accepted instead: +```rust,ignore +pub fn provide(providers: &mut rustc_middle::util::Providers) { + providers.queries.type_of = type_of; + // ... add more local providers here + + providers.extern_queries.type_of = extern_type_of; + // ... add more external providers here + + providers.hooks.some_hook = some_hook; + // ... add more hooks here +} +``` + +Note that `util::Providers` implements `DerefMut` to `query::Providers` so callers of the `provide` functions can pass in a `util::Providers` and it will just work for provider functions that accept `query::Providers` too + +- This function takes a mutable reference to the `query::Providers` struct and sets the fields to point to the correct provider functions. +- You can also assign queries individually, e.g. `providers.type_of = type_of;`. +- You can assign fields individually for each provider type (local, external, and hooks). #### Adding a new provider -Suppose you want to add a new query called `fubar`. You would: +Suppose you want to add a new query called `fubar`. This section focuses on wiring up the providers; for how to declare the query itself in the big `rustc_queries!` macro, see [Adding a new query](#adding-a-new-query) below. + +In practice you usually: -1. Implement the provider function: +1. Decide which crate "owns" the query (for example `rustc_hir_analysis`, `rustc_mir_build`, or another `rustc_*` crate). +2. In that crate, look for an existing `provide` function: + ```rust,ignore + pub fn provide(providers: &mut query::Providers) { + // existing assignments + } + ``` + If it exists, you will extend it to set the field for your new query. If the crate does not yet have a `provide` function, add one and make sure it is included in `DEFAULT_QUERY_PROVIDERS` in the `rustc_interface` crate so that it actually gets called during initialization (see the discussion above). +3. Implement the provider function itself: ```rust,ignore - fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... } + fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... } ``` -2. Register it in the `provide` function: +4. Register it in the crate's `provide` function: ```rust,ignore - pub fn provide(providers: &mut Providers) { - *providers = Providers { + pub fn provide(providers: &mut query::Providers) { + *providers = query::Providers { fubar, ..*providers }; } ``` +### How queries interact with external crate metadata + +When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata. +This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files. + +The process works like this: + +1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`. + This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct]. + +2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct. + The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like: + ```rust + pub fn provide_extern(providers: &mut ExternProviders) { + providers.foo = |tcx, def_id| { + // Load and decode metadata for external crate + let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate); + cdata.foo(def_id.index) + }; + // Register other external providers... + } + ``` + +3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate: + - Loads the `.rmeta` file for the external crate + - Decodes the metadata using the `Decodable` trait + - Returns the decoded information to the query system + +This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries. +Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will: +1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE` +2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata` +3. The provider will load and decode the type information from the external crate's metadata +4. Return the decoded type to the caller + +This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system. +The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers. + +When we define a new query that should work across crates, it does not automatically become cross-crate just because it is listed in `rustc_queries!`. You will typically need to: + +- Add the query to `rustc_queries!` with appropriate modifiers (for example whether it is cached on disk). +- Implement a local provider in the owning crate and register it via that crate's `provide` function. +- Add an external provider in `rustc_metadata` via `provide_extern`, and ensure the query's result is encoded and decoded in the crate metadata. + +An example of introducing such a cross-crate query can be found in commit [`996a185`](https://github.com/rust-lang/rust/commit/996a185) in the `rust-lang/rust` repository. + +[rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html +[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html +[extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html + --- ## Adding a new query @@ -263,4 +342,3 @@ More discussion and issues: [GitHub issue #42633]: https://github.com/rust-lang/rust/issues/42633 [Incremental Compilation Beta]: https://internals.rust-lang.org/t/incremental-compilation-beta/4721 [Incremental Compilation Announcement]: https://blog.rust-lang.org/2016/09/08/incremental.html -