Skip to content

Conversation

@LNSD
Copy link
Contributor

@LNSD LNSD commented Dec 23, 2025

Consolidate data storage and caching into a single abstraction to simplify the architecture and eliminate the distinction between cached/uncached stores.

  • Unified DataStore combines storage and cache in a single type
  • Integrated parquet metadata cache with memory-aware eviction
  • Centralized physical table operations and file naming
  • Single store interface across all services and modules

Note

Introduces a unified data access layer and refactors the codebase to use it.

  • Add new amp-data-store crate providing DataStore (object store + metadata DB) with integrated parquet footer cache (memory-weighted eviction via foyer)
  • Move physical-table utilities (PhyTableUrl, PhyTablePath, PhyTableRevisionPath) and FileName into amp-data-store; update error types and uses
  • Remove common::store (including CachedStore); replace with DataStore in common catalog/metadata/reader/query_context and related APIs
  • Refactor services (server, controller, worker, admin-api), CLI (ampd), dump, datasets-derived, and tests to depend on and construct DataStore (passing config.parquet.cache_size_mb)
  • Config/query env: drop parquet_cache_size_mb from QueryEnv/server config and Config::make_query_env; caching handled by DataStore::new
  • Minor API/log tweaks: StreamingQueryHandle::as_streaminto_stream, replace debug! with tracing::debug! in some paths

Written by Cursor Bugbot for commit 2f5a5ea. This will update automatically on new commits. Configure here.

@LNSD LNSD self-assigned this Dec 23, 2025
Consolidate data storage and caching into single abstraction to simplify architecture and eliminate the cached/uncached store distinction.

- Unified `DataStore` combines storage and cache in single type
- Integrated parquet metadata cache with memory-aware eviction
- Centralized physical table operations and file naming
- Single store interface across all services and modules

Signed-off-by: Lorenzo Delgado <lorenzo@edgeandnode.com>
@LNSD LNSD force-pushed the lnsd/feat-amp-data-store-crate branch from 703c629 to 2f5a5ea Compare December 23, 2025 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants