Skip to content

Conversation

@LNSD
Copy link
Contributor

@LNSD LNSD commented Dec 22, 2025

Encapsulate object store and metadata database operations into Store and CachedStore types. This centralizes data layer logic and makes dependencies explicit throughout the codebase.

  • Add Store type wrapping object store + metadata DB
  • Add CachedStore for read-optimized access patterns
  • Refactor PhysicalTable and catalog readers to use Store
  • Update all consumers (dump, server, worker) to pass Store explicitly

Note

Encapsulates object store + metadata DB into Store/CachedStore and refactors catalog, dump, query, and services to use it, consolidating metadata ops and parquet caching.

  • Core/Common:
    • Store API: Add store::Store (wraps metadata DB + object store) and store::CachedStore (adds parquet metadata caching) with helpers for table revision/file ops and object I/O; centralize related error types.
    • Catalog/Physical: Refactor PhysicalTable to hold Store; move table registration/activation, file listing/registration, and deletion through Store. Replace get_active_physical_table with Store::get_active_table_revision; update register_new_table_revision, restore_latest_revision, snapshots, and listing APIs.
    • Reader: Simplify parquet reader factory to fetch metadata via CachedStore and create readers via Store (remove direct foyer usage here).
    • Metadata helpers: Update parquet metadata extraction functions to take Store.
    • Query: QueryEnv now passes parquet_cache_size_mb; QueryContext::for_catalog builds a CachedStore and threads it through planning/registration.
  • Dump pipeline:
    • Update compactor, collector, raw/derived writers, and streaming query to use Store/CachedStore for listing, writing, deleting, and metadata registration.
    • ParquetFileWriter now receives Store, BufWriter, and FileName; footer extraction via Store.
    • consistency_check requires Store; object-store ops routed through it.
  • Services/CLIs:
    • Binaries (controller_cmd, server_cmd, solo_cmd, worker_cmd) construct Store with metadata_db; services pass Store instead of raw object store/DB.
  • Metadata DB:
    • Rename API: physical_table::get_active_physical_table -> get_active.
  • Tests:
    • Adjust helpers and fixtures to construct/pass Store/CachedStore; update restore and consistency checks; replace direct object-store calls with Store methods.

Written by Cursor Bugbot for commit a5be127. This will update automatically on new commits. Configure here.

@LNSD LNSD self-assigned this Dec 22, 2025
@LNSD LNSD force-pushed the lnsd/feat-common-data-store branch 4 times, most recently from e04b57d to 1ed3c2c Compare December 22, 2025 11:33
@LNSD LNSD marked this pull request as ready for review December 22, 2025 11:40
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

@LNSD LNSD force-pushed the lnsd/feat-common-data-store branch 2 times, most recently from 3df096a to 0f66e54 Compare December 22, 2025 12:14
Encapsulate object store and metadata database operations into `Store` and `CachedStore` types. This centralizes data layer logic and makes dependencies explicit throughout the codebase.

- Add `Store` type wrapping object store + metadata DB
- Add `CachedStore` for read-optimized access patterns
- Refactor `PhysicalTable` and catalog readers to use Store
- Update all consumers (dump, server, worker) to pass Store explicitly

Signed-off-by: Lorenzo Delgado <lorenzo@edgeandnode.com>
@LNSD LNSD force-pushed the lnsd/feat-common-data-store branch from 0f66e54 to a5be127 Compare December 22, 2025 16:47
@LNSD LNSD merged commit ac23370 into main Dec 22, 2025
9 checks passed
@LNSD LNSD deleted the lnsd/feat-common-data-store branch December 22, 2025 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants