Skip to content

Conversation

@niran
Copy link
Contributor

@niran niran commented Nov 14, 2025

Adds metered priority fee estimation to the Base node, enabling bundles to receive priority fee recommendations based on real-time resource consumption from recent flashblocks.

base_meteredPriorityFeePerGas - Simulates a bundle and returns a recommended priority fee based on recent flashblock congestion across four resource dimensions:

  • Gas - L2 gas consumption
  • Execution time - CPU time spent executing transactions
  • State root time - Time spent computing state roots
  • Data availability - L1 DA bytes (dynamically updated via miner_setMaxDASize)

The estimator uses a displacement algorithm: for each resource, it finds the minimum priority fee required to displace enough existing transactions to make room for the bundle's resource demand, then takes the maximum across all resources and median across recent blocks.

Architecture

The metering pipeline consists of:

  1. Kafka Consumer (KafkaBundleConsumer) - Consumes AcceptedBundle events from TIPS, extracting per-transaction resource usage
  2. Resource Annotator - Correlates Kafka metering data with flashblock inclusions from the websocket feed, storing transactions in sequencer order
  3. Metering Cache - Maintains recent block/flashblock metrics with automatic eviction and reorg detection
  4. Priority Fee Estimator - Computes displacement-based fee estimates across multiple resource dimensions

CLI Configuration

--enable-metering                      Enable metering RPC
--metering-kafka-properties-file       Path to Kafka properties file
--metering-kafka-topic                 Topic name (default: tips-ingress)
--metering-kafka-group-id              Consumer group ID override
--metering-gas-limit                   Gas limit per flashblock (default: 30M)
--metering-execution-time-us           Execution time budget (default: 50ms)
--metering-state-root-time-us          State root time budget (optional)
--metering-da-bytes                    DA bytes limit (default: 120KB)
--metering-priority-fee-percentile     Percentile for fee recommendation (default: 0.5)
--metering-uncongested-priority-fee    Default fee when uncongested (default: 1 wei)
--metering-cache-size                  Blocks to retain (default: 12)

The metering system supports miner_setMaxDASize to dynamically update the limit for DA bytes allowed in a block.

Test Plan

  • Unit tests for cache operations (insert, eviction, reorg clearing)
  • Unit tests for estimator logic (displacement algorithm, congestion detection)
  • Integration tests for annotator reorg handling
  • Existing RPC tests continue to pass
  • cargo +nightly fmt and cargo +nightly clippy pass

@cb-heimdall
Copy link
Collaborator

cb-heimdall commented Nov 14, 2025

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Dec 19, 2025
@niran niran force-pushed the node-reth-metered-priority-fee branch 2 times, most recently from 38f238a to 7d9523f Compare December 22, 2025 22:30
Add MeteringCache and PriorityFeeEstimator to crates/rpc for
resource-aware priority fee estimation in flashblocks.

- cache.rs: In-memory cache storing metered transactions by
  block/flashblock with resource totals (gas, execution time,
  state root time, data availability bytes)
- estimator.rs: Top-down fee estimation algorithm that determines
  minimum priority fee needed for bundle inclusion based on
  resource competition
- kafka.rs: KafkaBundleConsumer reads accepted bundle events from
  Kafka and extracts per-transaction metering data including gas,
  execution time, and data availability bytes
- annotator.rs: ResourceAnnotator correlates Kafka metering data
  with flashblock inclusion events to populate the metering cache
  with properly indexed transaction data
@github-actions github-actions bot removed the Stale label Dec 23, 2025
Wires priority fee estimation into the RPC layer:
- Add MeteredPriorityFeeResponse types with resource-specific estimates
- Add metered_priority_fee_per_gas method to MeteringApi trait
- Update MeteringApiImpl to accept optional PriorityFeeEstimator
- Add MeteringConfig, KafkaConfig, ResourceLimitsConfig to runner
- Wire estimator creation in BaseRpcExtension when Kafka is configured
- Add 10 CLI args for resource limits, cache size, and Kafka config
Port additional tests from original metering crate:
- compute_estimate_empty_transactions: verify uncongested behavior
- estimate_for_block_respects_limits: integration test for block estimates
- estimate_for_block_propagates_limit_errors: verify error propagation
- estimate_rolling_aggregates_across_blocks: verify rolling median calculation
Completes the metering pipeline by:
- Creating MeteringRuntime to hold cache, estimator, and channels
- Spawning ResourceAnnotator task to correlate transactions with flashblocks
- Spawning KafkaBundleConsumer task to consume AcceptedBundle events
- Adding CompositeFlashblocksReceiver that forwards to both FlashblocksState
  and the metering pipeline via FlashblockInclusion events
- Adding flashblock_inclusion_from_flashblock helper to extract tx hashes
- Supporting Kafka properties file loading

The metering cache is now populated when:
1. Kafka consumer receives AcceptedBundle events (transaction data)
2. FlashblocksSubscriber receives flashblocks (inclusion position)
3. ResourceAnnotator correlates both to update the cache
@niran niran force-pushed the node-reth-metered-priority-fee branch from 7d9523f to c381fa9 Compare December 23, 2025 05:11
- Add Kafka startup warning when metering is enabled but Kafka is not configured
- Wire shared OpDAConfig from BaseNodeConfig to OpNode and PriorityFeeEstimator
  so that miner_setMaxDASize affects priority fee estimation
- Add doc comment for metering-da-bytes explaining dynamic override capability
- Add reth-optimism-payload-builder dependency to runner and node crates
- Change FlashblockMetrics storage from IndexMap to sorted Vec
- Use binary search insertion to maintain descending order by priority fee
- Remove redundant sorting in compute_estimate (now expects pre-sorted input)
- Rename upsert_transaction to insert_transaction (upserts weren't used)
- Remove unused ResourceTotals::subtract method
- Keep sort only for aggregate resources (merging multiple flashblocks)
- Change fee fields from String to U256 (matches Ethereum RPC conventions)
- Rename RollingPriorityEstimates to RollingPriorityEstimate (singular)
- Rename recommended_priority_fee to priority_fee (shorter, clearer)
- Rename resource_estimates_to_vec to build_resource_estimate_responses
- Update variable names to match singular type
Keep only exceptional/diagnostic metrics:
- metering.kafka.errors_total (errors)
- metering.kafka.lag_ms (monitoring)
- metering.kafka.messages_skipped (data issues)
- metering.pending.evicted (capacity issues)
- metering.pending.size (queue depth monitoring)
- metering.streams.tx_misses_total (correlation failures)

Remove normal activity counters:
- metering.kafka.messages_total
- metering.kafka.tx_events_total
- metering.streams.flashblocks_total
- metering.streams.tx_matched_total
- Add contains_block() and clear_blocks_from() methods to MeteringCache
- Detect reorg when flashblock_index=0 arrives for existing block
- Clear affected blocks from cache on reorg detection
- Add metering.cache.reorgs_detected metric
- Add unit and integration tests for reorg handling
- Remove --metering-kafka-brokers CLI arg (use properties file instead)
- Make --metering-kafka-properties-file the only required arg for Kafka
- Default --metering-kafka-topic to tips-ingress
- Make --metering-kafka-group-id optional (overrides properties file)
- Load all rdkafka settings from properties file instead of hardcoding
@niran niran marked this pull request as ready for review December 23, 2025 08:27
- Move ResourceFeeEstimateResponse and MeteredPriorityFeeResponse to types.rs
- Move build_priority_fee_response helper to meter_rpc.rs
- Delete metered_fee_types.rs
- Rename insert_transaction to push_transaction
- Remove binary search insertion, just append in sequencer order
- Update docs and test to reflect sequencer ordering is preserved
- Add const fn annotations where possible (cache, estimator, runner)
- Add Debug impls for ResourceAnnotator and KafkaBundleConsumer
- Remove redundant clones in meter_rpc and runner
- Change pub mod to pub(crate) mod for internal modules
- Replace redundant closure with function reference
- Remove unused cache field from MeteringRuntime
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants