Skip to content

Conversation

@niran
Copy link
Contributor

@niran niran commented Jan 25, 2026

Overview

Add background state trie cache warming that calculates state roots to pre-warm state trie caches before witness generation. This addresses performance issues when state root calculation is disabled.

Problem Statement

When the builder runs with disable_state_root: true, witness generation for the state trie uses cold caches, causing slow performance (~500-2000ms). This can contribute to missed blocks during high-load periods.

Solution

Proactively calculate state roots in the background after each flashblock (even though the result isn't used) to warm state trie caches. The warming process runs asynchronously and the warm caches remain available for fast witness generation.

Key Features

  • Background warming: State root calculation runs on the blocking thread pool after each flashblock publishes, never blocking the main build pipeline
  • Smart throttling: Only one warming task runs at a time using atomic flags; additional attempts are skipped and tracked in metrics
  • Non-interruptible: State root calculation runs to completion once started, but doesn't block new FCU arrivals
  • Comprehensive metrics: Full observability with start/complete/skip/error counters and duration histograms

Configuration

New CLI flag and environment variable:

--flashblocks.enable-state-trie-warming  # (default: false)
FLASHBLOCKS_ENABLE_STATE_TRIE_WARMING=true

Implementation Details

  1. Added CLI flag to FlashblocksArgs (crates/builder/cli/src/flashblocks.rs)
  2. Added config field to FlashblocksConfig (crates/builder/core/src/flashblocks/config.rs)
  3. Created StateTrieWarmer module with background task spawning (crates/builder/core/src/flashblocks/state_trie_warmer.rs)
  4. Added 5 new metrics for observability (crates/builder/core/src/metrics.rs)
  5. Integrated warming after each flashblock publishes (crates/builder/core/src/flashblocks/payload.rs)
  6. Warming uses spawn_blocking for CPU-intensive work

Metrics

Monitor the feature with:

  • base_builder_state_trie_warming_started_count - Tasks started
  • base_builder_state_trie_warming_completed_count - Tasks completed successfully
  • base_builder_state_trie_warming_skipped_count - Tasks skipped (already warming)
  • base_builder_state_trie_warming_duration - Duration histogram
  • base_builder_state_trie_warming_error_count - Error count

Test Plan (Sepolia Alpha)

  1. Deploy to Sepolia Alpha with FLASHBLOCKS_ENABLE_STATE_TRIE_WARMING=true
  2. Validate warming metrics are being recorded (started, completed, skipped, duration)
  3. Measure witness generation latency with warming enabled vs disabled
  4. Verify no regression in flashblock build times or missed blocks
  5. Monitor CPU usage impact of background warming under sustained load
  6. Verify correct behavior across FCU boundaries (warming from stale blocks completes without issues)

Files Changed

  • crates/builder/cli/src/flashblocks.rs - CLI flag
  • crates/builder/cli/src/builder.rs - Config conversion
  • crates/builder/core/src/flashblocks/config.rs - Config field
  • crates/builder/core/src/flashblocks/state_trie_warmer.rs - Core warming logic (new file, with unit tests)
  • crates/builder/core/src/flashblocks/payload.rs - Integration point
  • crates/builder/core/src/flashblocks/mod.rs - Module exports
  • crates/builder/core/src/metrics.rs - New metrics

@cb-heimdall
Copy link
Collaborator

cb-heimdall commented Jan 25, 2026

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Feb 9, 2026
Add background state trie cache warming feature that calculates state
roots to pre-warm state trie caches before witness generation. This
addresses performance issues when state root calculation is disabled.

When the builder runs with `disable_state_root: true`, witness generation
for the state trie uses cold caches, causing slow performance. By
proactively calculating state roots in the background (even though the
result isn't used), we warm these caches. The warming process runs
asynchronously after each flashblock, and the warm caches remain available
for fast witness generation.

- **Background warming**: State root calculation runs in background threads
  after each flashblock publishes, never blocking the main build pipeline
- **Smart throttling**: Only one warming task runs at a time using atomic
  flags; additional attempts are skipped and tracked in metrics
- **Non-interruptible**: State root calculation runs to completion once
  started, but doesn't block new FCU arrivals
- **Comprehensive metrics**: Full observability with start/complete/skip/error
  counters and duration histograms

New CLI flag and environment variable:
- `--flashblocks.enable-state-trie-warming` (default: false)
- `FLASHBLOCKS_ENABLE_STATE_TRIE_WARMING`

- **Cold cache state root**: ~500-2000ms (disk I/O intensive)
- **Warm cache state root**: ~50-200ms (CPU-bound)
- **Net improvement**: 5-10x faster witness generation
- **Use case**: Helps avoid missed blocks during high-load periods

1. Added CLI flag to FlashblocksArgs
2. Added config field to FlashblocksConfig
3. Created StateTrieWarmer module with background task spawning
4. Added 5 new metrics for observability
5. Integrated warming after each flashblock publishes
6. Warming uses spawn_blocking for CPU-intensive work

This is a DRAFT implementation to illustrate potential performance
improvements. It has NOT been tested in production and requires
thorough testing and validation before deployment.

The primary goal is to demonstrate a possible solution for avoiding
missed blocks by pre-warming state trie caches.
@niran niran force-pushed the builder-new-payload branch from 64c7022 to 2064044 Compare February 9, 2026 19:18
@niran niran added this to the v0.5.0 milestone Feb 9, 2026
@github-actions github-actions bot removed the Stale label Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants