apollo_storage: flush mmap files concurrently on dirty commits

[gkaempfer]

What

FileHandlers<RW>::flush() (in apollo_storage) flushes 8 independent mmap-backed files sequentially on every StorageTxnRW::commit(). Each flush() call is an independent blocking msync syscall on its own file. This PR:

• Runs the 8 flushes concurrently via std::thread::scope when at least one file is dirty, overlapping their msync calls instead of paying for them one after another.
• Adds a cheap needs_flush() check (reads the existing should_flush bool under the file's own mutex) so the thread::scope/8-spawn machinery is skipped entirely when nothing is dirty — which, since batch_size defaults to 1 and most commits (e.g. header/marker-only writes) don't touch every file, is the common case. Skipping this matters: unconditionally spawning 8 threads on every commit measured as a ~3000x regression (~100ns → ~300µs) on that common no-op path in review, and it's paid while holding the crate's global SharedState mutex, which would have serialized all other storage writers.
• Addresses the flushing concurrently half of the pre-existing TODO(dan) comment on this function (the flushing only the relevant files half remains, since spawning only for the specific dirty files was measured to buy nothing further given production mmap_file_config.max_size is ~1TB and msync cost dominates any spawn overhead once ≥1 file is actually dirty).

Why

Follow-up performance review of #14927 and #14928 (both merged into main), triggered automatically after those PRs merged. Neither of those PRs itself needed further optimization (they're log-verbosity/cost changes), but #14927 touches this exact flush() function and its diff includes the TODO(dan) comment above, which flagged an unaddressed, real hot-path optimization opportunity: flush() runs on every block commit and was doing 8 sequential blocking I/O syscalls.

This PR went through two internal review-and-fix rounds (via an independent Opus review) before being opened:

1. First pass flagged that spawning threads unconditionally regressed the common all-clean-commit case (batching is off by default) — fixed by adding the needs_flush() gate.
2. Second pass flagged that the fast-path early return bypassed the #[latency_histogram] macro's latency recording (the macro's return_value = { ... } wrapper doesn't see returns from inside the wrapped block) — fixed by restructuring to an if block instead of an early return, so storage_file_handler_flush_latency_seconds still observes every call including the no-op case.

Test plan

• cargo build -p apollo_storage
• SEED=0 cargo test -p apollo_storage — 317 passed, 0 failed, 1 ignored (matches origin/main baseline)
• cargo clippy -p apollo_storage --all-targets — clean
• scripts/rust_fmt.sh — no diff

🤖 Generated with Claude Code

---

Generated by Claude Code

[cursor]

PR Summary

Medium Risk
Touches the hot storage commit path and concurrent msync on shared mmap state, though behavior is unchanged when files are dirty and the fast path only avoids work when nothing needs flushing.

Overview
FileHandlers::flush() no longer always runs eight sequential msync calls on every batch commit. It first ORs needs_flush() across all mmap-backed writers (new on the Writer trait, backed by the existing should_flush flag) and skips thread::scope entirely when nothing is dirty—a common path when batching is off and commits don’t touch every file.

When at least one file is dirty, the same eight flush() calls run in parallel via std::thread::scope so independent blocking I/O can overlap. The flush path stays inside the latency histogram wrapper (no early return) so no-op flushes are still measured.

^{Reviewed by Cursor Bugbot for commit ab72821. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[github-actions]

Artifacts upload workflows:

• Started at 2026-08-10T10:37:35Z