Skip to content

Cache and replay the mono item partitioning on green incremental rebuilds - #109

Draft
xmakro wants to merge 1 commit into
perf/mono-base-0809from
perf/mono-partition-replay
Draft

Cache and replay the mono item partitioning on green incremental rebuilds#109
xmakro wants to merge 1 commit into
perf/mono-base-0809from
perf/mono-partition-replay

Conversation

@xmakro

@xmakro xmakro commented Aug 10, 2026

Copy link
Copy Markdown
Owner

On an incremental rebuild where nothing changed, the compiler still re-runs the entire mono item collection and CGU partitioning pipeline: the root walk over the crate, the reachability fixpoint that green-replays and decodes items_of_instance for every mono item, usage map construction, placement, merging, internalization, per item size estimates, the stable sort of all items, the symbol uniqueness assertion that computes the symbol name of every single item, and finally a stable hash of the whole partition for the dep node fingerprint. All of that work only reproduces the same codegen units as last session so that determine_cgu_reuse can look up work products by name. collect_and_partition_mono_items is eval_always, so none of it can be skipped today. It is the largest remaining fully non-incremental pass in a warm debug or opt rebuild.

This change makes the query a normal tracked query and caches its value in the incremental cache. On a fully green rebuild the node is marked green by walking its recorded dependency edges, which is one fingerprint comparison per edge and loads no values, and the partition is decoded once from the cache when the codegen driver asks for it (79ms for the cargo lib crate, replacing the multi second pipeline above). Diagnostics from monomorphization time checks still replay, because check_mono_item and the const eval queries are dependencies of this node and marking forces them as needed. On a red rebuild the provider executes exactly as before and the result is stable hashed exactly as before (the eval_always node was already fingerprinted to enable backdating). Per CGU reuse on red rebuilds is unaffected: the driver pre-executes codegen_unit(name) for every CGU before determine_cgu_reuse, and those nodes backdate to green individually when their CGU is unchanged, which is the existing firewall between the whole crate partition and individual CGU invalidation. A pleasant side effect is that patches which do not change the mono item graph at all now skip collection even in the patched rebuild itself, because the partition node simply marks green.

Making this work exposed a hole in the incremental cache: values of green nodes that were never loaded during a session are only carried into the next cache file by the promotion pass, and that pass requires recovering the query key from the dep node. Queries keyed by an Instance have opaque key fingerprints, so their unloaded values were silently dropped. Today that never shows because the eval_always collector loads every items_of_instance, size_estimate and symbol_name value in every session; with the replay in place, one unchanged rebuild would drop them all and the next real change would re-execute the collector fixpoint from scratch (twelve thousand items_of_instance executions with zero cache hits on the image crate, and up to a doubled patched rebuild). The second half of this change is therefore a key-free carry-forward path, which also answers the FIXME in save.rs asking whether values can be promoted without decoding them into the memory cache: a new encode_cached_value_fn on the query vtable decodes the previous value by dep node index and re-encodes it directly into the new cache file, driven from serialization for green nodes whose value the result cache has not already written. It is deliberately restricted to the three queries the replay leaves unloaded. Extending it to const eval allocations is a pessimization (carrying ctfe-stress-5's huge allocation values cost thirty percent of its unchanged rebuild), so other opaque keyed queries keep the existing drop behavior; the cost of that choice is that a rebuild after a change re-evaluates the constants the collector mentions instead of loading them, which shows up as a fraction of a percent on the patched scenarios. To keep the pass free for crates that have no such values at all, the encoder records the affected indices in a small per kind side table in the cache footer while writing values, so enumerating carry candidates never scans the full value index (an earlier version that scanned it cost match-stress three percent of its check rebuild; with the side table the probe is at noise level). Two supporting fixes: the previous cache file's mmap now stays open until the new file is written (unlinking a mapped file is fine on unix hosts; Windows would need the old file to be renamed instead, which I have left out of the prototype), and the encoder's source file index now extends lazily because decoding carried values can import source files after the snapshot is taken.

Validation: tests/incremental (180 tests, including the module source reuse assertions) and tests/codegen-units pass, and every measurement ran with -Zincremental-verify-ich, which re-hashes all values loaded from the cache, including the new partition value, against the fingerprint recorded when they were computed. A synthetic crate covering global_asm, vtables, coroutines, closures, generic drop glue and thread local statics round-trips the cache across unchanged and patched sessions. One behavioral caveat: -Zprint-mono-items and -Zdump-mono-stats only produce output when the query actually executes, so on a fully green incremental rebuild they now print nothing; the tests that use them compile non incrementally, and both flags are dep tracked, so enabling them invalidates the previous session and the output is produced.

Numbers are from a clean from scratch A/B against the base commit in the dist configuration proxy (stage2, thin LTO, jemalloc), instructions:u over the full local rustc-perf suite (52 benchmarks, Check/Debug/Opt, Full/IncrFull/IncrUnchanged/IncrPatched, 551 cells):

profile / scenario mean movers (>=0.25%)
debug / incr-unchanged -2.38% 31 improved, 2 regressed
opt / incr-unchanged -2.32% 30 improved, 2 regressed
debug / incr-patched +0.20% 3 improved, 12 regressed
opt / incr-patched +0.07% 1 improved, 6 regressed
check, all scenarios +0.02% to +0.06% flat
full and incr-full, all profiles ~0.03% flat

The largest improvements are on the biggest real world crates: image opt incr-unchanged -13.7%, regression-31157 -13.3%, cargo debug incr-unchanged -13.1%, ripgrep -9.9%, large-workspace -7.4%, cranelift-codegen -6.1%, and the win scales with crate size, since the removed work is proportional to the number of mono items. The worst cell in the whole matrix is cargo debug incr-patched println at +1.31%, which is the const re-evaluation cost described above. Max RSS is flat on average (-0.12% mean). Pooled over all 551 cells the mean is -0.34%. A byte level carry-forward of cache regions would make the remaining carry pass nearly free and should recover a few more points on both the unchanged and patched sides; that mechanism exists separately and composes with this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant