diff --git a/changelog.d/11676-gc-per-object-trace-cost.md b/changelog.d/11676-gc-per-object-trace-cost.md new file mode 100644 index 0000000000..43b3791cf7 --- /dev/null +++ b/changelog.d/11676-gc-per-object-trace-cost.md @@ -0,0 +1,10 @@ +perf(gc): the copying minor's per-live-object trace cost is cut by about a third (#11549). A copying minor over a fully-live binary tree (131,076 promoted objects, `btree` n=20) went from 1,472 to 953 instructions per live object, not counting the first-cycle barrier-arming walk that #11668 addresses separately (callgrind, `gc_collect_minor_with_trigger_inner` only). The biggest part was plumbing, not marking. On that tree, instructions fall 24.6% (n=20) and 22.5% (n=40). `gc_ratchet` 02 falls 7.9%, 12 falls 8.7%, retain 4.9% and qs stringify 2.3%. Peak RSS with THP off is within ±1% on every row except moment, whose GC schedule is nondeterministic. + +- **Ordinary objects take a straight-line scan** (`gc/copying_object_scan.rs`). The drain used to reach an object's slots through the generic rewrite walk, the `HeapChildSlotIterator` state machine and two indirect visitor calls per slot. It now builds the same slots as a plan: the residual-prototype edge, the shape record, field range and payload selection (`heap_payload_slot_selection_from` itself, so the mask logic has one copy), the carrier note, the keys edge, the meta record, the payload and the overflow fields, in the generic walk's order. It declines only up front (a full trace or layout-scan trace in progress, or a type table that no longer describes `GC_TYPE_OBJECT` this way). In test and debug-assertion builds every object it scans is also enumerated by the generic walk and the two slot lists must match. `gc::tests::copying_object_scan` shows the path is taken, that it still scans with the residual-prototype latch armed, and that a plan that drops a slot is refused. +- **The parent's remembering question is asked once per object** (`ParentRemembering`), and the slot's generation is classified only when an answer depends on it. On a whole-block promoting cycle (`skip_remembering`), no slot is classified. Before this, every slot paid a page-map probe for an answer nothing read. It is checked against `barrier_parent_needs_remembering` over old, young and malloc parents, with a sabotaged twin. +- **Raw words consult the mark memo before classifying.** The memo holds only addresses that classified this cycle, and a classification cannot change within a cycle. Every shaped receiver's `keys` word hits it. Test and debug builds re-derive the premise. +- **`SHAPE_LAYOUTS` memoizes its last pointer-mask answer.** The table is reachable mutably only through `DerefMut`, which clears the memo, so the memo is exact by construction. `gc::tests::shape_layout_table` has a sabotaged twin that keeps the memo across a write. +- **The old-carrier note skips its generation probe when it would change nothing** (both record flags already set). `gc::tests::copying_object_scan` shows that a promoted receiver still notes its shape, and that claiming every shape noted loses the note. +- The drain's slot walk is instantiated for its visitor (`visit_gc_rewrite_slots_inline`), and its classification is inlined. Other callers keep the shared `dyn` walk. + +Not in this change: the full collection. dotenv's fulls are dominated by the valid-pointer-set build, the sweep and side-table clears, not by tracing. The per-cycle side-table passes that dominate `qs` minors are also not here: the closure box-capture prune, the per-object layout owner prune and its sort, and the remembered-set rebuild. diff --git a/crates/perry-runtime/src/gc/copying.rs b/crates/perry-runtime/src/gc/copying.rs index 6e2a8d615e..ad052d55fd 100644 --- a/crates/perry-runtime/src/gc/copying.rs +++ b/crates/perry-runtime/src/gc/copying.rs @@ -1,4 +1,4 @@ -use super::copying_parent_facts::weak_holder_fact; +use super::copying_parent_facts::{weak_holder_fact, ParentRemembering}; use super::copying_phase::{ finalize_dead_copied_minor_from_space_side_allocations, CopyingMinorPhase as Phase, CopyingMinorPhaseDiag as PhaseDiag, @@ -416,10 +416,17 @@ impl CopyingNurseryCollector { if addr == self.memo_addr { return Some(self.memo_result); } - let ptr = self.ptrs.classify(addr)?; + let ptr = self.ptrs.classify_inline(addr)?; Some(self.mark_classified(addr, ptr)) } + /// The memo's answer for `addr`, if `addr` is the last address a mark + /// classified successfully this cycle. See `memo_addr`. + #[inline(always)] + pub(super) fn memo_hit(&self, addr: usize) -> Option { + (addr == self.memo_addr).then_some(self.memo_result) + } + /// [`mark_addr`](Self::mark_addr) for an address the caller has already /// classified: the memo, then the mark, without classifying again. #[inline] @@ -710,17 +717,27 @@ impl CopyingNurseryCollector { } pub(super) unsafe fn scan_object_fields(&mut self, header: *mut GcHeader) { + // The common case, written out (#11549): see `gc/copying_object_scan.rs`. + if self.scan_plain_object(header) { + return; + } let mut changed = false; // LAZY, not eager. Reading the fact once per traced OBJECT regressed // all six fixtures (+0.88 % to +5.09 % instructions): a great many // traced objects — strings, pointer-free arrays — have no slot to // visit at all, and paid for an answer nobody then asked for. let mut weak_holder: Option = None; - visit_gc_rewrite_slots(header, |slot| unsafe { + // Same laziness as the weak fact, and the same per-object shape: see + // `ParentRemembering`. + let mut remembering: Option = None; + let skip_remembering = self.skip_remembering; + visit_gc_rewrite_slots_inline(header, |slot| unsafe { slot.record_layout_read(); let before = *slot.slot; let weak = *weak_holder.get_or_insert_with(|| weak_holder_fact(header)); - self.visit_slot_with_weak_fact(slot.slot, header, weak, slot.external()); + let remembering = + *remembering.get_or_insert_with(|| ParentRemembering::of(header, skip_remembering)); + self.visit_slot_with_parent_facts(slot, header, weak, remembering); changed |= *slot.slot != before; }); if changed { diff --git a/crates/perry-runtime/src/gc/copying_object_scan.rs b/crates/perry-runtime/src/gc/copying_object_scan.rs new file mode 100644 index 0000000000..2625238627 --- /dev/null +++ b/crates/perry-runtime/src/gc/copying_object_scan.rs @@ -0,0 +1,405 @@ +//! The copying drain's straight-line scan of an ordinary object (#11549). +//! +//! `scan_object_fields` reaches an object's slots through the generic walk: +//! `visit_gc_rewrite_slot_descriptors` → `visit_gc_layout_slot_descriptors` → +//! a `HeapChildSlotIterator` built by `gc_child_slots`, with every slot handed +//! through two visitor closures. For a fully-live binary tree that plumbing — +//! not the marking — was most of the ~1,470 instructions each promoted object +//! cost (callgrind, `gc_collect_minor_with_trigger_inner` only). +//! +//! This is the same walk for the one kind that dominates real heaps, +//! `GC_TYPE_OBJECT`, written out: the same residual-prototype edge first, the +//! same shape record, field range and payload selection +//! (`heap_payload_slot_selection_from` itself, so the mask logic has ONE copy), +//! the same shape-keys-edge bookkeeping, and the same slots in the same order — +//! residual prototype, keys edge, meta record, payload, overflow fields. +//! +//! It declines (returns `false` having done nothing) only up front, from the +//! header and thread state: a full trace or layout-scan trace in progress (the +//! generic walk records carrier notes and slot counters this path does not), or +//! a type-table entry that stopped describing `GC_TYPE_OBJECT` the way this +//! assumes. Once admitted it never hands the object back, so no edge is visited +//! twice. +//! +//! Drift is the risk of a second enumeration, so it is not trusted: in test and +//! debug-assertion builds every object this path handles is ALSO enumerated by +//! the generic walk, and the two slot lists must be identical +//! (`assert_matches_generic_walk`). `gc::tests::copying_object_scan` proves that +//! check fires on a sabotaged plan and that the path is actually taken. + +use super::copying_parent_facts::{weak_holder_fact, ParentRemembering}; +use super::*; + +/// The slots, in visit order, the generic walk's layout arm hands the drain for +/// this object: the keys edge and the meta record (null when absent), then the +/// payload slots its selection names. Iterated, not stored. +#[derive(Clone)] +struct PlainObjectPlan { + prefix: [*mut u64; 2], + next_prefix: usize, + payload: HeapSlotRange, + walk: PayloadWalk, +} + +/// The payload selection, as the generic walk would iterate it. +#[derive(Clone)] +enum PayloadWalk { + /// A one-word mask (`take_inline_mask_word`'s walk), already limited. + Word(u64), + /// Every slot `next..count` (`AllPointers` / `All`: a `Range`). + Range { next: usize, count: usize }, + /// A mask wider than one word (the iterator's `Masked` arm). + Mask { + mask: LayoutSlotMask, + cursor: usize, + count: usize, + }, +} + +impl PlainObjectPlan { + #[inline(always)] + unsafe fn next_slot(&mut self) -> Option<*mut u64> { + while self.next_prefix < 2 { + let slot = self.prefix[self.next_prefix]; + self.next_prefix += 1; + if !slot.is_null() { + return Some(slot); + } + } + let index = match &mut self.walk { + PayloadWalk::Word(word) => { + if *word == 0 { + return None; + } + let index = word.trailing_zeros() as usize; + *word &= *word - 1; + index + } + PayloadWalk::Range { next, count } => { + if *next >= *count { + return None; + } + *next += 1; + *next - 1 + } + PayloadWalk::Mask { + mask, + cursor, + count, + } => { + let index = mask.next_slot_at_or_after(*cursor, *count)?; + *cursor = index + 1; + index + } + }; + Some(self.payload.slot(index)) + } +} + +/// May this path scan the object at all? Decided before ANY side effect, from +/// the header and per-thread/process state alone. Past this point the path never +/// hands the object back to the generic walk. +#[inline(always)] +unsafe fn plain_object_admissible(header: *mut GcHeader) -> bool { + (*header).obj_type == GC_TYPE_OBJECT + && (*header).gc_flags & GC_FLAG_FORWARDED == 0 + // A full trace notes every carrier; a layout-scan trace counts every + // slot. Neither is reproduced here. + && !full_trace_active() + && !layout_scan_trace_active() + && matches!( + gc_type_rewrite_descriptor_kind(GC_TYPE_OBJECT), + GcRewriteDescriptorKind::Object + ) + && matches!( + gc_type_layout_slot_kind(GC_TYPE_OBJECT), + GcLayoutSlotKind::ObjectFields + ) +} + +/// `gc_child_slots`' ObjectFields arm and `visit_gc_layout_slot_descriptors`' +/// shape-keys bookkeeping, step for step and with the same side effects, as a +/// plan instead of an iterator. +#[inline(always)] +unsafe fn plain_object_plan(header: *mut GcHeader) -> PlainObjectPlan { + #[cfg(test)] + sabotage::note_plan_attempt(); + let user_ptr = (header as *mut u8).add(GC_HEADER_SIZE); + let obj = user_ptr as *mut crate::object::ObjectHeader; + let shape = crate::object::shapes::object_shape_record(obj); + let Some(range) = crate::object::gc_field_slot_range(obj, shape) else { + // `gc_child_slots` returns the EMPTY iterator: no shape, so no keys + // edge and no carrier note, no meta edge, no payload. + return PlainObjectPlan { + prefix: [std::ptr::null_mut(); 2], + next_prefix: 2, + payload: HeapSlotRange::new(std::ptr::null_mut(), 0), + walk: PayloadWalk::Word(0), + }; + }; + let meta = crate::object::gc_object_meta_slot(user_ptr as usize); + let count = range.slot_count(); + let walk = match heap_payload_slot_selection_from(header, range, shape) { + HeapPayloadSlotSelection::Empty | HeapPayloadSlotSelection::PointerFree { .. } => { + PayloadWalk::Word(0) + } + HeapPayloadSlotSelection::Masked { + mask: LayoutSlotMask::Inline(bits), + .. + } => { + // `take_inline_mask_word`'s limit, exactly. + let limit = count.min(64); + PayloadWalk::Word( + bits & if limit == 64 { + u64::MAX + } else { + (1u64 << limit) - 1 + }, + ) + } + HeapPayloadSlotSelection::Masked { + mask: LayoutSlotMask::AllPointers, + .. + } + | HeapPayloadSlotSelection::All { .. } => PayloadWalk::Range { next: 0, count }, + HeapPayloadSlotSelection::Masked { mask, .. } => PayloadWalk::Mask { + mask, + cursor: 0, + count, + }, + }; + #[cfg(test)] + let walk = sabotage::perturb(walk); + // The shape-keys edge. `full_trace_active` is false here, so only the + // old-carrier note applies — and when the note would change nothing, the + // generation probe that gates it is skipped. + #[cfg(test)] + let already_noted = sabotage::claiming_noted_carriers() + || crate::object::shapes::old_generation_carrier_already_noted(shape); + #[cfg(not(test))] + let already_noted = crate::object::shapes::old_generation_carrier_already_noted(shape); + if !already_noted && !crate::arena::pointer_in_nursery(user_ptr as usize) { + crate::object::shapes::note_old_generation_carrier(shape); + } + let keys_edge = crate::object::gc_shape_keys_edge_slot(shape); + // Visit order of the generic walk: prefix (none for objects), keys edge, + // meta, meta2 (none), payload. + PlainObjectPlan { + prefix: [ + keys_edge.unwrap_or(std::ptr::null_mut()), + meta.unwrap_or(std::ptr::null_mut()), + ], + next_prefix: 0, + payload: range, + walk, + } +} + +impl CopyingNurseryCollector { + /// Scan an ordinary object through its [`PlainObjectPlan`]. `false` means + /// nothing was done and the caller must take the generic walk. + #[inline(always)] + pub(super) unsafe fn scan_plain_object(&mut self, header: *mut GcHeader) -> bool { + if !plain_object_admissible(header) { + return false; + } + let user_ptr = (header as *mut u8).add(GC_HEADER_SIZE) as usize; + let mut changed = false; + // Asked lazily by the generic walk, at its first slot; both are + // properties of the header alone, so asking them up front is the same + // answer. + let weak = weak_holder_fact(header); + let remembering = ParentRemembering::of(header, self.skip_remembering); + // The generic walk's first edge, ahead of its kind arms: an explicit + // prototype in the residual registry. For `GC_TYPE_OBJECT` the + // per-owner half of the gate is conservatively `true`, so once the + // process latch is armed every object asks. + let mut residual_slots = 0usize; + if crate::object::prototype_chain::object_static_prototypes_maybe_nonempty() + && crate::object::prototype_chain::residual_prototype_owner_type(GC_TYPE_OBJECT) + && crate::object::prototype_chain::residual_entry_possible_for(header) + { + crate::object::prototype_chain::visit_object_static_prototype_slot_mut( + user_ptr, + |slot| { + residual_slots += 1; + let before = *slot; + self.visit_side_slot(slot, header, weak, remembering); + changed |= *slot != before; + }, + ); + } + let mut plan = plain_object_plan(header); + #[cfg(test)] + let cross_check = !sabotage::cross_check_disabled(); + #[cfg(all(not(test), debug_assertions))] + let cross_check = true; + #[cfg(any(test, debug_assertions))] + if cross_check { + assert_matches_generic_walk(header, &plan, residual_slots); + } + let _ = residual_slots; + while let Some(slot) = plan.next_slot() { + let before = *slot; + self.visit_slot_with_parent_facts( + GcMutableSlot::new(slot, None), + header, + weak, + remembering, + ); + changed |= *slot != before; + } + // The generic walk's last Object-arm edge, from the same side table. + crate::object::visit_overflow_field_slots_mut(user_ptr, |slot| { + let before = *slot; + self.visit_side_slot(slot, header, weak, remembering); + changed |= *slot != before; + }); + if changed { + run_gc_rewrite_hook(GC_TYPE_OBJECT, user_ptr); + } + true + } + + /// Side-table edges (residual prototype, overflow fields) are rare; keep + /// their visit out of the hot loop's code. + #[inline(never)] + unsafe fn visit_side_slot( + &mut self, + slot: *mut u64, + header: *mut GcHeader, + weak: bool, + remembering: ParentRemembering, + ) { + self.visit_slot_with_parent_facts( + GcMutableSlot::new(slot, None), + header, + weak, + remembering, + ); + } +} + +/// The drift check: the generic walk, run for its slot list only, must name +/// exactly the plan's slots followed by the overflow fields, after the +/// residual-prototype slots the scan already visited. Those are a stack +/// temporary of the registry's visitor, different on every call, so they are +/// matched by count, not address. +#[cfg(any(test, debug_assertions))] +unsafe fn assert_matches_generic_walk( + header: *mut GcHeader, + plan: &PlainObjectPlan, + residual_slots: usize, +) { + let mut generic = Vec::new(); + visit_gc_rewrite_slots(header, |slot| generic.push(slot.slot as usize)); + let mut replay = plan.clone(); + let mut expected = Vec::new(); + while let Some(slot) = replay.next_slot() { + expected.push(slot as usize); + } + let user_ptr = (header as *mut u8).add(GC_HEADER_SIZE) as usize; + crate::object::visit_overflow_field_slots_mut(user_ptr, |slot| expected.push(slot as usize)); + assert!( + generic.len() == residual_slots + expected.len() + && generic[residual_slots..] == expected[..], + "copying_object_scan: the plain-object plan enumerated different slots than the \ + generic walk for header {header:p} — a traced edge would be lost or invented \ + (generic {generic:x?}, residual {residual_slots}, plan {expected:x?})" + ); +} + +/// Test-only sabotage and liveness counters for the plain-object path. Witness: +/// `gc::tests::copying_object_scan`. +#[cfg(test)] +pub(crate) mod sabotage { + use std::cell::Cell; + + thread_local! { + static DROP_TOP_PAYLOAD_SLOT: Cell = const { Cell::new(false) }; + static CLAIM_NOTED_CARRIERS: Cell = const { Cell::new(false) }; + static NO_CROSS_CHECK: Cell = const { Cell::new(false) }; + static PLAN_ATTEMPTS: Cell = const { Cell::new(0) }; + } + + /// The generic-walk cross-check off. Its walk makes the old-carrier note + /// itself, so a test of whether THIS path makes the note must not run it. + pub(super) fn cross_check_disabled() -> bool { + NO_CROSS_CHECK.with(Cell::get) + } + + pub(crate) struct NoCrossCheck(bool); + + impl NoCrossCheck { + pub(crate) fn arm() -> Self { + Self(NO_CROSS_CHECK.with(|c| c.replace(true))) + } + } + + impl Drop for NoCrossCheck { + fn drop(&mut self) { + NO_CROSS_CHECK.with(|c| c.set(self.0)); + } + } + + /// Every shape reads as already noted, so the old-carrier note never runs. + pub(super) fn claiming_noted_carriers() -> bool { + CLAIM_NOTED_CARRIERS.with(Cell::get) + } + + pub(crate) struct ClaimNotedCarriers(bool); + + impl ClaimNotedCarriers { + pub(crate) fn arm() -> Self { + Self(CLAIM_NOTED_CARRIERS.with(|c| c.replace(true))) + } + } + + impl Drop for ClaimNotedCarriers { + fn drop(&mut self) { + CLAIM_NOTED_CARRIERS.with(|c| c.set(self.0)); + } + } + + pub(super) fn note_plan_attempt() { + PLAN_ATTEMPTS.with(|c| c.set(c.get().wrapping_add(1))); + } + + /// Objects that reached plan construction on this thread. + pub(crate) fn plan_attempts() -> u32 { + PLAN_ATTEMPTS.with(Cell::get) + } + + /// Forget the highest payload slot — the one a limit mistake loses. + pub(super) fn perturb(walk: super::PayloadWalk) -> super::PayloadWalk { + if !DROP_TOP_PAYLOAD_SLOT.with(Cell::get) { + return walk; + } + match walk { + super::PayloadWalk::Word(word) if word != 0 => { + super::PayloadWalk::Word(word & !(1u64 << (63 - word.leading_zeros()))) + } + super::PayloadWalk::Range { next, count } if count > next => { + super::PayloadWalk::Range { + next, + count: count - 1, + } + } + other => other, + } + } + + pub(crate) struct DropTopPayloadSlot(bool); + + impl DropTopPayloadSlot { + pub(crate) fn arm() -> Self { + Self(DROP_TOP_PAYLOAD_SLOT.with(|c| c.replace(true))) + } + } + + impl Drop for DropTopPayloadSlot { + fn drop(&mut self) { + DROP_TOP_PAYLOAD_SLOT.with(|c| c.set(self.0)); + } + } +} diff --git a/crates/perry-runtime/src/gc/copying_parent_facts.rs b/crates/perry-runtime/src/gc/copying_parent_facts.rs index 98037f62f6..3416b09994 100644 --- a/crates/perry-runtime/src/gc/copying_parent_facts.rs +++ b/crates/perry-runtime/src/gc/copying_parent_facts.rs @@ -27,6 +27,62 @@ pub(super) unsafe fn weak_holder_fact(header: *mut GcHeader) -> bool { crate::weakref::is_weak_holder_header(header) } +/// The parent's half of `barrier_parent_needs_remembering`, decided once per +/// traced object instead of once per slot. +/// +/// `barrier_parent_needs_remembering(parent, external)` is +/// `Old(parent) || (external && malloc_gc_parent_addr(parent))`. Both parent +/// terms read only the parent's address and header, which do not change while +/// that object's slots are visited (the visit moves CHILDREN), so the per-slot +/// question reduces to this three-way answer plus, for a malloc parent only, +/// the slot's own generation. `skip_remembering` — a per-cycle proof that no +/// entry can be created — folds into `Never`, so a whole-block promoting cycle +/// classifies neither the parent nor any slot. Before this every slot paid a +/// page-map classification of its own address for an answer nothing read. +/// +/// Witness: `gc::tests::copy_slot_hoists::the_per_object_remembering_fact_*`, +/// which checks it against `barrier_parent_needs_remembering` itself, with a +/// sabotaged twin that must disagree. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum ParentRemembering { + Never, + Always, + ExternalSlotsOnly, +} + +impl ParentRemembering { + #[inline] + pub(super) unsafe fn of(parent_header: *mut GcHeader, skip_remembering: bool) -> Self { + if parent_header.is_null() || skip_remembering { + return Self::Never; + } + let parent = (parent_header as *mut u8).add(GC_HEADER_SIZE) as usize; + #[cfg(test)] + if copy_hoist_sabotage::forgetting_remembering() { + return Self::Never; + } + if matches!( + crate::arena::classify_heap_generation(parent), + crate::arena::HeapGeneration::Old + ) { + Self::Always + } else if super::barrier::malloc_gc_parent_addr(parent) { + Self::ExternalSlotsOnly + } else { + Self::Never + } + } + + #[inline(always)] + pub(super) fn for_slot(self, slot: GcMutableSlot) -> bool { + match self { + Self::Never => false, + Self::Always => true, + Self::ExternalSlotsOnly => slot.external(), + } + } +} + /// Test-only sabotage for [`weak_holder_fact`]: forgetting the per-object fact /// must change what the collector does, or the hoist is documentation /// (CLAUDE.md, a gate that cannot fail). Its witness is @@ -37,6 +93,7 @@ pub(crate) mod copy_hoist_sabotage { thread_local! { static FORGET_WEAK: Cell = const { Cell::new(false) }; + static FORGET_REMEMBERING: Cell = const { Cell::new(false) }; } #[inline] @@ -44,6 +101,26 @@ pub(crate) mod copy_hoist_sabotage { FORGET_WEAK.with(Cell::get) } + /// `ParentRemembering::of` answers `Never` for every parent. + #[inline] + pub(crate) fn forgetting_remembering() -> bool { + FORGET_REMEMBERING.with(Cell::get) + } + + pub(crate) struct RememberingGuard(bool); + + impl RememberingGuard { + pub(crate) fn arm() -> Self { + Self(FORGET_REMEMBERING.with(|s| s.replace(true))) + } + } + + impl Drop for RememberingGuard { + fn drop(&mut self) { + FORGET_REMEMBERING.with(|s| s.set(self.0)); + } + } + pub(crate) struct WeakGuard(bool); impl WeakGuard { @@ -150,7 +227,30 @@ impl CopyingNurseryCollector { return None; } let addr = bits as usize; - let ptr = self.ptrs.classify(addr)?; + // The memo holds only an address that CLASSIFIED this cycle, and a + // classification cannot change within a cycle: its page range stays + // registered until the from-space reset after the last trace, and the + // header fields `plausible_gc_header` reads are not ones forwarding + // rewrites. So a raw word naming the memo is validated already — the + // per-object shape `keys` word hits this on every shaped receiver + // (#11549). Test and debug builds re-derive the premise. + if let Some(new_addr) = self.memo_hit(addr) { + #[cfg(test)] + if copy_decode_sabotage::forgetting(copy_decode_sabotage::RAW_MARK) { + return None; + } + #[cfg(any(test, debug_assertions))] + assert!( + self.ptrs.classify(addr).is_some(), + "a memoized raw address stopped classifying mid-cycle: {addr:#x}" + ); + return Some(( + new_addr, + (new_addr != addr).then_some(new_addr as u64), + true, + )); + } + let ptr = self.ptrs.classify_inline(addr)?; #[cfg(test)] if copy_decode_sabotage::forgetting(copy_decode_sabotage::RAW_MARK) { return None; @@ -189,6 +289,58 @@ impl CopyingNurseryCollector { parent_header: *mut GcHeader, weak_holder: bool, external: bool, + ) { + let skip_remembering = self.skip_remembering; + self.visit_slot_core( + slot, + parent_header, + weak_holder, + move || { + !parent_header.is_null() + && !skip_remembering + && barrier_parent_needs_remembering( + (parent_header as *mut u8).add(GC_HEADER_SIZE) as usize, + external, + ) + }, + move || external, + ); + } + + /// The drain's form of [`Self::visit_slot_with_weak_fact`]: the parent's + /// remembering question answered ONCE per traced object + /// ([`ParentRemembering`]), and the slot's own generation classified only + /// when an answer actually depends on it — never on a whole-block promoting + /// cycle, where `skip_remembering` settles every slot up front. + #[inline(always)] + pub(super) unsafe fn visit_slot_with_parent_facts( + &mut self, + slot: GcMutableSlot, + parent_header: *mut GcHeader, + weak_holder: bool, + remembering: ParentRemembering, + ) { + self.visit_slot_core( + slot.slot, + parent_header, + weak_holder, + move || remembering.for_slot(slot), + move || slot.external(), + ); + } + + /// The one slot visit. `remembering` and `external` are asked lazily, in + /// the order the visit has always asked them: remembering before the + /// child is decoded, the slot's generation only for a child that needs + /// tracking. + #[inline(always)] + unsafe fn visit_slot_core( + &mut self, + slot: *mut u64, + parent_header: *mut GcHeader, + weak_holder: bool, + remembering: impl FnOnce() -> bool, + external: impl FnOnce() -> bool, ) { if slot.is_null() { return; @@ -214,12 +366,7 @@ impl CopyingNurseryCollector { // Asked BEFORE the visit: it reads only the parent and the slot's own // address, never the child. Asked after, the optimizer duplicated the // call into both decode arms and then stopped inlining it. - let remembering = !parent_header.is_null() - && !self.skip_remembering - && barrier_parent_needs_remembering( - (parent_header as *mut u8).add(GC_HEADER_SIZE) as usize, - external, - ); + let remembering = remembering(); let visited = self.visit_value_bits_child(*slot); if let Some((_, Some(new_bits), _)) = visited { *slot = new_bits; @@ -254,7 +401,7 @@ impl CopyingNurseryCollector { // CopyingPointerKind::Malloc) but the NEXT minor's malloc sweep // needs the edge again. if crate::gc::barrier::remembered_child_needs_tracking(child_addr) { - self.sticky.remember_slot(parent_header, slot, external); + self.sticky.remember_slot(parent_header, slot, external()); } } } diff --git a/crates/perry-runtime/src/gc/copying_pointer_set.rs b/crates/perry-runtime/src/gc/copying_pointer_set.rs index b4c25903b6..faced1bdca 100644 --- a/crates/perry-runtime/src/gc/copying_pointer_set.rs +++ b/crates/perry-runtime/src/gc/copying_pointer_set.rs @@ -90,6 +90,21 @@ impl CopyingPointerSet { #[inline] pub(super) fn classify_arena(&self, addr: usize) -> Option { + self.classify_arena_inline(addr) + } + + /// [`Self::classify`] inlined into its caller: the copying drain's mark + /// (`CopyingNurseryCollector::mark_addr`), which classifies once per + /// visited reference. Out of line, the call frame alone was a fifth of the + /// classification. Same answer as `classify`, by construction. + #[inline(always)] + pub(super) fn classify_inline(&self, addr: usize) -> Option { + self.classify_arena_inline(addr) + .or_else(|| self.classify_malloc(addr)) + } + + #[inline(always)] + fn classify_arena_inline(&self, addr: usize) -> Option { if addr < GC_HEADER_SIZE { return None; } diff --git a/crates/perry-runtime/src/gc/hot_tls.rs b/crates/perry-runtime/src/gc/hot_tls.rs index ca124b64f4..b1024f1aaa 100644 --- a/crates/perry-runtime/src/gc/hot_tls.rs +++ b/crates/perry-runtime/src/gc/hot_tls.rs @@ -75,7 +75,7 @@ pub(super) fn hot_incremental_mark_minor_only() -> &'static Cell { type SlotMaskMap = crate::fast_hash::PtrHashMap; type TypedLayoutMap = crate::fast_hash::PtrHashMap; -type ShapeLayoutMap = crate::fast_hash::PtrHashMap>; +type ShapeLayoutMap = super::layout::ShapeLayoutTable; /// Address of this thread's `LAYOUT_SLOT_MASKS`. pub(crate) fn layout_slot_masks_hot_addr() -> *mut u8 { diff --git a/crates/perry-runtime/src/gc/layout.rs b/crates/perry-runtime/src/gc/layout.rs index 5efbc02208..27710496b8 100644 --- a/crates/perry-runtime/src/gc/layout.rs +++ b/crates/perry-runtime/src/gc/layout.rs @@ -118,12 +118,14 @@ pub(super) fn clear_typed_layout_intact_for_user(user_ptr: usize) { } } +pub(in crate::gc) mod shape_layout_table; mod slot_mask; #[cfg(test)] mod test_accessors; mod transfer; mod typed_shape; +pub(in crate::gc) use shape_layout_table::{ShapeLayoutTable, ShapeMaskMemo}; pub(in crate::gc) use slot_mask::LayoutSlotMask; #[cfg(test)] pub(crate) use test_accessors::{ @@ -185,8 +187,9 @@ thread_local! { // array cannot stale this index. Nothing to prune on object death (entries are // per-shape, shared). thread_local! { - pub(in crate::gc) static SHAPE_LAYOUTS: RefCell>> = - RefCell::new(crate::fast_hash::new_ptr_hash_map()); + pub(in crate::gc) static SHAPE_LAYOUTS: RefCell< + ShapeMaskMemo>>, + > = RefCell::new(ShapeMaskMemo::new()); } fn shape_layout_keyed_enabled() -> bool { @@ -337,7 +340,17 @@ unsafe fn shape_shared_pointer_mask_from( if (*header)._reserved & GC_OBJ_TYPED_LAYOUT_INTACT == 0 { return None; } - with_shape_shared_descriptor_from(user_ptr, shape, |d| d.pointer_mask.clone()) + // `with_shape_shared_descriptor_from(.., |d| d.pointer_mask.clone())`, + // through the table's exact one-entry memo (`shape_layout_table.rs`). + let shape_id = + crate::object::shapes::object_shape_stamp(user_ptr as *const crate::object::ObjectHeader); + if shape_id == 0 { + return None; + } + let field_count = shape.map_or(0, |shape| shape.live_inline_slot_count() as usize); + hot_shape_layouts() + .borrow() + .shared_pointer_mask(shape_id, field_count) } /// Install `descriptor` as the canonical layout for `shape_id` and set the @@ -1587,6 +1600,7 @@ impl HeapChildSlotIterator { /// caller already resolved (#8122). The payload-mask selection reuses it /// instead of probing the shape table, and it is retained on the iterator /// for the slot visitor. + #[inline(always)] pub(super) fn new_object( header: *mut GcHeader, prefix_slot: Option<*mut u64>, @@ -1743,6 +1757,7 @@ pub(super) unsafe fn heap_payload_slot_selection( /// [`heap_payload_slot_selection`] for an ObjectFields receiver whose shape /// record the caller already resolved (#8122): the shared-shape /// pointer-mask lookup reuses it instead of probing the shape table twice. +#[inline(always)] pub(super) unsafe fn heap_payload_slot_selection_from( header: *mut GcHeader, payload: HeapSlotRange, @@ -1753,7 +1768,7 @@ pub(super) unsafe fn heap_payload_slot_selection_from( }) } -#[inline] +#[inline(always)] unsafe fn heap_payload_slot_selection_impl( header: *mut GcHeader, payload: HeapSlotRange, @@ -1815,6 +1830,9 @@ unsafe fn heap_payload_slot_selection_impl( /// #10362: every arm returns the iterator it builds, never through an `Option` /// combinator whose temporary is copied out — a per-object memmove per GC walk. +/// Inlined (#11549): the descriptor walk has two instantiations now, and out of +/// line the iterator came back by memory on every traced object. +#[inline(always)] pub(super) unsafe fn gc_child_slots(header: *mut GcHeader) -> HeapChildSlotIterator { if header.is_null() || (*header).gc_flags & GC_FLAG_FORWARDED != 0 { return HeapChildSlotIterator::empty(); @@ -1942,6 +1960,13 @@ pub(super) enum GcMutableSlotDescriptor { impl GcMutableSlotDescriptor { pub(super) unsafe fn visit_slots(self, visit: &mut dyn FnMut(GcMutableSlot)) { + self.visit_slots_inline(visit) + } + + /// [`Self::visit_slots`] monomorphized for one visitor, so the copying + /// minor's per-slot closure inlines instead of taking a dyn call per slot. + #[inline(always)] + pub(super) unsafe fn visit_slots_inline(self, visit: &mut F) { match self { GcMutableSlotDescriptor::Slot(slot) => visit(slot), GcMutableSlotDescriptor::Range { range, layout_kind } => { diff --git a/crates/perry-runtime/src/gc/layout/shape_layout_table.rs b/crates/perry-runtime/src/gc/layout/shape_layout_table.rs new file mode 100644 index 0000000000..a6833d6484 --- /dev/null +++ b/crates/perry-runtime/src/gc/layout/shape_layout_table.rs @@ -0,0 +1,133 @@ +//! `SHAPE_LAYOUTS` with a one-entry memo of its last pointer-mask answer +//! (#11549). +//! +//! Every traced class instance asks the table for its shape's shared pointer +//! mask: a `RefCell` borrow, a hash probe and a mask clone, ~50 instructions +//! per object in a copying minor over a binary tree — for the SAME shape, object +//! after object. The memo replays the last answer instead. +//! +//! It is exact, not a cache that can go stale, because it is cleared by +//! construction: the map is reachable mutably ONLY through `DerefMut`, and +//! `deref_mut` empties the memo before handing the map out. Insert, poison +//! (`Some` → `None`), `entry`, `iter_mut`, `clear` — every write path goes +//! through it. Replacing the whole table replaces the memo with it. So a +//! memoized answer is always the answer the map would give now. +//! `gc::tests::shape_layout_table` pins that, with a sabotaged twin that keeps +//! the memo across a write. + +use super::{LayoutSlotMask, TypedLayoutDescriptor}; +use std::cell::Cell; + +type ShapeLayoutMap = crate::fast_hash::PtrHashMap>; + +/// The empty memo. ShapeId 0 is "unstamped" and never looked up here. +const NO_MEMO: (u32, usize, u64) = (0, 0, 0); + +/// Generic over the map only so that `SHAPE_LAYOUTS`' declaration still names +/// its map type: `scripts/registry_lifetime_check.py` finds registries by the +/// container named in the declaration, and must keep seeing this one. +pub(in crate::gc) struct ShapeMaskMemo { + map: M, + /// `(shape_id, descriptor slot_count, inline pointer mask)` of the last + /// `Some` descriptor with a one-word pointer mask that + /// [`Self::shared_pointer_mask`] read. + memo: Cell<(u32, usize, u64)>, +} + +/// The table `SHAPE_LAYOUTS` holds. +pub(in crate::gc) type ShapeLayoutTable = ShapeMaskMemo; + +impl ShapeMaskMemo { + pub(in crate::gc) fn new() -> Self { + Self { + map: crate::fast_hash::new_ptr_hash_map(), + memo: Cell::new(NO_MEMO), + } + } + + /// `shape_id`'s shared pointer mask for a receiver whose live inline bound + /// is `field_count`: exactly + /// `map.get(&shape_id)?.as_ref()` filtered to `slot_count == field_count`, + /// with the mask cloned out. + #[inline] + pub(in crate::gc) fn shared_pointer_mask( + &self, + shape_id: u32, + field_count: usize, + ) -> Option { + let (memo_id, memo_count, memo_bits) = self.memo.get(); + if memo_id == shape_id && shape_id != 0 { + return (memo_count == field_count).then_some(LayoutSlotMask::Inline(memo_bits)); + } + let desc = self.map.get(&shape_id)?.as_ref()?; + if let LayoutSlotMask::Inline(bits) = desc.pointer_mask { + if shape_id != 0 { + self.memo.set((shape_id, desc.slot_count, bits)); + } + } + if desc.slot_count != field_count { + return None; + } + Some(desc.pointer_mask.clone()) + } +} + +impl std::ops::Deref for ShapeMaskMemo { + type Target = M; + + #[inline] + fn deref(&self) -> &M { + &self.map + } +} + +impl std::ops::DerefMut for ShapeMaskMemo { + /// The ONLY way to the map mutably, and it forgets the memo first. + #[inline] + fn deref_mut(&mut self) -> &mut M { + #[cfg(test)] + if sabotage::keeping_memo() { + return &mut self.map; + } + self.memo.set(NO_MEMO); + &mut self.map + } +} + +/// Test-only sabotage: a write that keeps the memo. Witness: +/// `gc::tests::shape_layout_table`. +#[cfg(test)] +pub(crate) mod sabotage { + use std::cell::Cell; + + thread_local! { + static KEEP_MEMO: Cell = const { Cell::new(false) }; + } + + pub(super) fn keeping_memo() -> bool { + KEEP_MEMO.with(Cell::get) + } + + pub(crate) struct KeepMemo(bool); + + impl KeepMemo { + pub(crate) fn arm() -> Self { + Self(KEEP_MEMO.with(|c| c.replace(true))) + } + } + + impl Drop for KeepMemo { + fn drop(&mut self) { + KEEP_MEMO.with(|c| c.set(self.0)); + } + } +} + +#[cfg(test)] +pub(crate) fn test_descriptor(slot_count: usize, pointer_bits: u64) -> TypedLayoutDescriptor { + TypedLayoutDescriptor { + slot_count, + raw_f64_mask: LayoutSlotMask::Inline(0), + pointer_mask: LayoutSlotMask::Inline(pointer_bits), + } +} diff --git a/crates/perry-runtime/src/gc/layout_slot_visit.rs b/crates/perry-runtime/src/gc/layout_slot_visit.rs index 3bc0157204..68dddbb181 100644 --- a/crates/perry-runtime/src/gc/layout_slot_visit.rs +++ b/crates/perry-runtime/src/gc/layout_slot_visit.rs @@ -79,6 +79,22 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors( header: *mut GcHeader, visit: &mut dyn FnMut(GcMutableSlotDescriptor), ) { + visit_gc_layout_slot_descriptors_inline(header, visit); +} + +/// The ONE body of [`visit_gc_layout_slot_descriptors`], generic over the +/// visitor. Every caller but the copying minor's drain goes through the `dyn` +/// wrapper above (one copy of this body); the drain instantiates it directly +/// (`visit_gc_rewrite_slots_inline`) so its per-slot closure inlines instead of +/// paying two indirect calls per visited slot. Same enumeration either way: +/// there is no second copy of the slot logic to drift. +#[inline(always)] +pub(super) unsafe fn visit_gc_layout_slot_descriptors_inline( + header: *mut GcHeader, + visit: &mut F, +) where + F: FnMut(GcMutableSlotDescriptor) + ?Sized, +{ let mut child_slots = gc_child_slots(header); // #8213: drained async box cells are weak registry entries during a full // trace. A closure proven live by the mark set is their owner, so enumerate @@ -231,9 +247,29 @@ impl GcMutableSlotDescriptor { } pub(super) unsafe fn visit_gc_rewrite_slot_descriptors( + header: *mut GcHeader, + visit: impl FnMut(GcMutableSlotDescriptor), +) { + visit_gc_rewrite_slot_descriptors_with::(header, visit); +} + +/// The one body of [`visit_gc_rewrite_slot_descriptors`]. `INLINE_LAYOUT` +/// selects only HOW the layout-descriptor walk is called — through the shared +/// `dyn` copy, or instantiated for this visitor — never what it enumerates. +#[inline(always)] +unsafe fn visit_gc_rewrite_slot_descriptors_with( header: *mut GcHeader, mut visit: impl FnMut(GcMutableSlotDescriptor), ) { + macro_rules! layout_descriptors { + () => { + if INLINE_LAYOUT { + visit_gc_layout_slot_descriptors_inline(header, &mut visit) + } else { + visit_gc_layout_slot_descriptors(header, &mut visit) + } + }; + } if header.is_null() || (*header).gc_flags & GC_FLAG_FORWARDED != 0 { return; } @@ -263,7 +299,7 @@ pub(super) unsafe fn visit_gc_rewrite_slot_descriptors( } match gc_type_rewrite_descriptor_kind((*header).obj_type) { GcRewriteDescriptorKind::Array => { - visit_gc_layout_slot_descriptors(header, &mut visit); + layout_descriptors!(); // #10166 (brief 4): an array's named properties live in reserve // slots in front of logical element 0 (`array/named_props.rs`): // a pairs pointer, or inline exec-result values. Those words sit @@ -286,17 +322,17 @@ pub(super) unsafe fn visit_gc_rewrite_slot_descriptors( // already emits it — no explicit `gc_object_meta_slot` visit // here, or the rewrite pass would hand the same slot to the // visitor twice and double-count in verification statistics. - visit_gc_layout_slot_descriptors(header, &mut visit); + layout_descriptors!(); crate::object::visit_overflow_field_slots_mut(user_ptr as usize, |slot| { visit(fixed_slot(slot)); }); } GcRewriteDescriptorKind::RegExp => { - visit_gc_layout_slot_descriptors(header, &mut visit); + layout_descriptors!(); } GcRewriteDescriptorKind::Closure => { // Captures and the own-property bag edge (`ClosureCaptures`). - visit_gc_layout_slot_descriptors(header, &mut visit); + layout_descriptors!(); } GcRewriteDescriptorKind::Promise => { let promise = user_ptr as *mut crate::promise::Promise; @@ -501,6 +537,20 @@ pub(super) unsafe fn visit_gc_rewrite_slots( }); } +/// [`visit_gc_rewrite_slots`] with the whole enumeration instantiated for +/// `visit`: the copying minor's drain, where the two per-slot indirect calls +/// (descriptor visitor, slot visitor) were a measured share of the per-object +/// trace cost. Enumerates exactly the same slots, in the same order. +#[inline(always)] +pub(super) unsafe fn visit_gc_rewrite_slots_inline( + header: *mut GcHeader, + mut visit: impl FnMut(GcMutableSlot), +) { + visit_gc_rewrite_slot_descriptors_with::(header, |descriptor| unsafe { + descriptor.visit_slots_inline(&mut visit); + }); +} + /// Test-only sabotage for the inline mask walk /// ([`HeapChildSlotIterator::take_inline_mask_word`]): a fast path that /// enumerates a DIFFERENT set than the iterator it replaces must be caught, so diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 899590e2c5..d45f023f0a 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -168,6 +168,7 @@ pub(crate) mod prefetch; mod copying; mod copying_first_cycle; +mod copying_object_scan; mod copying_parent_facts; mod copying_phase; mod copying_pointer_set; diff --git a/crates/perry-runtime/src/gc/tests/copy_slot_hoists.rs b/crates/perry-runtime/src/gc/tests/copy_slot_hoists.rs index 9c295b7fe1..d92b208213 100644 --- a/crates/perry-runtime/src/gc/tests/copy_slot_hoists.rs +++ b/crates/perry-runtime/src/gc/tests/copy_slot_hoists.rs @@ -5,15 +5,17 @@ //! hoisted value back, and paired with a sabotaged twin that forgets the fact, //! so the hoist is shown to be load-bearing rather than merely present. //! -//! The parent's old-generation fact is deliberately NOT hoisted, so there is no -//! test for it here. No sabotage of that hoist could be made to fail: sticky -//! dirty-page coverage carries an old→young edge independently of the -//! remembered-set re-insertion the fact controls. A future hoist of it needs -//! its own witness first (#10388). +//! The parent's remembering fact (`ParentRemembering`) is hoisted too, but its +//! witness is NOT a collection: no sabotage of it could be made to fail that +//! way, because sticky dirty-page coverage carries an old→young edge +//! independently of the remembered-set re-insertion the fact controls +//! (#10388). It is pinned instead against the predicate it replaces, +//! `barrier_parent_needs_remembering`, over every parent kind that predicate +//! distinguishes — with a sabotaged twin that must disagree. use super::super::*; use super::support::*; -use crate::gc::copying_parent_facts::copy_hoist_sabotage; +use crate::gc::copying_parent_facts::{copy_hoist_sabotage, ParentRemembering}; /// A young target reachable ONLY through a rooted `WeakRef`'s weak slot. /// A copying minor must not evacuate through that slot, so the target dies @@ -73,3 +75,74 @@ fn sabotaged_weak_holder_fact_evacuates_through_the_weak_slot() { treated as strong and the target survives the minor" ); } + +/// Every (parent, slot) pair on which the per-object remembering fact and the +/// per-slot predicate it replaces disagree, plus which answers the parents +/// produced — so the caller can require all three arms were exercised. +fn remembering_fact_disagreements(sabotaged: bool) -> (usize, Vec) { + std::thread::spawn(move || { + let _guard = CopyingNurseryTestGuard::new(1); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _scan = ConservativeScanDisabledGuard::new(); + let (old, old_fields) = unsafe { alloc_old_test_object(2) }; + let (young, young_fields) = unsafe { alloc_nursery_test_object(2) }; + let malloc = alloc_tracked_test_symbol() as *mut u8; + let header = |user: *mut u8| unsafe { header_from_user_ptr(user) as *mut GcHeader }; + let parents = [ + header(old as *mut u8), + header(young as *mut u8), + header(malloc), + ]; + // An inline slot of each generation, and a slot outside every arena + // block (the malloc object's own payload word): the three answers + // `external()` can give a slot. + let slots = [old_fields, young_fields, malloc as *mut u64]; + let _sabotage = sabotaged.then(copy_hoist_sabotage::RememberingGuard::arm); + let mut disagreements = 0usize; + let mut answers = Vec::new(); + for &parent in &parents { + let parent_user = unsafe { (parent as *mut u8).add(GC_HEADER_SIZE) } as usize; + let fact = unsafe { ParentRemembering::of(parent, false) }; + answers.push(fact); + let skipped = unsafe { ParentRemembering::of(parent, true) }; + for &slot in &slots { + let slot = GcMutableSlot::new(slot, None); + let expected = crate::gc::barrier::barrier_parent_needs_remembering( + parent_user, + slot.external(), + ); + disagreements += usize::from(fact.for_slot(slot) != expected); + // `skip_remembering` is a proof that nothing is remembered. + disagreements += usize::from(skipped.for_slot(slot)); + } + } + (disagreements, answers) + }) + .join() + .expect("remembering-fact test thread must not panic") +} + +#[test] +fn the_per_object_remembering_fact_agrees_with_the_per_slot_predicate() { + let (disagreements, answers) = remembering_fact_disagreements(false); + assert_eq!( + answers, + vec![ + ParentRemembering::Always, + ParentRemembering::Never, + ParentRemembering::ExternalSlotsOnly, + ], + "premise: an old, a young and a malloc parent exercise all three answers" + ); + assert_eq!(disagreements, 0); +} + +#[test] +fn the_per_object_remembering_fact_sabotaged_disagrees_with_the_per_slot_predicate() { + let (disagreements, _) = remembering_fact_disagreements(true); + assert!( + disagreements > 0, + "a fact that forgets every parent must disagree with the predicate on \ + the old parent's slots" + ); +} diff --git a/crates/perry-runtime/src/gc/tests/copying_object_scan.rs b/crates/perry-runtime/src/gc/tests/copying_object_scan.rs new file mode 100644 index 0000000000..69c6656de7 --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/copying_object_scan.rs @@ -0,0 +1,198 @@ +//! The copying drain's plain-object scan (`gc/copying_object_scan.rs`) is a +//! second enumeration of an ordinary object's slots, so it is pinned three ways: +//! it is actually TAKEN by a minor over ordinary objects (a fast path nothing +//! reaches would make every other test here vacuous), every child it visits is +//! evacuated and rewritten, and a plan that drops a slot is REFUSED by the +//! generic-walk cross-check that runs in test and debug-assertion builds. + +use super::super::*; +use super::support::*; +use crate::gc::copying_object_scan::sabotage; + +fn string_bytes(addr: usize) -> Vec { + unsafe { + let s = addr as *const crate::StringHeader; + std::slice::from_raw_parts(crate::string::string_data(s), (*s).byte_len as usize).to_vec() + } +} + +const FIELDS: usize = 3; + +/// A rooted young object whose every field holds a young string, then a minor. +/// `Ok((plan attempts, every child moved and intact))`; `Err` is the +/// collection thread's panic message. +fn minor_over_a_plain_object(sabotaged: bool) -> Result<(u32, bool), String> { + minor_over_a_plain_object_with(sabotaged, false) +} + +/// `residual_armed`: first give an (old) array an explicit prototype, which +/// arms the residual-prototype registry's process latch — after which the +/// generic walk asks the registry for EVERY ordinary object, and so must this +/// path, without declining. +fn minor_over_a_plain_object_with( + sabotaged: bool, + residual_armed: bool, +) -> Result<(u32, bool), String> { + std::thread::spawn(move || { + let _guard = CopyingNurseryTestGuard::new(1); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _scan = ConservativeScanDisabledGuard::new(); + let _roots = ShadowAndGlobalRootResetGuard; + if residual_armed { + let array = unsafe { alloc_old_test_array(1).0 }; + let proto = unsafe { alloc_old_test_object(0).0 } as usize; + crate::object::prototype_chain::object_set_static_prototype( + array as usize, + ptr_bits(proto), + ); + assert!( + crate::object::prototype_chain::object_static_prototypes_maybe_nonempty(), + "premise: the residual registry latch is armed" + ); + } + let (parent, fields) = unsafe { alloc_nursery_test_object(FIELDS as u32) }; + let mut children = Vec::new(); + for i in 0..FIELDS { + let child = young_leaf(); + unsafe { *fields.add(i) = string_bits(child) }; + children.push((child, string_bytes(child))); + } + js_shadow_slot_set(0, ptr_bits(parent as usize)); + let before = sabotage::plan_attempts(); + { + let _sabotage = sabotaged.then(sabotage::DropTopPayloadSlot::arm); + let _ = gc_collect_minor(); + } + let attempts = sabotage::plan_attempts() - before; + let parent_after = (js_shadow_slot_get(0) & POINTER_MASK) as usize; + let fields_after = unsafe { + (parent_after as *const u8).add(std::mem::size_of::()) + as *const u64 + }; + let intact = children.iter().enumerate().all(|(i, (old, bytes))| { + let word = unsafe { *fields_after.add(i) }; + let now = (word & POINTER_MASK) as usize; + // Checked before any read through `now`: a stale word names from-space. + now != *old && string_bytes(now) == *bytes + }); + (attempts, intact) + }) + .join() + .map_err(|payload| { + payload + .downcast_ref::() + .cloned() + .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string())) + .unwrap_or_default() + }) +} + +#[test] +fn a_minor_scans_a_plain_object_through_its_plan_and_moves_every_child() { + let (attempts, intact) = minor_over_a_plain_object(false).expect("minor must not panic"); + assert!( + attempts > 0, + "premise: the minor must have taken the plain-object path at least once" + ); + assert!( + intact, + "every field's young child must be evacuated and its word rewritten" + ); +} + +#[test] +fn an_armed_residual_prototype_registry_does_not_turn_the_plain_object_path_away() { + // Recording an ARRAY's prototype also latches the process-wide array + // prototype flags; restore them, as `dyn_eval/tests.rs`' + // `ArrayPrototypeLatchGuard` does, or later tests inherit them. + let _lock = crate::typed_feedback::typed_feedback_test_lock(); + let latch = crate::object::prototype_chain::array_static_proto_recorded(); + let invalidated = crate::array::PERRY_ARRAY_INDEX_FAST_PATH_INVALIDATED + .load(std::sync::atomic::Ordering::Relaxed); + let outcome = minor_over_a_plain_object_with(false, true); + crate::object::prototype_chain::test_swap_array_static_proto_recorded(latch); + crate::array::test_swap_array_index_fast_path_invalidated(invalidated); + let (attempts, intact) = outcome.expect("minor must not panic"); + assert!( + attempts > 0, + "the plain-object path must still scan with the latch armed" + ); + assert!( + intact, + "every field's young child must be evacuated and its word rewritten" + ); +} + +#[test] +fn a_plan_that_drops_a_payload_slot_is_refused_by_the_generic_walk_cross_check() { + let outcome = minor_over_a_plain_object(true); + assert!( + matches!(&outcome, Err(message) if message.contains("copying_object_scan")), + "a plan missing the top payload slot must be caught by the cross-check; got {outcome:?}" + ); +} + +/// A young object whose shape has never had an old carrier, promoted IN PLACE +/// by a traced minor — so it is scanned at an old address, and (the malloc +/// registry being empty) the cycle skips the remembered-set rebuild whose +/// generic walk would make the note itself. +/// Returns whether its shape record was noted as old-carried, `(before, +/// after)`: the note the plain-object path skips only when it would change +/// nothing. +fn promoted_receiver_notes_its_shape(sabotaged: bool) -> (bool, bool) { + std::thread::spawn(move || { + let _guard = CopyingNurseryTestGuard::new(1); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let _promote = InPlacePromotionTestGuard::enabled(1000); + let _scan = ConservativeScanDisabledGuard::new(); + let (parent, fields) = unsafe { alloc_nursery_test_object(2) }; + unsafe { *fields = string_bits(young_leaf()) }; + js_shadow_slot_set(0, ptr_bits(parent as usize)); + let noted = |obj: usize| unsafe { + crate::object::shapes::old_generation_carrier_already_noted( + crate::object::shapes::object_shape_record( + obj as *const crate::object::ObjectHeader, + ), + ) + }; + let before = noted(parent as usize); + { + let _no_cross_check = sabotage::NoCrossCheck::arm(); + let _sabotage = sabotaged.then(sabotage::ClaimNotedCarriers::arm); + let attempts = sabotage::plan_attempts(); + // Not `collect_minor_trace`: a traced cycle arms the layout-scan + // trace, which the plain-object path declines. + let _ = gc_collect_minor(); + assert!( + sabotage::plan_attempts() > attempts, + "premise: the plain-object path scanned the promoted receiver" + ); + } + assert!( + (js_shadow_slot_get(0) & POINTER_MASK) as usize == parent as usize + && crate::arena::pointer_in_old_gen(parent as usize), + "premise: the receiver was promoted where it stood" + ); + (before, noted(parent as usize)) + }) + .join() + .expect("carrier-note test thread must not panic") +} + +#[test] +fn a_promoted_receiver_notes_its_shape_as_old_carried() { + assert_eq!( + promoted_receiver_notes_its_shape(false), + (false, true), + "a fresh shape starts un-noted and its promoted carrier must note it" + ); +} + +#[test] +fn claiming_every_shape_already_noted_loses_the_old_carrier_note() { + assert_eq!( + promoted_receiver_notes_its_shape(true), + (false, false), + "with the skip claiming every shape noted, the note never runs" + ); +} diff --git a/crates/perry-runtime/src/gc/tests/mod.rs b/crates/perry-runtime/src/gc/tests/mod.rs index df76e87870..8550fa19e4 100644 --- a/crates/perry-runtime/src/gc/tests/mod.rs +++ b/crates/perry-runtime/src/gc/tests/mod.rs @@ -22,6 +22,7 @@ mod contract; mod copy_slot_decode; mod copy_slot_hoists; mod copying; +mod copying_object_scan; mod copying_side_tables; mod cycle_state; mod dead_owner_side_tables; @@ -87,6 +88,7 @@ mod scrub_dead_stack; mod shadow_stack_ops; mod shape_descriptor_authority; mod shape_keys_descriptor_edge; +mod shape_layout_table; mod smoke; mod start_bitmap; mod step_bounds; diff --git a/crates/perry-runtime/src/gc/tests/shape_layout_table.rs b/crates/perry-runtime/src/gc/tests/shape_layout_table.rs new file mode 100644 index 0000000000..c4b5cd274f --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/shape_layout_table.rs @@ -0,0 +1,57 @@ +//! `SHAPE_LAYOUTS`' one-entry pointer-mask memo (`gc/layout/shape_layout_table.rs`) +//! must never answer differently from the map. Every write reaches the map +//! through `DerefMut`, which forgets the memo; the sabotaged twin keeps it and +//! must be caught returning the answer from before the write. + +use crate::gc::layout::shape_layout_table::{sabotage, test_descriptor}; +use crate::gc::layout::{LayoutSlotMask, ShapeLayoutTable}; + +const SHAPE: u32 = 7; + +/// Memoize SHAPE's mask, then run each kind of write and ask again. Returns +/// the answers after each write. +fn answers_across_writes(sabotaged: bool) -> Vec> { + let _sabotage = sabotaged.then(sabotage::KeepMemo::arm); + let mut table = ShapeLayoutTable::new(); + table.insert(SHAPE, Some(test_descriptor(3, 0b101))); + let mut answers = Vec::new(); + // Populate the memo, and read it back once from the memo. + answers.push(table.shared_pointer_mask(SHAPE, 3)); + answers.push(table.shared_pointer_mask(SHAPE, 3)); + // A field-count mismatch is answered from the memo too. + answers.push(table.shared_pointer_mask(SHAPE, 2)); + // Poison: the shape became ambiguous. + table.insert(SHAPE, None); + answers.push(table.shared_pointer_mask(SHAPE, 3)); + // Re-learned with a different layout through `entry`. + table.remove(&SHAPE); + let _ = table.shared_pointer_mask(SHAPE, 3); + table + .entry(SHAPE) + .or_insert(Some(test_descriptor(3, 0b011))); + answers.push(table.shared_pointer_mask(SHAPE, 3)); + answers +} + +fn expected() -> Vec> { + vec![ + Some(LayoutSlotMask::Inline(0b101)), + Some(LayoutSlotMask::Inline(0b101)), + None, + None, + Some(LayoutSlotMask::Inline(0b011)), + ] +} + +#[test] +fn the_shape_mask_memo_always_answers_what_the_map_answers() { + assert!(answers_across_writes(false) == expected()); +} + +#[test] +fn a_shape_mask_memo_kept_across_a_write_answers_stale() { + assert!( + answers_across_writes(true) != expected(), + "a memo that survives a write must be caught answering from before it" + ); +} diff --git a/crates/perry-runtime/src/object/shapes.rs b/crates/perry-runtime/src/object/shapes.rs index 53f6978407..85cbdf600d 100644 --- a/crates/perry-runtime/src/object/shapes.rs +++ b/crates/perry-runtime/src/object/shapes.rs @@ -1726,6 +1726,22 @@ pub(crate) unsafe fn note_old_generation_carrier(record: Option) } } +/// Would [`note_old_generation_carrier`] change nothing for `record` right now? +/// +/// True when both of its flags are already set — the note then re-sets them +/// and finds it is not the first this epoch — or when there is no record. The +/// copying drain asks this BEFORE classifying the receiver's generation, so a +/// shape whose carrier was already noted this epoch skips that page-map probe +/// entirely (#11549). Exact: it reads the same two flags the note would. +#[inline] +pub(crate) unsafe fn old_generation_carrier_already_noted(record: Option) -> bool { + let Some(record) = record else { + return true; + }; + let record = record.0.as_ptr(); + (*record).has(RECORD_FLAG_OLD_CARRIER) && (*record).has(RECORD_FLAG_OLD_CARRIER_SEEN) +} + /// Note that a complete full trace visited a receiver carrying this shape. /// Unlike the old-generation gate, this answers receiver liveness regardless /// of generation and is consumed by post-trace descriptor retirement. diff --git a/scripts/gc_runtime_root_holders.json b/scripts/gc_runtime_root_holders.json index e82b63760e..e64244f47b 100644 --- a/scripts/gc_runtime_root_holders.json +++ b/scripts/gc_runtime_root_holders.json @@ -362,7 +362,7 @@ "file": "crates/perry-runtime/src/gc/census.rs", "name": "PASS1_MARKED", "verdict": "non_moving_snapshot", - "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged. Re-audited 2026-09-07 for #9965 after 1ec9e0e8a touched `gc/cycle.rs` and `gc/mod.rs`: `gc/mod.rs:216-217` only declares and imports the failure-attribution module, while `gc/cycle.rs:1414-1417` reads the trigger and diagnostic counters immediately before evacuation verification inside `atomic_finalize_minor_prelude`. Full cycles bypass `MinorPrelude` at `gc/cycle.rs:1192-1196`; evacuation remains guarded by the minor-only context at `gc/cycle.rs:1330-1372`. The snapshot store remains at `gc/cycle.rs:963-964` after synchronous full marking, and its take remains at `gc/cycle.rs:1454-1457` before sweep. No new write, relocation, collection, or JS callback was added to that full-cycle interval, so the PASS1_MARKED window is unaffected. Re-audited 2026-09-07 for the regex census rows: all new work is in `take_census` after `census_take_if_armed_at_full_sweep_start` has taken PASS1_MARKED out of TLS; neither boundary nor the intervening cycle control flow changed. Re-audited 2026-09-08 (train144) after #9976 and #9977 touched pinned files. `gc/mod.rs` gains exactly three lines: `mod copying_phase;` and `mod regex_census;` (declarations) and one `reg_scanner!(regex::site_test::scan_roots_mut)` registration. A scanner registration adds a root SOURCE for the mutable-root walks; it does not move either census boundary and runs nowhere between them. `gc/census.rs` widens `side_tables()` to `pub(super)`, extends it with regex rows and adds a test module \u2014 all census REPORTING, which runs from the diagnostic dump, not inside a cycle. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-08 for #9849 JSON construction deferral. `gc/mod.rs` adds the `json_defer` module/re-export and a trusted-header layout helper used only by already-validated JSON emitters; neither changes or runs in collector phase control flow. `gc/policy.rs` adds JSON completion scheduling, construction-grace checks, and safepoint deferral predicates. These are called from mutator-side JSON allocation/output boundaries and ordinary safepoint entry; they do not alter `step_mark_propagation`, `step_sweep`, or invoke callbacks or relocation between the census boundaries. The mark-complete to sweep-entry window is unchanged. The follow-up adds a cfg(test)-only one-shot boolean for deterministic explicit-pressure fixtures; it is absent from production builds and cannot affect the census window. The first predicate read consumes it, so post-parse accounting exercises normal pricing. Re-audited 2026-09-09 for bounded tiny-JSON completion polling. The policy.rs changes split the mutator-side pending-parse check into an inlined empty fast path plus an outlined debt-service path, and amortize the mutator-side arena-pressure read across 64 bounded parse completions. Neither function is reachable from step_mark_propagation or step_sweep; neither census boundary nor the synchronous full-cycle interval between them changes. Re-audited 2026-09-09 for lazy JSON record batches: policy.rs only widens gc_budgeted_cycle_active visibility from pub(super) to pub(crate). Its body remains a read-only Cell query. The new caller is lazy_get materialization in the mutator; run_to_completion, step_mark_propagation, census snapshot consumption at step_sweep, and the synchronous non-moving window are unchanged. Re-audited 2026-09-09 for completed JSON-output debt: the added gc_service_json_output_sweep function calls the existing trigger check from a rooted mutator boundary and reports whether its malloc-count request remains due. It is not called from any census or collector phase; the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-09 for the JSON byte-debt carry: the same mutator-only service helper now distinguishes requests satisfied before its call from those satisfied by its trigger check. The added enum contains no payload, both count reads are scalar, and no census boundary or collector phase changed. Re-audited 2026-09-11 for #10055: gc/mod.rs only registers the weak UTF-16 index scanner during gc_init. It neither marks strings nor allocates GC objects or runs JS; offset vectors use the Rust allocator. The mark-complete to sweep-entry census window and cycle control flow are unchanged. Re-audited 2026-09-11 for #10054: gc/mod.rs adds only the trim-cache mutable-root scanner registration in gc_init. Its scanner visits two existing string slots without allocating or invoking JS. Root scanning still precedes mark completion, and neither census boundary nor the synchronous mark-complete to sweep-entry window changes. Re-audited 2026-09-11 for #10060: the census array classifier now reads the logical element start and bounds its scan by the remaining capacity. The helper only reads the existing GC/header words and performs pointer arithmetic; it cannot allocate, collect, or call JS. This classifier runs in take_census after PASS1_MARKED has been taken out of TLS. Neither census boundary nor the mark-complete to sweep-entry control flow changed. Re-audited for #8512: gc/mod.rs only enables the existing PTY mutable-root scanner on Windows; it changes no mark/sweep phase or census boundary. The scanner visits NaN-boxed slots without running JS callbacks. Re-audited 2026-09-12 for the single regular-expression engine: `gc/mod.rs` changes `mod prefetch;` to `pub(crate) mod prefetch;` so the RegExp owner-table walks can prefetch headers, a visibility change with no new call in collector control flow; `gc/census.rs` changes only its `#[cfg(test)]` `regex_census_tests` module, dropping assertions for the previous engine's cache rows. Neither boundary (`census_pass1_if_armed` in `step_mark_propagation`, `census_take_if_armed_at_full_sweep_start` in `step_sweep`) nor the synchronous mark-complete to sweep-entry interval changes. Re-audited 2026-09-13 after the #10169 fix touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` gains only `pub(crate) use` re-exports (`policy::note_young_leaf_born_old`, `policy::young_generation_holds_a_nursery`, `promote_in_place::{young_generation_measured_dying, young_generation_measured_retained}`, and cfg(test) survival seeders). `gc/policy.rs` gains a `Cell` thread-local (`GC_YOUNG_LEAF_BORN_OLD`, no pointer), its setter, a pure predicate over `copying_from_space_in_use_bytes` vs the base nursery cap, and a consumed-once branch at the top of `gc_budgeted_due_trigger` that may answer `YoungScavengeCap` ahead of `OldReclaim`. That branch decides WHICH collection a safepoint starts (a minor instead of a full); it runs before any cycle begins and never inside one, so the mark-complete \u2192 sweep-entry window of a synchronous full \u2014 where PASS1_MARKED is populated and consumed within one `run_to_completion` \u2014 is unchanged, and neither hunk adds an allocation, a JS callback, or a relocation to it. Re-audited 2026-09-13 for the heap generation (#10164 cross-call search positions): `gc/mod.rs` only declares `pub(crate) mod heap_generation;`. `gc/cycle.rs` wraps the `Sweep` and `Reclaim` arms of `GcCycleState::step` in a `HeapChange` scope and opens one inside `atomic_finalize_minor_prelude`'s evacuation branch (with a nested one around old-page defrag). Opening and closing a scope only increments two thread-local integer cells (`HEAP_GENERATION`, `OPEN_HEAP_CHANGES`); a first thread-local read may allocate a key through the global allocator, which neither relocates nor runs JS. The `Sweep` scope opens immediately before `step_sweep`, i.e. before `census_take_if_armed_at_full_sweep_start` takes PASS1_MARKED out of TLS, and adds no relocation, collection or JS callback to the synchronous mark-complete to sweep-entry window; the minor-prelude scope is unreachable from a full cycle, which bypasses `MinorPrelude`. Neither boundary nor the intervening control flow changed. Re-audited 2026-09-13 for #10182 block-granular reclamation, which touched `gc/cycle.rs`. Two hunks: (a) in the `RememberedSetRebuild` subphase of AtomicFinalize \u2014 INSIDE the window \u2014 the require-marked old-to-young rebuild is now constructed with `OldToYoungRememberedRebuildState::new_skipping`, whose cursor never enters blocks the census recorded as holding no reached, pinned or pre-marked object (`BlockCensus::unmarked_blocks`); computing that list reads `arena_block_snapshots()` and allocates one `Vec` through the global allocator. It visits a subset of the same objects the rebuild already walked (every skipped object would have been rejected as unmarked), and it neither allocates a GC object, relocates anything, nor runs a JS callback. (b) In `step_sweep`, `IncrementalSweepState::with_block_skip` runs after `census_take_if_armed_at_full_sweep_start` has already taken PASS1_MARKED out of TLS. Neither boundary moved and the synchronous mark-complete to sweep-entry interval gains no relocation, collection or callback. Re-audited 2026-09-11 for the startup memory profile: gc/mod.rs only retains the pre-main allocator-policy constructor in js_gc_init. The constructor applies process allocation options, without invoking GC or JS. No census boundary, collector phase, or mark-complete to sweep-entry control flow changed. Re-audited 2026-09-13 for #10179: census.rs only adds a native regex cache metadata row and its unit assertion; snapshot consumption and the full-cycle window are unchanged. Re-audited 2026-09-14 for the GC due-check fast path, which touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` only changes the safepoint re-exports: `gc_runtime_safepoint` becomes cfg(test) and `gc_runtime_safepoint_poll` is added. `gc/policy.rs`: the budgeted step returns a debt-free `GcStepReport` (debt is attached by the FFI and test entry points after the step returns) and moves cycle start/step into an out-of-line `gc_budgeted_start_or_step`; `gc_check_trigger` reuses a repeatable due-trigger answer through `DueTriggerMemo`, placed after its `GC_FLAG_IN_ALLOC` and suppression early returns; the young scavenge cap reuses the old-gen pressure value the due trigger already read and checks the census-seeded flag first. All of it runs from mutator safepoints, allocation-point trigger checks and the host step API, before a cycle starts or between budgeted steps. None of it is reachable between `census_pass1_if_armed` in `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` in `step_sweep` of a synchronous full: an allocation inside that window reaches `gc_check_trigger` with `GC_FLAG_IN_ALLOC` set and returns before the changed code. No allocation, relocation, collection or JS callback is added to the window. Re-audited 2026-09-14 for the tiny-parse nursery-cap boundary, which touched `gc/policy.rs`. It adds `tiny_parse_generational_collection_due`, a pure predicate (the existing `tiny_parse_pressure_due` OR the existing `young_scavenge_cap_due` read), and calls it instead of `tiny_parse_pressure_due` from `gc_bump_malloc_trigger_inner` and `gc_collect_pending_suppressed_parse_slow` (generational branch only) and from `gc_schedule_parse_boundary_collection_if_pressure`. All three are JSON.parse mutator-side boundaries, none reachable from `step_mark_propagation` or `step_sweep`; the predicate reads counters and allocates nothing. Neither census boundary nor the synchronous mark-complete to sweep-entry interval changed. Re-audited 2026-09-13 for #10182's full-collection throughput follow-up, which touched `gc/cycle.rs` in one hunk, INSIDE the window: the `RememberedSetRebuild` subphase of a synchronous full now first asks `verify::full_remembered_rebuild_provably_empty` and, when it holds, installs `OldToYoungRememberedRebuildState::provably_empty()` (an empty sticky set, no walk) instead of the require-marked rebuild. The predicate reads `arena_block_snapshots()` (one `Vec` through the global allocator), the census's per-block reached/pre-marked facts and the malloc registry's length; the constructor bumps a `Cell` counter and prints one line under `PERRY_GC_DIAG`. None of it allocates a GC object, relocates anything, collects, or runs a JS callback, and both census boundaries stay where they were. Re-audited 2026-09-14 for #10182's pacing-full work, which touched `gc/cycle.rs`, `gc/mod.rs` and `gc/policy.rs`. `gc/cycle.rs`: `GcCycleState::new_full` no longer calls `materialize_all_promoted_page_runs`; that call ran in the constructor, before the census and far before `census_pass1_if_armed`, and removing it adds nothing to the window. `gc/mod.rs`: one `mod promoted_cohort;` declaration. `gc/policy.rs`: (a) `credit_promoted_bytes_to_old_baseline` also credits a `Cell` cohort counter (it runs after a copying minor completes); (b) `finish_full_old_reclaim_baseline` also records the verified old live bytes and resets that counter (Publish, after `step_sweep` consumed the snapshot); (c) `gc_safepoint_moving_minor` arms and disarms the promotion-census record around its nursery minor and calls `run_promoted_cohort_full_if_due`, which starts a synchronous full through the same `gc_collect_full_mark_sweep_with_trigger` entry and reads byte counters before and after it. All of it runs before a cycle starts or after it completes; none of it runs between mark completion and sweep entry, allocates a GC object, relocates anything, or calls into JS. The census the promoted-cohort full may adopt from the promotion walk is built in `BuildValidPointerSet`, before either boundary. Both boundaries are unchanged. Re-audited 2026-09-14 for the #10182 dead-stack scrub in `gc/cycle.rs`: `step_build_valid_pointer_set` now calls `scrub_dead_stack_below`, which zeroes a local array in its own frame (dead stack below the caller), right after the census finishes \u2014 in `BuildValidPointerSet`, before the root scan and far before `census_pass1_if_armed`. It writes no heap memory, allocates nothing, relocates nothing and calls no JS; both boundaries are unchanged. Re-audited 2026-09-14 for #10241 (cohort survival), which touched `gc/cycle.rs` and `gc/policy.rs`. `gc/cycle.rs`: one call, `promoted_cohort::survival::check_minor_view_at_full_sweep_start()`, in `step_sweep` immediately AFTER `census_take_if_armed_at_full_sweep_start` has taken PASS1_MARKED out of TLS, i.e. outside the window. It is a no-op unless a promoted-cohort full armed its survival probe; when armed it walks the old page index over the preceding minor's dirty pages (`old_arena_walk_objects_on_pages`, Rust-allocator Vecs), reads GC headers' mark flags and the slots of unmarked ones, and records one enum. It writes no heap memory, allocates no GC object, relocates nothing and calls no JS. `gc/policy.rs`: `run_promoted_cohort_full_if_due` arms the probe before `gc_collect_full_mark_sweep_with_trigger` and takes it after the full returns (feeding `note_full_measured_promotion_survival` and one diagnostic line); both run before a cycle starts or after it completes. Both boundaries are unchanged. Re-audited 2026-09-14 for #10241's in-place-only cohort: `gc/policy.rs` drops the `promoted_cohort::note_promoted` call from `credit_promoted_bytes_to_old_baseline` (the copying minor now calls `promoted_cohort::note_minor_promotion` itself, after the credit). Both run at the end of a copying minor, outside any full cycle; the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-14 for the parse-boundary side-allocation band (medium-parse pacing), which touched `gc/policy.rs`. Three hunks: (a) a `Cell` thread-local (`GC_LAST_COLLECTION_EXTERNAL_SIDE_BYTES`, a byte COUNT, no pointer) plus three pure predicates over it and `external_side_live_bytes()`; (b) that predicate added as a third disjunct of `tiny_parse_generational_collection_due`, which is read only from the three JSON.parse mutator-side boundaries (`gc_bump_malloc_trigger_inner`, `gc_collect_pending_suppressed_parse_slow`, `gc_schedule_parse_boundary_collection_if_pressure`), none of them reachable from `step_mark_propagation` or `step_sweep`; and (c) one extra `Cell` store in `note_collection_finished_arena_occupancy` plus two extra reads in the `PERRY_GC_DIAG` tiny-parse line. `note_collection_finished_arena_occupancy` runs from `publish_reclaim_outcome` in the Publish subphase, i.e. AFTER `step_sweep` has already `take()`n the snapshot out of the thread-local, exactly as #9831's store on the same line does. Nothing added allocates a GC object, relocates anything, or runs a JS callback, and neither census boundary moved. Re-audited 2026-09-14 for the drained-bytes counterweight to that band, which touched `gc/policy.rs` again. Four hunks: a second `Cell` thread-local (`GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL`, a byte COUNT); one increment of it inside `gc_note_external_side_free`; a pure read (`external_side_old_reclaim_pressure_bytes`) substituted for `external_side_live_bytes()` at the four old-reclaim pressure sites; and one `Cell` store at the top of `finish_full_old_reclaim_baseline`. None of it can run between the census boundaries. `gc_note_external_side_free` is also reached by mutator-side tape materialization, regex scratch teardown, native-addon adjustments and buffer replacement. Its added operation is only a saturating increment of a scalar Cell, with no GC allocation, relocation, collection or JS callback, so this wider caller set does not invalidate the census window. `finish_full_old_reclaim_baseline` runs from `publish_reclaim_outcome` in the Publish subphase, the same place #9831's store already sits. The pressure reads happen at trigger decisions, before a cycle starts. No allocation, relocation, collection or JS callback is added to the mark-complete -> sweep-entry window, and neither boundary moved. Re-audited 2026-09-15 for turnloop P0, which touched `gc/mod.rs` with one added call: `crate::event_pump::shutdown_wait_driver()` inside `js_gc_release_current_thread_collection_side_allocations`, the process-exit funnel. That function runs once no more JavaScript can run on the thread, never from inside a collection cycle; the added call drops the thread's turnloop wait loop (closing its kqueue/epoll descriptor) and may print a diagnostic line. It allocates no GC object, relocates nothing, starts no collection and runs no JS callback. The census boundaries and the mark-complete -> sweep-entry window are untouched.. Re-audited 2026-09-16 for the copying minor's per-parent weak-holder fact: `gc/mod.rs` gains exactly one line, `mod copying_parent_facts;`, a module declaration. The module it declares holds `weak_holder_fact` (a read of the parent's `obj_type`/`class_id` via `weakref::is_weak_holder_header`) and the copying minor's `visit_slot_with_parent`, moved verbatim out of `gc/copying.rs` for the 2000-line lint. Both run only inside a COPYING MINOR, which skips both census boundaries (`census_pass1_if_armed` in `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` in `step_sweep` are synchronous-full only). Nothing was added to any full-cycle phase, and the declaration itself executes no code. Neither boundary moved and the synchronous mark-complete to sweep-entry window gains no allocation, relocation, collection or JS callback. Re-audited 2026-09-18 for the #10532 follow-up argument-list rooting fix, which touched `gc/mod.rs`. The only change there is `mod collection_points;` plus a `pub(crate) use collection_points::collection_point;` re-export (and, under `#[cfg(test)]`, `arm_collection_point`). `collection_point` is an inline no-op outside `cfg(test)`; under test it only runs a copying minor when called from ordinary MUTATOR code (`proxy.rs`'s `Reflect.apply` rebind path and `registry.rs`'s rest-array bundler), never from inside `step_mark_propagation` or `step_sweep`. Neither `census_pass1_if_armed` nor `census_take_if_armed_at_full_sweep_start` is reachable from it, so the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-18 (same PR, round 2) for the added `arm_collection_point_after` re-export in `gc/mod.rs`: another pure re-export line, same as the `collection_point`/`arm_collection_point` one already covered above. `arm_collection_point_after` only changes test-only arming state in `collection_points.rs` (which named site fires and on which hit); it still runs no mark/sweep control flow. Re-audited 2026-09-19 for #10735 (require.main threading): gc/mod.rs gains exactly one line, `reg_scanner!(crate::module_require::scan_cjs_main_module_root_mut);`, registering the new CJS_MAIN_MODULE thread-local's mutable-root scanner beside the existing `scan_module_path_roots_mut` registration. A scanner registration adds a root SOURCE for the mutable-root walks; it runs during root scanning, before mark propagation completes, and does not execute between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start`. Neither census boundary moved and the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-20 for #10834 (inherited-property read cache). `gc/mod.rs` gains exactly one line: `reg_scanner!(crate::object::inherited_read_cache::scan_inherited_read_cache_roots_mut);` in `gc_init()`. A scanner registration adds a root SOURCE for the mutable-root walks. The walk runs inside `RootScanCycleState::step_current_subphase`, i.e. entirely within the RootScan phase: `step_root_scan` only sets `self.phase = GcCyclePhase::MarkPropagation` once that loop reports done (`gc/cycle.rs:958-961`), and `census_pass1_if_armed()` fires at the END of `step_mark_propagation` (`gc/cycle.rs:982`). The scanner therefore runs strictly BEFORE the window opens and can never execute between the boundaries. Its body is a bounded walk of a fixed 512-entry thread-local array calling `visit_tagged_usize_slot` / `visit_usize_slot`; it allocates nothing, relocates nothing and runs no JS callback. Same shape as #9769, #9976/#9977, #10054, #10055 and #10735, all previously cleared. The PR also adds an `INHERITED_READ_CACHE` entry to `DEAD_KEY_PRUNES` in `gc/dead_owner.rs` (not a pinned source). That registry is consumed by `IncrementalSweepState::with_dead_collection_finalize` at `gc/cycle.rs:1548`, which is AFTER `census_take_if_armed_at_full_sweep_start` at `gc/cycle.rs:1505` has already `take()`n the snapshot out of the thread-local -- the same argument that cleared #9845's `collect_dead_registered_regexps_post_trace`. The prune reads addresses and zeroes entries; no GC allocation, relocation or callback. Both additions sit outside the window, on opposite sides of it. Neither boundary moved and the synchronous mark-complete to sweep-entry interval is unchanged. Re-audited 2026-09-22 for #10399 (per-thread module init), which touched `gc/mod.rs`. Two hunks, both init-time: a new free function `raise_default_thread_stack_floor()` and one call to it at the top of `js_gc_init`, before `enter_current_thread_image`'s successor statements. The function reads `RUST_MIN_STACK` from the environment and, only when it is unset, sets it to 32 MiB so a thread spawned against a multi-megabyte static TLS block still has usable stack (glibc carves static TLS out of the thread's stack mapping). It touches no heap object, allocates no GC object, relocates nothing and runs no JS callback. `js_gc_init` is the first runtime call of a compiled `main`, so it runs once before any cycle exists, and it is not reachable from `step_mark_propagation` or `step_sweep`. Same shape as the 2026-09-11 startup-memory-profile re-audit, which cleared the pre-main allocator-policy constructor in the same function. Neither census boundary moved and the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-23 (size/runtime-decouple, #11135) after the binary-size branch touched `gc/census.rs`, `gc/mod.rs` and `gc/policy.rs`. census.rs: `census_pass1_if_armed` / `census_take_if_armed_at_full_sweep_start` keep their bodies verbatim, moved into `_impl` functions compiled only with the new `gc-instruments` feature (without it both are empty and `census_path()` is `None`, so nothing is ever armed); the take still empties PASS1_MARKED before `take_census`. gc/mod.rs: `gc_init` gains a startup env check that aborts when an instrument knob is set without the feature, before any cycle exists. gc/policy.rs: env-knob OnceLock caches now initialize through `crate::once_init::get_or_init` (same closures, same values). No mark/sweep control flow between the two census boundaries changed; the window is unchanged. Re-audited for Fetch handle reclamation: cycle.rs only redirects the incomplete-cycle Drop cancellation hook to also cancel the Fetch trace. The full-trace finish hook removes native records and cached slots without allocating GC objects or invoking JS; it cannot relocate the census addresses before sweep entry. Re-audited 2026-09-22 for #10928 (one proportional old-reclaim rule), which touched `gc/policy.rs`. Six hunks. (a) Two new thread-locals, `GC_OLD_RECLAIM_PRE_IN_USE_BYTES` (`Cell`) and `GC_OLD_RECLAIM_BACKOFF_SHIFT` (`Cell`): both are byte/shift COUNTS, neither holds a pointer. (b) `gc_old_reclaim_growth_band_bytes` gains a `Cell` read and a left shift -- pure arithmetic over byte counts. (c) `old_reclaim_pressure_due` loses the #7937 absolute first-crossing arm, splits its pure form out as `old_reclaim_pressure_due_inner`, and calls `note_old_reclaim_cycle_started()` when the answer is true. That predicate is read at TRIGGER decisions only -- the allocation-point `gc_check_trigger` and `gc_budgeted_due_trigger` at safepoints -- i.e. before a cycle starts, never between the boundaries; an allocation inside the window reaches `gc_check_trigger` with `GC_FLAG_IN_ALLOC` set and returns before this code, the same argument the 2026-09-14 due-check fast-path re-audit made for the same function. Even if it did run there it would be sound: `note_old_reclaim_cycle_started` stores one scalar `Cell` from `pacing_arena_in_use_bytes()` (a read of `arena_live_allocated_bytes`), which allocates no GC object, relocates nothing and runs no JS callback -- the window's contract. (d) `update_old_reclaim_backoff` is called only from `finish_full_old_reclaim_baseline`, which runs from `publish_reclaim_outcome` in the Publish subphase, AFTER `step_sweep` has already `take()`n the snapshot out of the thread-local -- exactly where #9831's store and the medium-parse pacing store already sit. (e) `gc_old_reclaim_debt_bytes` drops the absolute arm it mirrored; it remains pure arithmetic read at debt/trigger decisions. (f) `#[cfg(test)]` seams, absent from production builds. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback between them; the change alters only WHEN a collection is scheduled, never what runs inside one. Neither boundary moved and the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-24 for #10960 (growth-aware old-reclaim backoff), which touched `gc/policy.rs` again. One new thread-local, `GC_OLD_RECLAIM_LAST_POST_IN_USE_BYTES` (`Cell`), a byte COUNT that holds no pointer. It is written only by `update_old_reclaim_backoff`, which runs from `finish_full_old_reclaim_baseline` in the Publish subphase, after `step_sweep` has already taken the snapshot out of the thread-local; the change there is pure integer arithmetic deciding whether to widen the band. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback, and neither window boundary moved. Re-audited 2026-09-24 after the class-capture environment added one `reg_scanner!` registration (`scan_class_env_roots_mut`, visiting each guarded class environment's owner class object) to `gc/mod.rs`: a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-25 after the inherited-access lane touched `gc/mod.rs`: the change is one `reg_scanner!` registration for `object::chain_store::scan_chain_store_roots_mut`, a root scanner that visits store-site chain verdicts (one interned key pointer each) during root scanning. It runs at the start of a cycle, never between mark completion and sweep entry, relocates nothing and runs no JS callback. The synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-25 for #10698 (allocation-point trigger watermark), which touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` gains `pub(crate) mod trigger_watermark;` and a `pub(crate) use` re-export -- no control flow. In `gc/policy.rs`: (a) `gc_check_trigger` becomes an out-of-line wrapper over an inlined fast path that reads the watermark cell, the malloc registry's length through an unguarded shared borrow and the inline allocator's offset, then either returns or runs the previous body unchanged (`gc_check_trigger_evaluate`). An allocation inside the window reaches it with `GC_FLAG_IN_ALLOC` set: the fast path returns without acting, which is the outcome of the `GC_FLAG_IN_ALLOC` early return it would otherwise reach, and the slow path still takes that early return. (b) The due-trigger evaluation also returns a watermark -- integer arithmetic over values it already read -- published only on the no-action return past every entry guard. (c) The ladder's thresholds and flags are retyped `Cell` -> `TriggerInput`, whose writes add one store retiring the watermark; heap-generation advances and `Arena::set_current` retire it too. (d) `#[cfg(test)]` seams. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback; `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`, so the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-25 for #10498 (class-accessor cache), which adds one `reg_scanner!` registration (`scan_class_accessor_cache_roots_mut`, marking and rewriting the cache's key strings) to `gc/mod.rs`: again a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-26 for #10572: `gc/mod.rs` gains one `#[cfg(not(feature = \"hot-diag\"))]` call to `hot_diag::refuse_knobs_without_hot_diag()` in `gc_init`, a startup-only knob check that runs before any cycle and alters no mark/sweep control flow; the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-27 for #11416: `gc/mod.rs` loses the `reg_scanner!` registration for `class_accessor_cache::scan_class_accessor_cache_roots_mut` (the cache is deleted); removing a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 for the method-calls lane, which adds one `reg_scanner!` registration (`scan_method_site_roots_mut`, marking and rewriting the method closures inherited method-site entries hold) to `gc/mod.rs`: a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-28 (charter step 5, P0): the field-representation census adds `mod census_field_repr;` to `gc/mod.rs` and, in `census.rs`, one Rust-owned accumulator fed from `visit_object` inside `take_census`, which runs after `census_take_if_armed_at_full_sweep_start` has already taken PASS1_MARKED; it reads slots and the per-object layout maps (try_borrow), allocates nothing on the JS heap and alters no mark/sweep control flow. The window is unchanged. Re-audited 2026-09-27 for #11416: `gc/mod.rs` loses the `reg_scanner!` registration for `class_accessor_cache::scan_class_accessor_cache_roots_mut` (the cache is deleted); removing a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 for the method-calls lane, which adds one `reg_scanner!` registration (`scan_method_site_roots_mut`, marking and rewriting the method closures inherited method-site entries hold) to `gc/mod.rs`: a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 (class constructors as function objects, #11414): `gc/mod.rs` gains one `reg_scanner!` registration (`object::class_value::scan_class_value_roots_mut`, the per-agent class function-object table, visited and rewritten) \u2014 a root-scanner registration that alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-29 after #11659 added `verify::verify_array_hole_tails_at_collection()` to `gc/mod.rs` and `gc/policy.rs`. Every call sits at a collection or budgeted-cycle ENTRY, beside `roots::ensure_stack_maps_built()`, i.e. before mark begins and so before `census_pass1_if_armed` opens the window. The call is a read-only walk of array headers (debug builds, or release with PERRY_GC_VERIFY_ARRAY_HOLES): it neither allocates GC memory, relocates, runs JS, nor reads or writes PASS1_MARKED, and it panics rather than continuing on a violation. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Pin re-audited 2026-09-29 after PR #11646 added one call inside that window, in `cycle.rs` beside `census_take_if_armed_at_full_sweep_start`: `object::shapes::store_kind::audit_heap_at_full_sweep_start()`. It compiles to nothing unless the `shape-fact-audit` feature is on; when on it walks the arena read-only (`gc::for_each_live_object_at_sweep_start`, in `gc/verify.rs`) and reads each marked object header and its shape record. It allocates no GC object, moves nothing, runs no JS callback and holds no address past the walk, so PASS1_MARKED stays valid across it; a disagreement panics rather than continuing. Re-audited 2026-09-29 after the pinned-roots fix touched `gc/cycle.rs` and `gc/mod.rs`: the block-persistence live-block predicate now reads GC_FLAG_MARKED alone (pinned objects are marked as roots, so the set of live blocks is unchanged), and `gc/mod.rs` adds one `reg_scanner!` registration. Neither relocates, runs JS, or moves the mark-complete to sweep-entry window. Re-audited again 2026-09-29 (same fix, copying-minor follow-up): `BlockPersistCycleState`'s force-mark in `gc/cycle.rs` no longer skips an unmarked pinned header, so it marks and pushes it like any other object of a live block. That is marking inside the budgeted cycle's persistence step: nothing relocates, no JS runs, and the budgeted path still skips both census boundaries. The window is unchanged. Re-audited 2026-09-29 for this-as-a-parameter stage 3: `gc/mod.rs` registers `scan_dispatch_binding_roots_mut` in place of `scan_implicit_this_roots_mut` (the implicit-`this` cell is deleted; the same scanner body keeps `new.target`, the static-`this` override and the static private-owner stack) and rewords its comment; a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-28 for the JsFunctionInfo lane: `gc/census.rs`'s `side_tables()` loses its `closure_registry_census()` row (the closure-body registry is deleted); a census report row, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-29 for the read-site holder entry: `gc/mod.rs` adds one `reg_scanner!` registration, `read_holder::scan_read_holder_roots_mut`, which visits the holder and hop words of registered read-site caches during ROOT SCAN, before mark propagation completes. It rewrites root slots only through the visitor (as every registered scanner does), runs no JS, and nothing it does executes between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start`. The window is unchanged.", + "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged. Re-audited 2026-09-07 for #9965 after 1ec9e0e8a touched `gc/cycle.rs` and `gc/mod.rs`: `gc/mod.rs:216-217` only declares and imports the failure-attribution module, while `gc/cycle.rs:1414-1417` reads the trigger and diagnostic counters immediately before evacuation verification inside `atomic_finalize_minor_prelude`. Full cycles bypass `MinorPrelude` at `gc/cycle.rs:1192-1196`; evacuation remains guarded by the minor-only context at `gc/cycle.rs:1330-1372`. The snapshot store remains at `gc/cycle.rs:963-964` after synchronous full marking, and its take remains at `gc/cycle.rs:1454-1457` before sweep. No new write, relocation, collection, or JS callback was added to that full-cycle interval, so the PASS1_MARKED window is unaffected. Re-audited 2026-09-07 for the regex census rows: all new work is in `take_census` after `census_take_if_armed_at_full_sweep_start` has taken PASS1_MARKED out of TLS; neither boundary nor the intervening cycle control flow changed. Re-audited 2026-09-08 (train144) after #9976 and #9977 touched pinned files. `gc/mod.rs` gains exactly three lines: `mod copying_phase;` and `mod regex_census;` (declarations) and one `reg_scanner!(regex::site_test::scan_roots_mut)` registration. A scanner registration adds a root SOURCE for the mutable-root walks; it does not move either census boundary and runs nowhere between them. `gc/census.rs` widens `side_tables()` to `pub(super)`, extends it with regex rows and adds a test module \u2014 all census REPORTING, which runs from the diagnostic dump, not inside a cycle. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-08 for #9849 JSON construction deferral. `gc/mod.rs` adds the `json_defer` module/re-export and a trusted-header layout helper used only by already-validated JSON emitters; neither changes or runs in collector phase control flow. `gc/policy.rs` adds JSON completion scheduling, construction-grace checks, and safepoint deferral predicates. These are called from mutator-side JSON allocation/output boundaries and ordinary safepoint entry; they do not alter `step_mark_propagation`, `step_sweep`, or invoke callbacks or relocation between the census boundaries. The mark-complete to sweep-entry window is unchanged. The follow-up adds a cfg(test)-only one-shot boolean for deterministic explicit-pressure fixtures; it is absent from production builds and cannot affect the census window. The first predicate read consumes it, so post-parse accounting exercises normal pricing. Re-audited 2026-09-09 for bounded tiny-JSON completion polling. The policy.rs changes split the mutator-side pending-parse check into an inlined empty fast path plus an outlined debt-service path, and amortize the mutator-side arena-pressure read across 64 bounded parse completions. Neither function is reachable from step_mark_propagation or step_sweep; neither census boundary nor the synchronous full-cycle interval between them changes. Re-audited 2026-09-09 for lazy JSON record batches: policy.rs only widens gc_budgeted_cycle_active visibility from pub(super) to pub(crate). Its body remains a read-only Cell query. The new caller is lazy_get materialization in the mutator; run_to_completion, step_mark_propagation, census snapshot consumption at step_sweep, and the synchronous non-moving window are unchanged. Re-audited 2026-09-09 for completed JSON-output debt: the added gc_service_json_output_sweep function calls the existing trigger check from a rooted mutator boundary and reports whether its malloc-count request remains due. It is not called from any census or collector phase; the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-09 for the JSON byte-debt carry: the same mutator-only service helper now distinguishes requests satisfied before its call from those satisfied by its trigger check. The added enum contains no payload, both count reads are scalar, and no census boundary or collector phase changed. Re-audited 2026-09-11 for #10055: gc/mod.rs only registers the weak UTF-16 index scanner during gc_init. It neither marks strings nor allocates GC objects or runs JS; offset vectors use the Rust allocator. The mark-complete to sweep-entry census window and cycle control flow are unchanged. Re-audited 2026-09-11 for #10054: gc/mod.rs adds only the trim-cache mutable-root scanner registration in gc_init. Its scanner visits two existing string slots without allocating or invoking JS. Root scanning still precedes mark completion, and neither census boundary nor the synchronous mark-complete to sweep-entry window changes. Re-audited 2026-09-11 for #10060: the census array classifier now reads the logical element start and bounds its scan by the remaining capacity. The helper only reads the existing GC/header words and performs pointer arithmetic; it cannot allocate, collect, or call JS. This classifier runs in take_census after PASS1_MARKED has been taken out of TLS. Neither census boundary nor the mark-complete to sweep-entry control flow changed. Re-audited for #8512: gc/mod.rs only enables the existing PTY mutable-root scanner on Windows; it changes no mark/sweep phase or census boundary. The scanner visits NaN-boxed slots without running JS callbacks. Re-audited 2026-09-12 for the single regular-expression engine: `gc/mod.rs` changes `mod prefetch;` to `pub(crate) mod prefetch;` so the RegExp owner-table walks can prefetch headers, a visibility change with no new call in collector control flow; `gc/census.rs` changes only its `#[cfg(test)]` `regex_census_tests` module, dropping assertions for the previous engine's cache rows. Neither boundary (`census_pass1_if_armed` in `step_mark_propagation`, `census_take_if_armed_at_full_sweep_start` in `step_sweep`) nor the synchronous mark-complete to sweep-entry interval changes. Re-audited 2026-09-13 after the #10169 fix touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` gains only `pub(crate) use` re-exports (`policy::note_young_leaf_born_old`, `policy::young_generation_holds_a_nursery`, `promote_in_place::{young_generation_measured_dying, young_generation_measured_retained}`, and cfg(test) survival seeders). `gc/policy.rs` gains a `Cell` thread-local (`GC_YOUNG_LEAF_BORN_OLD`, no pointer), its setter, a pure predicate over `copying_from_space_in_use_bytes` vs the base nursery cap, and a consumed-once branch at the top of `gc_budgeted_due_trigger` that may answer `YoungScavengeCap` ahead of `OldReclaim`. That branch decides WHICH collection a safepoint starts (a minor instead of a full); it runs before any cycle begins and never inside one, so the mark-complete \u2192 sweep-entry window of a synchronous full \u2014 where PASS1_MARKED is populated and consumed within one `run_to_completion` \u2014 is unchanged, and neither hunk adds an allocation, a JS callback, or a relocation to it. Re-audited 2026-09-13 for the heap generation (#10164 cross-call search positions): `gc/mod.rs` only declares `pub(crate) mod heap_generation;`. `gc/cycle.rs` wraps the `Sweep` and `Reclaim` arms of `GcCycleState::step` in a `HeapChange` scope and opens one inside `atomic_finalize_minor_prelude`'s evacuation branch (with a nested one around old-page defrag). Opening and closing a scope only increments two thread-local integer cells (`HEAP_GENERATION`, `OPEN_HEAP_CHANGES`); a first thread-local read may allocate a key through the global allocator, which neither relocates nor runs JS. The `Sweep` scope opens immediately before `step_sweep`, i.e. before `census_take_if_armed_at_full_sweep_start` takes PASS1_MARKED out of TLS, and adds no relocation, collection or JS callback to the synchronous mark-complete to sweep-entry window; the minor-prelude scope is unreachable from a full cycle, which bypasses `MinorPrelude`. Neither boundary nor the intervening control flow changed. Re-audited 2026-09-13 for #10182 block-granular reclamation, which touched `gc/cycle.rs`. Two hunks: (a) in the `RememberedSetRebuild` subphase of AtomicFinalize \u2014 INSIDE the window \u2014 the require-marked old-to-young rebuild is now constructed with `OldToYoungRememberedRebuildState::new_skipping`, whose cursor never enters blocks the census recorded as holding no reached, pinned or pre-marked object (`BlockCensus::unmarked_blocks`); computing that list reads `arena_block_snapshots()` and allocates one `Vec` through the global allocator. It visits a subset of the same objects the rebuild already walked (every skipped object would have been rejected as unmarked), and it neither allocates a GC object, relocates anything, nor runs a JS callback. (b) In `step_sweep`, `IncrementalSweepState::with_block_skip` runs after `census_take_if_armed_at_full_sweep_start` has already taken PASS1_MARKED out of TLS. Neither boundary moved and the synchronous mark-complete to sweep-entry interval gains no relocation, collection or callback. Re-audited 2026-09-11 for the startup memory profile: gc/mod.rs only retains the pre-main allocator-policy constructor in js_gc_init. The constructor applies process allocation options, without invoking GC or JS. No census boundary, collector phase, or mark-complete to sweep-entry control flow changed. Re-audited 2026-09-13 for #10179: census.rs only adds a native regex cache metadata row and its unit assertion; snapshot consumption and the full-cycle window are unchanged. Re-audited 2026-09-14 for the GC due-check fast path, which touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` only changes the safepoint re-exports: `gc_runtime_safepoint` becomes cfg(test) and `gc_runtime_safepoint_poll` is added. `gc/policy.rs`: the budgeted step returns a debt-free `GcStepReport` (debt is attached by the FFI and test entry points after the step returns) and moves cycle start/step into an out-of-line `gc_budgeted_start_or_step`; `gc_check_trigger` reuses a repeatable due-trigger answer through `DueTriggerMemo`, placed after its `GC_FLAG_IN_ALLOC` and suppression early returns; the young scavenge cap reuses the old-gen pressure value the due trigger already read and checks the census-seeded flag first. All of it runs from mutator safepoints, allocation-point trigger checks and the host step API, before a cycle starts or between budgeted steps. None of it is reachable between `census_pass1_if_armed` in `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` in `step_sweep` of a synchronous full: an allocation inside that window reaches `gc_check_trigger` with `GC_FLAG_IN_ALLOC` set and returns before the changed code. No allocation, relocation, collection or JS callback is added to the window. Re-audited 2026-09-14 for the tiny-parse nursery-cap boundary, which touched `gc/policy.rs`. It adds `tiny_parse_generational_collection_due`, a pure predicate (the existing `tiny_parse_pressure_due` OR the existing `young_scavenge_cap_due` read), and calls it instead of `tiny_parse_pressure_due` from `gc_bump_malloc_trigger_inner` and `gc_collect_pending_suppressed_parse_slow` (generational branch only) and from `gc_schedule_parse_boundary_collection_if_pressure`. All three are JSON.parse mutator-side boundaries, none reachable from `step_mark_propagation` or `step_sweep`; the predicate reads counters and allocates nothing. Neither census boundary nor the synchronous mark-complete to sweep-entry interval changed. Re-audited 2026-09-13 for #10182's full-collection throughput follow-up, which touched `gc/cycle.rs` in one hunk, INSIDE the window: the `RememberedSetRebuild` subphase of a synchronous full now first asks `verify::full_remembered_rebuild_provably_empty` and, when it holds, installs `OldToYoungRememberedRebuildState::provably_empty()` (an empty sticky set, no walk) instead of the require-marked rebuild. The predicate reads `arena_block_snapshots()` (one `Vec` through the global allocator), the census's per-block reached/pre-marked facts and the malloc registry's length; the constructor bumps a `Cell` counter and prints one line under `PERRY_GC_DIAG`. None of it allocates a GC object, relocates anything, collects, or runs a JS callback, and both census boundaries stay where they were. Re-audited 2026-09-14 for #10182's pacing-full work, which touched `gc/cycle.rs`, `gc/mod.rs` and `gc/policy.rs`. `gc/cycle.rs`: `GcCycleState::new_full` no longer calls `materialize_all_promoted_page_runs`; that call ran in the constructor, before the census and far before `census_pass1_if_armed`, and removing it adds nothing to the window. `gc/mod.rs`: one `mod promoted_cohort;` declaration. `gc/policy.rs`: (a) `credit_promoted_bytes_to_old_baseline` also credits a `Cell` cohort counter (it runs after a copying minor completes); (b) `finish_full_old_reclaim_baseline` also records the verified old live bytes and resets that counter (Publish, after `step_sweep` consumed the snapshot); (c) `gc_safepoint_moving_minor` arms and disarms the promotion-census record around its nursery minor and calls `run_promoted_cohort_full_if_due`, which starts a synchronous full through the same `gc_collect_full_mark_sweep_with_trigger` entry and reads byte counters before and after it. All of it runs before a cycle starts or after it completes; none of it runs between mark completion and sweep entry, allocates a GC object, relocates anything, or calls into JS. The census the promoted-cohort full may adopt from the promotion walk is built in `BuildValidPointerSet`, before either boundary. Both boundaries are unchanged. Re-audited 2026-09-14 for the #10182 dead-stack scrub in `gc/cycle.rs`: `step_build_valid_pointer_set` now calls `scrub_dead_stack_below`, which zeroes a local array in its own frame (dead stack below the caller), right after the census finishes \u2014 in `BuildValidPointerSet`, before the root scan and far before `census_pass1_if_armed`. It writes no heap memory, allocates nothing, relocates nothing and calls no JS; both boundaries are unchanged. Re-audited 2026-09-14 for #10241 (cohort survival), which touched `gc/cycle.rs` and `gc/policy.rs`. `gc/cycle.rs`: one call, `promoted_cohort::survival::check_minor_view_at_full_sweep_start()`, in `step_sweep` immediately AFTER `census_take_if_armed_at_full_sweep_start` has taken PASS1_MARKED out of TLS, i.e. outside the window. It is a no-op unless a promoted-cohort full armed its survival probe; when armed it walks the old page index over the preceding minor's dirty pages (`old_arena_walk_objects_on_pages`, Rust-allocator Vecs), reads GC headers' mark flags and the slots of unmarked ones, and records one enum. It writes no heap memory, allocates no GC object, relocates nothing and calls no JS. `gc/policy.rs`: `run_promoted_cohort_full_if_due` arms the probe before `gc_collect_full_mark_sweep_with_trigger` and takes it after the full returns (feeding `note_full_measured_promotion_survival` and one diagnostic line); both run before a cycle starts or after it completes. Both boundaries are unchanged. Re-audited 2026-09-14 for #10241's in-place-only cohort: `gc/policy.rs` drops the `promoted_cohort::note_promoted` call from `credit_promoted_bytes_to_old_baseline` (the copying minor now calls `promoted_cohort::note_minor_promotion` itself, after the credit). Both run at the end of a copying minor, outside any full cycle; the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-14 for the parse-boundary side-allocation band (medium-parse pacing), which touched `gc/policy.rs`. Three hunks: (a) a `Cell` thread-local (`GC_LAST_COLLECTION_EXTERNAL_SIDE_BYTES`, a byte COUNT, no pointer) plus three pure predicates over it and `external_side_live_bytes()`; (b) that predicate added as a third disjunct of `tiny_parse_generational_collection_due`, which is read only from the three JSON.parse mutator-side boundaries (`gc_bump_malloc_trigger_inner`, `gc_collect_pending_suppressed_parse_slow`, `gc_schedule_parse_boundary_collection_if_pressure`), none of them reachable from `step_mark_propagation` or `step_sweep`; and (c) one extra `Cell` store in `note_collection_finished_arena_occupancy` plus two extra reads in the `PERRY_GC_DIAG` tiny-parse line. `note_collection_finished_arena_occupancy` runs from `publish_reclaim_outcome` in the Publish subphase, i.e. AFTER `step_sweep` has already `take()`n the snapshot out of the thread-local, exactly as #9831's store on the same line does. Nothing added allocates a GC object, relocates anything, or runs a JS callback, and neither census boundary moved. Re-audited 2026-09-14 for the drained-bytes counterweight to that band, which touched `gc/policy.rs` again. Four hunks: a second `Cell` thread-local (`GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL`, a byte COUNT); one increment of it inside `gc_note_external_side_free`; a pure read (`external_side_old_reclaim_pressure_bytes`) substituted for `external_side_live_bytes()` at the four old-reclaim pressure sites; and one `Cell` store at the top of `finish_full_old_reclaim_baseline`. None of it can run between the census boundaries. `gc_note_external_side_free` is also reached by mutator-side tape materialization, regex scratch teardown, native-addon adjustments and buffer replacement. Its added operation is only a saturating increment of a scalar Cell, with no GC allocation, relocation, collection or JS callback, so this wider caller set does not invalidate the census window. `finish_full_old_reclaim_baseline` runs from `publish_reclaim_outcome` in the Publish subphase, the same place #9831's store already sits. The pressure reads happen at trigger decisions, before a cycle starts. No allocation, relocation, collection or JS callback is added to the mark-complete -> sweep-entry window, and neither boundary moved. Re-audited 2026-09-15 for turnloop P0, which touched `gc/mod.rs` with one added call: `crate::event_pump::shutdown_wait_driver()` inside `js_gc_release_current_thread_collection_side_allocations`, the process-exit funnel. That function runs once no more JavaScript can run on the thread, never from inside a collection cycle; the added call drops the thread's turnloop wait loop (closing its kqueue/epoll descriptor) and may print a diagnostic line. It allocates no GC object, relocates nothing, starts no collection and runs no JS callback. The census boundaries and the mark-complete -> sweep-entry window are untouched.. Re-audited 2026-09-16 for the copying minor's per-parent weak-holder fact: `gc/mod.rs` gains exactly one line, `mod copying_parent_facts;`, a module declaration. The module it declares holds `weak_holder_fact` (a read of the parent's `obj_type`/`class_id` via `weakref::is_weak_holder_header`) and the copying minor's `visit_slot_with_parent`, moved verbatim out of `gc/copying.rs` for the 2000-line lint. Both run only inside a COPYING MINOR, which skips both census boundaries (`census_pass1_if_armed` in `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` in `step_sweep` are synchronous-full only). Nothing was added to any full-cycle phase, and the declaration itself executes no code. Neither boundary moved and the synchronous mark-complete to sweep-entry window gains no allocation, relocation, collection or JS callback. Re-audited 2026-09-18 for the #10532 follow-up argument-list rooting fix, which touched `gc/mod.rs`. The only change there is `mod collection_points;` plus a `pub(crate) use collection_points::collection_point;` re-export (and, under `#[cfg(test)]`, `arm_collection_point`). `collection_point` is an inline no-op outside `cfg(test)`; under test it only runs a copying minor when called from ordinary MUTATOR code (`proxy.rs`'s `Reflect.apply` rebind path and `registry.rs`'s rest-array bundler), never from inside `step_mark_propagation` or `step_sweep`. Neither `census_pass1_if_armed` nor `census_take_if_armed_at_full_sweep_start` is reachable from it, so the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-18 (same PR, round 2) for the added `arm_collection_point_after` re-export in `gc/mod.rs`: another pure re-export line, same as the `collection_point`/`arm_collection_point` one already covered above. `arm_collection_point_after` only changes test-only arming state in `collection_points.rs` (which named site fires and on which hit); it still runs no mark/sweep control flow. Re-audited 2026-09-19 for #10735 (require.main threading): gc/mod.rs gains exactly one line, `reg_scanner!(crate::module_require::scan_cjs_main_module_root_mut);`, registering the new CJS_MAIN_MODULE thread-local's mutable-root scanner beside the existing `scan_module_path_roots_mut` registration. A scanner registration adds a root SOURCE for the mutable-root walks; it runs during root scanning, before mark propagation completes, and does not execute between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start`. Neither census boundary moved and the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-20 for #10834 (inherited-property read cache). `gc/mod.rs` gains exactly one line: `reg_scanner!(crate::object::inherited_read_cache::scan_inherited_read_cache_roots_mut);` in `gc_init()`. A scanner registration adds a root SOURCE for the mutable-root walks. The walk runs inside `RootScanCycleState::step_current_subphase`, i.e. entirely within the RootScan phase: `step_root_scan` only sets `self.phase = GcCyclePhase::MarkPropagation` once that loop reports done (`gc/cycle.rs:958-961`), and `census_pass1_if_armed()` fires at the END of `step_mark_propagation` (`gc/cycle.rs:982`). The scanner therefore runs strictly BEFORE the window opens and can never execute between the boundaries. Its body is a bounded walk of a fixed 512-entry thread-local array calling `visit_tagged_usize_slot` / `visit_usize_slot`; it allocates nothing, relocates nothing and runs no JS callback. Same shape as #9769, #9976/#9977, #10054, #10055 and #10735, all previously cleared. The PR also adds an `INHERITED_READ_CACHE` entry to `DEAD_KEY_PRUNES` in `gc/dead_owner.rs` (not a pinned source). That registry is consumed by `IncrementalSweepState::with_dead_collection_finalize` at `gc/cycle.rs:1548`, which is AFTER `census_take_if_armed_at_full_sweep_start` at `gc/cycle.rs:1505` has already `take()`n the snapshot out of the thread-local -- the same argument that cleared #9845's `collect_dead_registered_regexps_post_trace`. The prune reads addresses and zeroes entries; no GC allocation, relocation or callback. Both additions sit outside the window, on opposite sides of it. Neither boundary moved and the synchronous mark-complete to sweep-entry interval is unchanged. Re-audited 2026-09-22 for #10399 (per-thread module init), which touched `gc/mod.rs`. Two hunks, both init-time: a new free function `raise_default_thread_stack_floor()` and one call to it at the top of `js_gc_init`, before `enter_current_thread_image`'s successor statements. The function reads `RUST_MIN_STACK` from the environment and, only when it is unset, sets it to 32 MiB so a thread spawned against a multi-megabyte static TLS block still has usable stack (glibc carves static TLS out of the thread's stack mapping). It touches no heap object, allocates no GC object, relocates nothing and runs no JS callback. `js_gc_init` is the first runtime call of a compiled `main`, so it runs once before any cycle exists, and it is not reachable from `step_mark_propagation` or `step_sweep`. Same shape as the 2026-09-11 startup-memory-profile re-audit, which cleared the pre-main allocator-policy constructor in the same function. Neither census boundary moved and the synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-23 (size/runtime-decouple, #11135) after the binary-size branch touched `gc/census.rs`, `gc/mod.rs` and `gc/policy.rs`. census.rs: `census_pass1_if_armed` / `census_take_if_armed_at_full_sweep_start` keep their bodies verbatim, moved into `_impl` functions compiled only with the new `gc-instruments` feature (without it both are empty and `census_path()` is `None`, so nothing is ever armed); the take still empties PASS1_MARKED before `take_census`. gc/mod.rs: `gc_init` gains a startup env check that aborts when an instrument knob is set without the feature, before any cycle exists. gc/policy.rs: env-knob OnceLock caches now initialize through `crate::once_init::get_or_init` (same closures, same values). No mark/sweep control flow between the two census boundaries changed; the window is unchanged. Re-audited for Fetch handle reclamation: cycle.rs only redirects the incomplete-cycle Drop cancellation hook to also cancel the Fetch trace. The full-trace finish hook removes native records and cached slots without allocating GC objects or invoking JS; it cannot relocate the census addresses before sweep entry. Re-audited 2026-09-22 for #10928 (one proportional old-reclaim rule), which touched `gc/policy.rs`. Six hunks. (a) Two new thread-locals, `GC_OLD_RECLAIM_PRE_IN_USE_BYTES` (`Cell`) and `GC_OLD_RECLAIM_BACKOFF_SHIFT` (`Cell`): both are byte/shift COUNTS, neither holds a pointer. (b) `gc_old_reclaim_growth_band_bytes` gains a `Cell` read and a left shift -- pure arithmetic over byte counts. (c) `old_reclaim_pressure_due` loses the #7937 absolute first-crossing arm, splits its pure form out as `old_reclaim_pressure_due_inner`, and calls `note_old_reclaim_cycle_started()` when the answer is true. That predicate is read at TRIGGER decisions only -- the allocation-point `gc_check_trigger` and `gc_budgeted_due_trigger` at safepoints -- i.e. before a cycle starts, never between the boundaries; an allocation inside the window reaches `gc_check_trigger` with `GC_FLAG_IN_ALLOC` set and returns before this code, the same argument the 2026-09-14 due-check fast-path re-audit made for the same function. Even if it did run there it would be sound: `note_old_reclaim_cycle_started` stores one scalar `Cell` from `pacing_arena_in_use_bytes()` (a read of `arena_live_allocated_bytes`), which allocates no GC object, relocates nothing and runs no JS callback -- the window's contract. (d) `update_old_reclaim_backoff` is called only from `finish_full_old_reclaim_baseline`, which runs from `publish_reclaim_outcome` in the Publish subphase, AFTER `step_sweep` has already `take()`n the snapshot out of the thread-local -- exactly where #9831's store and the medium-parse pacing store already sit. (e) `gc_old_reclaim_debt_bytes` drops the absolute arm it mirrored; it remains pure arithmetic read at debt/trigger decisions. (f) `#[cfg(test)]` seams, absent from production builds. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback between them; the change alters only WHEN a collection is scheduled, never what runs inside one. Neither boundary moved and the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-24 for #10960 (growth-aware old-reclaim backoff), which touched `gc/policy.rs` again. One new thread-local, `GC_OLD_RECLAIM_LAST_POST_IN_USE_BYTES` (`Cell`), a byte COUNT that holds no pointer. It is written only by `update_old_reclaim_backoff`, which runs from `finish_full_old_reclaim_baseline` in the Publish subphase, after `step_sweep` has already taken the snapshot out of the thread-local; the change there is pure integer arithmetic deciding whether to widen the band. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback, and neither window boundary moved. Re-audited 2026-09-24 after the class-capture environment added one `reg_scanner!` registration (`scan_class_env_roots_mut`, visiting each guarded class environment's owner class object) to `gc/mod.rs`: a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-25 after the inherited-access lane touched `gc/mod.rs`: the change is one `reg_scanner!` registration for `object::chain_store::scan_chain_store_roots_mut`, a root scanner that visits store-site chain verdicts (one interned key pointer each) during root scanning. It runs at the start of a cycle, never between mark completion and sweep entry, relocates nothing and runs no JS callback. The synchronous mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-25 for #10698 (allocation-point trigger watermark), which touched `gc/mod.rs` and `gc/policy.rs`. `gc/mod.rs` gains `pub(crate) mod trigger_watermark;` and a `pub(crate) use` re-export -- no control flow. In `gc/policy.rs`: (a) `gc_check_trigger` becomes an out-of-line wrapper over an inlined fast path that reads the watermark cell, the malloc registry's length through an unguarded shared borrow and the inline allocator's offset, then either returns or runs the previous body unchanged (`gc_check_trigger_evaluate`). An allocation inside the window reaches it with `GC_FLAG_IN_ALLOC` set: the fast path returns without acting, which is the outcome of the `GC_FLAG_IN_ALLOC` early return it would otherwise reach, and the slow path still takes that early return. (b) The due-trigger evaluation also returns a watermark -- integer arithmetic over values it already read -- published only on the no-action return past every entry guard. (c) The ladder's thresholds and flags are retyped `Cell` -> `TriggerInput`, whose writes add one store retiring the watermark; heap-generation advances and `Arena::set_current` retire it too. (d) `#[cfg(test)]` seams. Nothing added allocates a GC object, relocates anything, collects, or invokes a JS callback; `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`, so the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-25 for #10498 (class-accessor cache), which adds one `reg_scanner!` registration (`scan_class_accessor_cache_roots_mut`, marking and rewriting the cache's key strings) to `gc/mod.rs`: again a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-26 for #10572: `gc/mod.rs` gains one `#[cfg(not(feature = \"hot-diag\"))]` call to `hot_diag::refuse_knobs_without_hot_diag()` in `gc_init`, a startup-only knob check that runs before any cycle and alters no mark/sweep control flow; the mark-complete to sweep-entry window is unchanged. Re-audited 2026-09-27 for #11416: `gc/mod.rs` loses the `reg_scanner!` registration for `class_accessor_cache::scan_class_accessor_cache_roots_mut` (the cache is deleted); removing a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 for the method-calls lane, which adds one `reg_scanner!` registration (`scan_method_site_roots_mut`, marking and rewriting the method closures inherited method-site entries hold) to `gc/mod.rs`: a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-28 (charter step 5, P0): the field-representation census adds `mod census_field_repr;` to `gc/mod.rs` and, in `census.rs`, one Rust-owned accumulator fed from `visit_object` inside `take_census`, which runs after `census_take_if_armed_at_full_sweep_start` has already taken PASS1_MARKED; it reads slots and the per-object layout maps (try_borrow), allocates nothing on the JS heap and alters no mark/sweep control flow. The window is unchanged. Re-audited 2026-09-27 for #11416: `gc/mod.rs` loses the `reg_scanner!` registration for `class_accessor_cache::scan_class_accessor_cache_roots_mut` (the cache is deleted); removing a root-scanner registration alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 for the method-calls lane, which adds one `reg_scanner!` registration (`scan_method_site_roots_mut`, marking and rewriting the method closures inherited method-site entries hold) to `gc/mod.rs`: a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-27 (class constructors as function objects, #11414): `gc/mod.rs` gains one `reg_scanner!` registration (`object::class_value::scan_class_value_roots_mut`, the per-agent class function-object table, visited and rewritten) \u2014 a root-scanner registration that alters no mark/sweep control flow and runs nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-29 after #11659 added `verify::verify_array_hole_tails_at_collection()` to `gc/mod.rs` and `gc/policy.rs`. Every call sits at a collection or budgeted-cycle ENTRY, beside `roots::ensure_stack_maps_built()`, i.e. before mark begins and so before `census_pass1_if_armed` opens the window. The call is a read-only walk of array headers (debug builds, or release with PERRY_GC_VERIFY_ARRAY_HOLES): it neither allocates GC memory, relocates, runs JS, nor reads or writes PASS1_MARKED, and it panics rather than continuing on a violation. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Pin re-audited 2026-09-29 after PR #11646 added one call inside that window, in `cycle.rs` beside `census_take_if_armed_at_full_sweep_start`: `object::shapes::store_kind::audit_heap_at_full_sweep_start()`. It compiles to nothing unless the `shape-fact-audit` feature is on; when on it walks the arena read-only (`gc::for_each_live_object_at_sweep_start`, in `gc/verify.rs`) and reads each marked object header and its shape record. It allocates no GC object, moves nothing, runs no JS callback and holds no address past the walk, so PASS1_MARKED stays valid across it; a disagreement panics rather than continuing. Re-audited 2026-09-29 after the pinned-roots fix touched `gc/cycle.rs` and `gc/mod.rs`: the block-persistence live-block predicate now reads GC_FLAG_MARKED alone (pinned objects are marked as roots, so the set of live blocks is unchanged), and `gc/mod.rs` adds one `reg_scanner!` registration. Neither relocates, runs JS, or moves the mark-complete to sweep-entry window. Re-audited again 2026-09-29 (same fix, copying-minor follow-up): `BlockPersistCycleState`'s force-mark in `gc/cycle.rs` no longer skips an unmarked pinned header, so it marks and pushes it like any other object of a live block. That is marking inside the budgeted cycle's persistence step: nothing relocates, no JS runs, and the budgeted path still skips both census boundaries. The window is unchanged. Re-audited 2026-09-29 for this-as-a-parameter stage 3: `gc/mod.rs` registers `scan_dispatch_binding_roots_mut` in place of `scan_implicit_this_roots_mut` (the implicit-`this` cell is deleted; the same scanner body keeps `new.target`, the static-`this` override and the static private-owner stack) and rewords its comment; a root-scanner registration, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-28 for the JsFunctionInfo lane: `gc/census.rs`'s `side_tables()` loses its `closure_registry_census()` row (the closure-body registry is deleted); a census report row, altering no mark/sweep control flow and running nothing inside the mark-complete to sweep-entry window. Re-audited 2026-09-29 for the read-site holder entry: `gc/mod.rs` adds one `reg_scanner!` registration, `read_holder::scan_read_holder_roots_mut`, which visits the holder and hop words of registered read-site caches during ROOT SCAN, before mark propagation completes. It rewrites root slots only through the visitor (as every registered scanner does), runs no JS, and nothing it does executes between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start`. The window is unchanged. Re-audited 2026-09-29 (#11549 trace cost): `gc/mod.rs` gains one `mod copying_object_scan;` declaration, the copying minor's plain-object slot walk. It is reached only from the copying drain, never from a full cycle, and changes no mark or sweep control flow. The window is unchanged.", "window": { "start": { "file": "crates/perry-runtime/src/gc/census.rs", @@ -379,7 +379,7 @@ "sources": { "crates/perry-runtime/src/gc/census.rs": "aed9e85c2c5cf17869f2e50408ca2dcaddb052aedea6d019431a9b57904dbda7", "crates/perry-runtime/src/gc/cycle.rs": "9b992d27700382c84758f6f4a5574115a9862234bed2e18e0c5a3cca25c25236", - "crates/perry-runtime/src/gc/mod.rs": "4f52b4b204f13ef144994ea5ace0372615e68bb920c9f76ba05bd12a53b17ced", + "crates/perry-runtime/src/gc/mod.rs": "927e1a83913a508ecda18f099973794b306b6d215c36124895e845ab31ad5407", "crates/perry-runtime/src/gc/policy.rs": "84a869e0aa09e932a4d2b6601186260129ba851b4482f29579047713c60fc95e", "crates/perry-runtime/src/gc/progress.rs": "a5ad3971bbe4047229ca57325234780daa85921dbc778e1c08dff4ad07ccfb96" } @@ -3145,6 +3145,10 @@ "file": "crates/perry-runtime/src/gc/copying.rs", "name": "UNTRACED_DECLINE_REASON" }, + { + "file": "crates/perry-runtime/src/gc/copying_object_scan.rs", + "name": "PLAN_ATTEMPTS" + }, { "file": "crates/perry-runtime/src/gc/cycle_malloc_trim.rs", "name": "TEST_MALLOC_TRIM_CALLS" diff --git a/scripts/shape_descriptor_census.py b/scripts/shape_descriptor_census.py index a4cecd0307..2143d138de 100644 --- a/scripts/shape_descriptor_census.py +++ b/scripts/shape_descriptor_census.py @@ -319,6 +319,7 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: "crates/perry-runtime/src/object/live_slots.rs", "crates/perry-codegen/src/lower_call/new_alloc.rs", "crates/perry-runtime/src/gc/layout_slot_visit.rs", + "crates/perry-runtime/src/gc/copying_object_scan.rs", "crates/perry-runtime/src/object/field_set_by_name/tail.rs", "crates/perry-runtime/src/typed_feedback/guards.rs", "crates/perry-runtime/src/object/native_call_method.rs", @@ -360,6 +361,7 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: live_slots = clean["crates/perry-runtime/src/object/live_slots.rs"] codegen_alloc = clean["crates/perry-codegen/src/lower_call/new_alloc.rs"] layout_visit = clean["crates/perry-runtime/src/gc/layout_slot_visit.rs"] + copying_object_scan = clean["crates/perry-runtime/src/gc/copying_object_scan.rs"] transition_tail = clean[ "crates/perry-runtime/src/object/field_set_by_name/tail.rs" ] @@ -485,19 +487,32 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: + ", ".join(sorted(scanner_slot_apis)) ) - layout_body = function_body(layout_visit, "visit_gc_layout_slot_descriptors") - require_code( - layout_body, - r"gc_shape_keys_edge_slot\s*\(", - "descriptor keys edge enumerated as a child slot", - ) - # Nothing in the visit reads the deleted mirror. The descriptor record is - # both the strong edge and the stable rewritable location. - if re.search(r"keys_array", layout_body): - raise CensusError( - "the GC slot visitor reads ObjectHeader::keys_array again; the " - "descriptor is the authoritative edge since #8112" + # #11549: the walk's ONE body is the generic `_inline` form (the `dyn` + # wrapper only forwards to it), and the copying drain's plain-object scan + # is a second enumeration of the same object slots. Both must emit the + # descriptor's keys edge and neither may read the deleted mirror. + for body_name, body in ( + ( + "visit_gc_layout_slot_descriptors_inline", + function_body(layout_visit, "visit_gc_layout_slot_descriptors_inline"), + ), + ( + "plain_object_plan", + function_body(copying_object_scan, "plain_object_plan"), + ), + ): + require_code( + body, + r"gc_shape_keys_edge_slot\s*\(", + f"descriptor keys edge enumerated as a child slot ({body_name})", ) + # Nothing in the visit reads the deleted mirror. The descriptor record + # is both the strong edge and the stable rewritable location. + if re.search(r"keys_array", body): + raise CensusError( + f"the GC slot visitor ({body_name}) reads ObjectHeader::keys_array " + "again; the descriptor is the authoritative edge since #8112" + ) # The insert/reverse-index body lives in the `_with_holes` variant since # the tombstone-delete work; `_with_generation` is a thin forwarding @@ -1176,6 +1191,18 @@ def run_sabotage_selftests(sources: dict[str, str], baseline: dict[str, object]) lambda: assert_authority_surfaces(header_fact_read), ) + plan_fact_read = dict(sources) + path = "crates/perry-runtime/src/gc/copying_object_scan.rs" + plan_fact_read[path] = plan_fact_read[path].replace( + "let keys_edge = crate::object::gc_shape_keys_edge_slot(shape);", + "let _mirror = (*obj).keys_array;\n let keys_edge = crate::object::gc_shape_keys_edge_slot(shape);", + 1, + ) + expect_rejected( + "copying plain-object scan reads the header mirror for a fact", + lambda: assert_authority_surfaces(plan_fact_read), + ) + inverted_publication = dict(sources) path = "crates/perry-runtime/src/object/shapes.rs" publication_body = function_body(