Skip to content

docs: record lessons 91 and 92 and two rules - #1378

Closed
dekobon wants to merge 2 commits into
perf/1375-nesting-map-lifetimefrom
docs/lessons-91-92
Closed

docs: record lessons 91 and 92 and two rules#1378
dekobon wants to merge 2 commits into
perf/1375-nesting-map-lifetimefrom
docs/lessons-91-92

Conversation

@dekobon

@dekobon dekobon commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Two lessons and two rules from the 2026-08-29/30 window. Both lessons
name a mechanism nothing in the file covers; both non-qualifying
takeaways were routed to rules instead, per the file's own "standing
obligations live in rules, not here" preamble.

Lesson 91 — a gate can filter out its own subject before the check runs

The belt-and-braces guard for RUSTSEC-2026-0258 would be a [bans] deny
on h2 <0.4.16. It is dead by construction: krates, cargo-deny's graph
builder, filters the crate out before the advisory and ban checks run
(-L debug logs filtered h2 0.3.27), on 0.19 and on the 0.20 that
CI's action bundles. Re-enabling actix-web/http2 under both versions
left cargo-deny silent — so the entry would have looked like coverage
and could never have fired.

What makes it worth an entry is the shape: a config entry naming the
crate and the version range reads as a guard to every future
maintainer, and the tool exits 0, which is exactly what it prints when
the tree is genuinely clean. The working guard reads Cargo.lock from a
test, and deny.toml carries a comment where the dead entry would have
gone.

Cited by commit hash rather than PR — both are direct pushes verified
reachable on main with no PR, which is the one case the skill's
guardrail permits.

Lesson 92 — an optimization's rationale can encode the waste it optimizes for

#1069 measured that the cognitive nesting map "converges on one entry
per visited node" and reserved descendant_count() up front to skip the
doubling chain. That is a genuine speedup — for growth that should not
have existed. The slot is dead after its children are seeded and was
never removed, so the reserve was sized to the leak: 540 MB of a
1,265 MB peak on a 28 MB generated parser.c, fixed in #1375 / PR
#1377.

The second mechanism is how it was found. All three hypotheses in the
issue (reorder buffer, path list materialized twice, glibc arenas) were
wrong, and code reading could not discriminate them because every
allocation suspected really is present. Bisecting by selection settled
it: dump measured 722 MB and every non-cognitive metric 723–725 MB
against cognitive's 1,264 MB, --output-dir matched stdout in peak, and
bca check — which has no reorder buffer at all — still peaked at
4.5 GB.

Routed to rules instead

Notes

  • Entries are 34 and 38 lines, inside the 45-line single-mechanism and
    ~60-line two-mechanism budgets. rumdl passes; both index anchors
    resolve.
  • Numbering appends at 91/92, so no existing citation moves.
  • Not done, and flagged rather than acted on: a one-clause merge of the
    parser_cache doc-claim instance into lesson 84 (one more instance of
    a mechanism it already documents with eight), and lesson 59's 81-line
    overrun against the 75-line stop.

@dekobon dekobon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: docs: record lessons 91 and 92 and two rules

Verdict: REQUEST CHANGES

Severity Count
HIGH 1
MEDIUM 3

Findings

# Severity File:Line Title Category
1 HIGH docs/development/lessons_learned.md:3603 This lesson currently lands before the fix it describes correctness
2 MEDIUM docs/development/lessons_learned.md:3573 Actix also supports plaintext HTTP/2 via H2C correctness
3 MEDIUM docs/development/lessons_learned.md:3604 The slot lifecycle omits the paths that needed separate freeing correctness
4 MEDIUM docs/development/lessons_learned.md:3614 This attribution paragraph sharpens the evidence past what #1375 established correctness

Review passes executed

  • Documentation correctness and logic
  • Factual verification against issue, PR, and source history
  • Documentation consistency and test-guidance review

Files reviewed: 3 changed files, plus the referenced source and issue history.

moves whether they are freed or kept.

**The nesting map's reserve was sized to the leak it should have
prevented** (#1069, fixed in #1375 / PR #1377). `NestingMap` carries one

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Location: docs/development/lessons_learned.md:3603

HIGH -- This lesson currently lands before the fix it describes

PR #1377 is still open, and #1378 is based directly on main rather than on perf/1375-nesting-map-lifetime. In this branch src/spaces/compute.rs still allocates NestingMap::with_capacity_and_hasher(node.descendant_count(), ...), still uses get in propagate_nesting_to_children, and has no nesting_slots_retained or final debug_assert!. If this merges first, both this lesson and the new testing rule state that the leak is fixed while main still contains the old behavior. Please stack/rebase this branch on #1377 or ensure #1377 lands first.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — you are right, and my split reasoning was wrong. I branched off main to keep the unrelated h2 lesson out of the code fix, but that traded a topical problem for a correctness one: merged first, both the lesson and the testing rule would describe a fix main does not contain.

Rebased onto perf/1375-nesting-map-lifetime and retargeted this PR's base to it, so #1378 is now stacked on #1377 and its diff is just the two docs commits. It cannot land first.

Comment thread docs/development/lessons_learned.md Outdated
alert #776; `3df3b2c3`, `b60943b5` — direct pushes to `main`, no PR).
The crate reached the tree only through `actix-web`'s default `http2`
feature, and no patched 0.3 release exists, so the fix was to drop the
feature: `bca-web` binds plaintext and actix negotiates HTTP/2 only over

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Location: docs/development/lessons_learned.md:3573

MEDIUM -- Actix also supports plaintext HTTP/2 via H2C

HttpServer 4.15 exposes bind_auto_h2c() and listen_auto_h2c() for plaintext HTTP/2; only bca-web's current .bind((host, port)) call keeps HTTP/2 unreachable here. Saying Actix negotiates HTTP/2 only over TLS ALPN makes the rationale false and would mislead a future switch to H2C. Please tie the statement to this daemon using .bind() rather than to Actix generally.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, and thank you — the claim was false and I inherited it from 3df3b2c3's own commit message rather than checking it.

Verified: bind_auto_h2c is at actix-web-4.14.0/src/server.rs:513 (listen_auto_h2c at 685), so actix does speak plaintext HTTP/2. The entry now ties the rationale to this daemon's plain HttpServer::bind at web/server.rs:303 instead.

One thing the check turned up that strengthens the fix: bind_auto_h2c is itself #[cfg(feature = "http2")], so dropping the feature removes the H2C entry point along with the vulnerable crate. That is now in the entry, since it means the guard does not rest on the bind call staying as it is.

Comment thread docs/development/lessons_learned.md Outdated

**The nesting map's reserve was sized to the leak it should have
prevented** (#1069, fixed in #1375 / PR #1377). `NestingMap` carries one
`Nesting` per node id, seeded by the parent, read once by the node's own

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Location: docs/development/lessons_learned.md:3604

MEDIUM -- The slot lifecycle omits the paths that needed separate freeing

Before #1377, a leaf slot was not read again during child seeding because propagate_nesting_to_children returned before looking it up when children.is_empty(), and an exclude_tests-pruned node skipped both compute and propagation. Those are exactly why #1377 removes the leaf bail-out and adds a separate prune-arm removal. As written, this says every slot had the same two reads and hides the edge cases the fix has to preserve.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The simplification hid exactly the two shapes the fix has to treat specially, which is the opposite of what the entry should leave a reader with.

It now names both: a leaf returned early on children.is_empty(), and an exclude_tests-pruned node reached neither compute nor propagation — so freeing in the last reader means dropping the leaf bail-out and adding a removal on the prune arm.

Comment thread docs/development/lessons_learned.md Outdated
the live set bounded by the traversal stack: 735 MB, and 3.3 s against
5.1 s.

**All three hypotheses in the issue were wrong** (#1375). It proposed

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Location: docs/development/lessons_learned.md:3614

MEDIUM -- This attribution paragraph sharpens the evidence past what #1375 established

#1375's body does not propose the twice-materialized path list as one of its three suspects; that came from #1158. Its follow-up says the body suspects were secondary, not all wrong, and specifically records MALLOC_ARENA_MAX=1 lowering the peak to 3.4 GB, so glibc arena retention was real even if it was not the dominant term. The same table reports stdout at 1,265 MB and --output-dir at 1,266 MB, not "byte-for-byte" identical. Please preserve those measured distinctions.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, all three. Checked each against the issue body and the follow-up:

Worth noting this finding is lesson 84 catching the entry that sits two above it in the same file.

@dekobon
dekobon force-pushed the docs/lessons-91-92 branch from 64fa25e to 57af48e Compare August 30, 2026 21:37
@dekobon
dekobon changed the base branch from main to perf/1375-nesting-map-lifetime August 30, 2026 21:37
Two mechanisms from the 2026-08-29/30 window that nothing in the file
named, plus the two takeaways that are standing obligations rather than
bug classes.

Lesson 91 — a gate can filter out its own subject before the check
runs. The `[bans] deny` on `h2 <0.4.16` that RUSTSEC-2026-0258 invites
is dead by construction: krates drops the crate from the graph before
cargo-deny's advisory and ban checks run, on 0.19 and on the 0.20 that
CI's action bundles, so the entry stays silent even with
`actix-web/http2` re-enabled. The lockfile test is the guard that
actually fails.

Lesson 92 — an optimization's rationale can encode the waste it
optimizes for. #1069 measured that the cognitive nesting map converges
on one entry per visited node and reserved that much up front, which is
a real speedup for growth that should not have existed: 540 MB of a
1,265 MB peak on a 28 MB generated `parser.c`. It carries the
attribution technique as its second mechanism, because all three
hypotheses in #1375 were wrong and only bisecting by selection settled
it.

`AGENTS.md` gains the issue-tracker form of lesson 84: re-verify a
deferred issue's premises, since #1158 proposed an ordering contract
that #1303 had shipped the opposite of and its own go/no-go measurement
had never been run.

`.claude/rules/testing.md` gains the whole-run-invariant rule: when an
invariant holds at the end of every execution of a hot path, a
`debug_assert!` there recruits the whole corpus as input, which is a
different order of coverage from any fixture list. Both guards, and
keep the assertion O(1) — a per-node one is how the exact ancestor
check went quadratic.
Ties the h2 rationale to this daemon's plain `HttpServer::bind`
rather than to actix, which does speak plaintext HTTP/2 through
`bind_auto_h2c` — itself gated on the feature being dropped.

Names the two slot shapes that skip the last read, since the leaf
bail-out and the `exclude_tests` prune arm are exactly what the fix has
to handle separately.

Corrects the attribution paragraph: the twice-materialized path list was
#1158's suspect, not #1375's; arena retention was real but not dominant
(`MALLOC_ARENA_MAX=1` reached 3.4 GB); and `--output-dir` came within a
megabyte of stdout, 1,266 against 1,265 MB, rather than matching it.
@dekobon
dekobon force-pushed the docs/lessons-91-92 branch from 57af48e to 9c93e05 Compare August 30, 2026 21:48
@dekobon
dekobon deleted the branch perf/1375-nesting-map-lifetime August 30, 2026 23:33
@dekobon dekobon closed this Aug 30, 2026
@dekobon

dekobon commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

Landed on main — GitHub shows this as closed rather than merged, which is an artifact of how it went in, not of the work being dropped.

main had not moved from the branch point, and the repo ruleset protect-main requires required_signatures and required_linear_history, so neither merge button was usable: GitHub refuses a rebase merge outright ("Rebase merges cannot be automatically signed by GitHub") and a merge commit violates linear history. A fast-forward satisfies every rule and keeps the original GPG signatures, so the stack went in that way:

9c93e053 sig=G docs(lessons): correct four review findings in 91 and 92  <- this PR
632345d8 sig=G docs: record lessons 91 and 92 and two rules              <- this PR
6424a983 sig=G docs(ci): scope the RSS rule to the in-flight term        <- #1377
03cb037f sig=G perf(metrics): bound the cognitive nesting map            <- #1377

#1377 registered as merged because its head was reachable from its base. This PR could not: its base was the stacked branch, and once main and docs/lessons-91-92 became identical GitHub refused to retarget it ("There are no new commits between base branch main and head branch"). Deleting the merged parent branch then closed it.

Both commits from this PR are on main and signed. All four review findings were addressed in 9c93e053.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant