diff-check: rank items for review use, so the bug site makes the top 15 #737

Activity

1. zaebee

   added

   tailLoose end: something left to check or finish laterLoose end: something left to check or finish later

   on Oct 22, 2026
2. zaebee commented on Oct 22, 2026

   

   zaebee

   on Oct 22, 2026

   OwnerAuthor

   More actions

   Pre-registration (before any rank is looked at)

   Data. The 9 outside PRs of the #733 holdout, the ones whose sealed bug site lies outside the diff. For each, diff-check is run uncapped on a data-key graph at the PR's merge commit. The data is local because the corpus is private, and only aggregates are reported here.

   Measures.

   • Ceiling: is the sealed site in the uncapped list at all? A site counts as found when an item's site ends with the sealed symbol and its file matches.
   • For each ranking below: the site's rank, and whether it is in the top 15.

   Rankings, fixed now. No new ones are added after looking.

   • R0, current: score, then the fixed kind order.
   • R1, calls first: caller and subclass items before key items, then R0.
   • R2, keys first: key items before callers, then R0.
   • R3, proximity: items in a directory that contains a changed file first, then the parent directory, then the rest; within each group, R0.
   • R4, one line per site: collapse items that share a site, keep each site's best item, then R0.
   • R5: R3 and R4 combined.

   Decision.

   • A ranking is a candidate if it puts the site in the top 15 for ≥ 5 of 9 (current: 2 of 9).
   • Nine PRs is small, and these rankings are chosen with knowledge of the domain, so a candidate is not adopted on this result alone. It needs a fresh sealed outside-diff holdout and then the feat(guardian): give the finder diff-check's dependants as context (flagged, pre-registered) #733 review protocol.
   • If the ceiling is low (site in the uncapped list for < 5 of 9), no ranking can pass. The finding is then about diff-check's recall, not its order.
3. zaebee commented on Oct 22, 2026

   

   zaebee

   on Oct 22, 2026

   OwnerAuthor

   More actions

   Result: ranking is not the bottleneck, recall is

   Ceiling. In the uncapped diff-check list, the sealed site appears for 2 of 9 outside-diff PRs. No ordering can do better than that, and none did:

   ranking	site in top 15
   R0 current, R1 calls first, R3 proximity, R4 one line per site, R5 R3+R4	2 / 9
   R2 keys first	1 / 9 (pushes one site from rank 2 to 28)

   Uncapped list sizes were 0 to 187 items. I checked the misses by hand for matching errors: the site's file appears among the items for some, but the site's symbol never does.

   This corrects my #733 note. There I read the misses as "cut by the cap". The uncapped run shows the sites are not in the list at all.

   What the 7 missing sites are, as categories from the sealed mechanisms (details stay local):

   category	n	what diff-check would need
   reads a module-level name the PR changed: a constant, or a rebound logger	2	readers of a changed module-level assignment, as items
   a sibling writer of a model that gained a field, still building it without that field	1	other constructors and writers of a model whose fields changed
   handles a value set the PR extended (new event names reach a dispatch that does not know them)	1	sites that switch on or test membership in the changed set
   a data reference the key layer does not model: a templated index name, or rows of a new shape read by an unchanged reader	2	f-string / templated index names; shape changes on the write side
   lifecycle semantics outside the graph: a cascade delete strands external objects	1	none reasonable

   Verdict, as registered. The ceiling is below 5 of 9, so no ranking is a candidate and nothing changes in diff-check's order. The leverage is in what diff-check finds at all.

   The first two rows (3 of 7) are structural and testable without a model: a changed module-level name, and a model with new fields. They are filed as a follow-up.
4. zaebee commented on Oct 22, 2026

   

   zaebee

   on Oct 22, 2026

   OwnerAuthor

   More actions

   Closed with the result above. Ranking cannot help: the ceiling is 2 of 9. Recall work continues in the follow-up.
5. zaebee

   closed this as completedon Oct 22, 2026