Skip to content

t27o: a declaration index, a line cache and a bounded for-search make tri t27o linear in the file; compiler.rs finishes in 1.3 s (Closes #10544) - #10631

Merged
gHashTag merged 6 commits into
masterfrom
t27o/linear-lookups-10544
Oct 18, 2026
Merged

t27o: a declaration index, a line cache and a bounded for-search make tri t27o linear in the file; compiler.rs finishes in 1.3 s (Closes #10544)#10631
gHashTag merged 6 commits into
masterfrom
t27o/linear-lookups-10544

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Stacked on #10611 (branch t27o/check-defects-10534), which is itself on #10593: the diff below holds their commits until they land, then only the six commits of this lane. Lane P of the epic #10522.

What

tri t27o found a struct, a const or a fn by walking the file token by token from byte 0, once per expression that needs a type, in each of up to 16 passes: quadratic in the file. This change builds one index per conversion and reads it.

  • prescan calls index_decls, which walks the file once and enters, per name, the position of every keyword token that has that name next to it, in file order: struct enum type trait union const static (also by the name after mut), fn, the typed lets, and the impl blocks (with the end of each inherent block). The index lives at the top of the state vector (buckets first, entries below, four slots each); the name table stops where it begins (add_name). The glue cli/tri/src/main.rs is not touched: its 65536-slot vector holds the index of compiler.rs (about 4400 entries; with the index off that file would not have finished in seconds).
  • find_struct, const_type, fn_ret_type read it, so do the same-kind rescans the profile showed next: ret_class, let_owner, method_kind/method_owner, impl_around, struct_open, declared_fn, declared_type. Each runs the very check its old walk made (struct_hit, const_hit, fn_hit, let_at, kind_at, impl_owner_at, item_named, ...) on the entries of the name only, in file order, so the first match wins exactly as before (a fn g in an impl that shadows the free fn g, the last let p before the use, the first impl that has the method).
  • With the index off (IXON 0: a file with more declarations than the vector has room for, or a vector of fewer than 120 slots) the same checks run over every token of the file, as before. The index takes at most half of the free slots.
  • Two more rescans the profile reached: put_refusal/put_reason counted newlines from byte 0 for every refusal (line_at keeps the last position in the slots a pass clears), and impl_item/impl_header searched for for past the impl's { to the next top-level for of the file (find_tok_to stops at the brace).

Only specs/tools/t27o/rust_convert.t27, its generated gen/rust/tools/t27o/rust_convert.rs (regenerated) and .trinity/seals/t27o_T27oRust.json change.

Profile (macOS sample, 1 ms, top of stack, top 8)

Before, tri t27o cli/tri/src/skillnum.rs on the base (#10611), 367 samples: tok_end 147, fn_ret_type 39, const_type 37, op3 25, quoted_end 22, op2 21, callee_class 20, postfix 16. Inclusive, const_type + fn_ret_type + find_struct are 214 of 367 (58 %).

After, bootstrap/src/compiler.rs (skillnum.rs now ends in 0.02 s, too short to sample), 816 samples, 2.1 s: tok_end 228, deps 141, empty_arg 61, named 59, strloc_in 44, op3 34, op2 33, block_end 22. No declaration lookup is left in the top 8; deps, named and strloc_in scan the name table, which is linear in the names, not in the file.

How the stages moved bootstrap/src/compiler.rs (1.9 MB):

stage compiler.rs
base (#10611) killed at 600 s, no output
first index only (struct, const, fn first match) killed at 560 s; gates.rs 11.0 s -> 5.1 s; next frames: let_owner, callee_class/ret_class, method_owner inlined in postfix
keyword-position index (the list above) 16 s; 80 % in line_of
line cache 3.5 s; 60 % in find_tok from impl_item
find_tok_to 2.1 s under load, 1.3 s quiet

compiler.rs finishes. Its count line: // t27o: 1766 fn seen, 109 converted (39 of 702 tests), 1657 refused (21 kept as stubs); coverage 6%.

Timing (wall, same machine, base and new run back to back, best of two, load average 4.6 to 6.7)

file bytes before (s) after (s)
cli/dlc10/src/lib.rs 94,077 0.59 0.030
cli/tri/src/skillnum.rs 95,804 0.84 0.024
cli/tri/src/mutate.rs 161,298 1.95 0.037
bootstrap/src/main.rs 509,113 0.77 0.051
bootstrap/src/service.rs 289,445 1.04 0.040
cli/tri/src/gates.rs 384,942 11.0 0.15
cli/tri/src/fpga.rs 419,926 7.5 0.064
cli/t27b/src/lower.rs 368,568 4.8 0.076
bootstrap/src/compiler.rs 1,902,640 not finished at 600 s (killed) 1.3 (2.1 under load)

The base is faster here than the figures in the issue (dlc10 5.9 s there, 0.59 s here), so the ratios, not the seconds, are what carries over.

Output is byte-identical

tri t27o <file> and tri t27o --oracle <file> on the base binary and on this one, cmp on stdout and the exit code (60 s cap, 240 s for the five files above 300 KB):

  • 481 of the 482 tracked .rs files (every one except compiler.rs): 962 comparisons, all identical, all exit 0, none timed out.
  • compiler.rs: the base never finishes, so there is nothing to compare it with. Four prefixes cut at a line (300, 600, 900 and 1200 KB) are identical in convert and in --oracle mode (8 comparisons, the base finished each within 240 s).

Tests (t27b test: 101 of 101; t27c test-report: 101 pass, 0 fail, vacuous passes 0 of 101; seal verify: all hashes MATCH)

  • the_declaration_index_answers_the_three_lookups_as_the_scans_do: a text with a shadowed struct, a shadowed fn (the impl's g has no ->, the free g does), static mut, const fn: all_names_agree runs every lookup for every name of the text with the index on and off, and the answers are pinned (first struct A has x, the free f returns u32).
  • negative_control_names_that_only_look_declared_are_not_in_the_index (unit and tuple structs, a struct in a comment or a text, fn(u32) -> u32, a const asked for as a fn), negative_control_a_wrong_entry_makes_the_lookup_disagree_with_its_scan (point an entry into the middle of its keyword and the check fails, so it can fail).
  • a_file_with_more_declarations_than_fit_leaves_the_index_off_and_the_scans_answer, the_name_table_stops_where_the_index_begins (300 names pushed at a table that stops at the index, the index stays whole), a_conversion_reads_the_same_with_the_index_on_and_off.
  • the_index_names_the_impl_that_owns_a_method_and_the_last_let_that_types_a_name (method in two impls with and without a receiver type, let p shadowed, the type after for), plus line_at (agrees with line_of going forward, back and past the end), find_tok_to and a negative control for each.

Not done here

deps, named, strloc_in scan the name table per identifier (linear in the names; next in the profile after this change). find_enum/enum_variant (the oracle's sampling), slice_param_elem and three line_of callers outside the refusal path still walk the file; none is in the profile of a conversion.

Closes #10544

🤖 Generated with Claude Code

This was referenced Oct 18, 2026
Merged
Closed
Open
Merged
Merged
@gHashTag
gHashTag enabled auto-merge (squash) October 18, 2026 20:16
@gHashTag
gHashTag force-pushed the t27o/linear-lookups-10544 branch from 62561ed to 77b82ad Compare October 18, 2026 20:17
@gHashTag

gHashTag commented Oct 18, 2026

Copy link
Copy Markdown
Owner Author

Review (orchestrator, Opus):

This was referenced Oct 18, 2026
Open
Closed
This was referenced Oct 18, 2026
Merged
Draft
Merged
@gHashTag gHashTag mentioned this pull request Oct 18, 2026
Closed
This was referenced Oct 18, 2026
Merged
Merged
Draft
Merged
@github-actions

github-actions Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 20:43:57 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 46
PRs with All Checks Green 4
READY 2
FAILING 46
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 46 + 0 + 0 = 48, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag gHashTag mentioned this pull request Oct 18, 2026
Merged
claude and others added 6 commits October 19, 2026 03:47
@claude
t27o: the declaration index -- struct, const and fn lookups read a ha…
6daf338 
…sh built once per conversion (Closes #10544)

find_struct, const_type and fn_ret_type each walked the file token by token from byte 0 on every
call, once per expression that needs a type: quadratic in the file. prescan now builds one index
(index_decls) at the top of the state vector -- buckets first, entries below, the name table stops
where it begins -- holding the first declaration of every struct X {, const|static X : and fn X(..).
The three lookups read it (ix_get); the scans stay as *_scan, the answer when the file has more
declarations than the vector has room for, and the check the index is built against.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude
t27o: tests pin the declaration index against the scans, with shadowi…
fdecaec 
…ng, an overflow and a spoiled-entry control (Closes #10544)

lookups_agree / all_names_agree compare find_struct, const_type and fn_ret_type with their *_scan
versions for every name a text spells. Six tests: the answers on a text with shadowing (first struct A,
the free fn f, the impl's g without a return type), names that only look declared (unit and tuple
structs, comments, texts, fn pointer types), a spoiled entry that the check catches, a file with more
declarations than fit (index off, scans answer), the name table stopping at the index, and a
conversion that reads the same with the index on and off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude
t27o: the index holds keyword positions by name; ret_class, let_owner…
be17cce 
…, method_owner, impl_around, struct_open and the declared_* checks read it too (Closes #10544)

The profile after the first index (gates.rs): callee_class -> ret_class 27 %, let_owner 17 %, method_owner /
method_kind (inlined into postfix) the rest -- each a walk of the file from byte 0 per call, the same
kind of rescan. The index now stores, per name, the position of every keyword token that has the name
next to it (struct enum type trait union const static; fn; typed lets; the impls in file order, with the
end of each inherent block), and every lookup runs the check of its old walk on those tokens only:
const_hit, fn_hit, struct_hit, item_named, let_at, kind_at, impl_owner_at. With the index off the
same checks run over every token of the file, so the answer cannot differ. bootstrap/src/compiler.rs
(1.9 MB), which did not finish in 9 minutes, now finishes in 16 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude
t27o: refusal lines come from a cached line count, and an impl header…
abbd563 
… looks for its for only before its brace (Closes #10544)

After the index, the profile of compiler.rs was two more rescans: put_refusal / put_reason counted the
newlines of the file from byte 0 for every refusal (line_of, twice a refusal, in each of up to 16
passes) and impl_item / impl_header searched for `for` with find_tok, which ran on to the next
top-level `for` of the file once per impl. line_at keeps the last position and its newline count in
the state (the slots a pass clears) and counts on from there; find_tok_to stops at a bound, and the
two impl headers bound it at their `{`. compiler.rs 15.8 s -> 2.1 s, output byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude
t27o: a test pins the impl that owns a method and the last let that t…
8a3a75a 
…ypes a name, index on and off (Closes #10544)

method_owner picks the first inherent impl with a method taking self (P), or the receiver's own
(Q); let_owner gives the last `let p` before the use (Q, not P); impl_around gives the type after
`for`. all_names_agree runs the same text with the index on and off for every name and every
method-name / receiver-type pair.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude
t27o: re-seal rust_convert.t27 (101 of 101 tests, 0 vacuous) (Closes #…
079522e 
…10544)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag force-pushed the t27o/linear-lookups-10544 branch from 77b82ad to 079522e Compare October 18, 2026 20:47
@github-actions github-actions Bot mentioned this pull request Oct 18, 2026
Draft
@github-actions

github-actions Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 20:55:15 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 43
PRs with All Checks Green 7
READY 2
FAILING 43
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 43 + 0 + 0 = 45, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions github-actions Bot mentioned this pull request Oct 18, 2026
Merged
@gHashTag
gHashTag merged commit 8b7263a into master Oct 18, 2026
31 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

t27o: quadratic run time -- find_struct, const_type and fn_ret_type rescan the whole file per expression (40 % of samples in tok_end)

2 participants

@gHashTag @claude