Repository navigation
lexer: \xNN in a string lowers to one byte, and every backend writes the byte back - #10812
Merged
Merged
Conversation
…s the byte back (Closes #10793) The string lexer kept "\xD0" as four characters, so a test such as assert "\xD0\x96".len == 2 failed at run time with no compiler diagnostic. It now lowers a backslash, x and exactly two hex digits to one byte, held as the char of the same value (U+0000..U+00FF). Any other \x keeps its old spelling. zig_escape writes bytes 0x00-0x1F and 0x7F-0xFF back as \xNN for Zig and Rust, or three-digit octal for C (a C \x escape swallows following hex digits) and Verilog. Rust writes non-ASCII text as \u{..} when the bytes are valid UTF-8 and emits compile_error! otherwise, since a str cannot hold a lone high byte. The Verilog escape chain collapses into zig_escape. specs/compiler/string_escapes.t27 pins the bytes; terminal.t27 stops claiming the escape is not lowered. The dead recover_to_stmt_boundary is deleted so the hand-written Rust shrinks (36 lines net -11). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…un at run time (refs #10793) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…0793) logging.t27 spells "\x1b[36m" and now emits the ESC byte; terminal.t27 changed a comment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… bytes (refs #10793) The old backends wrote each source byte of a non-ASCII literal as its own UTF-8 char (double encoding: S3 with a superscript became two mojibake chars). Zig and Rust now write the bytes back, so the gen hashes move. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
…0793) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
The lexer holds a byte as the char of the same value, so the literal's length and bytes are the chars, not the UTF-8 of those chars. The t27b-native-ratchet job failed string_escapes zhe_is_two_bytes on it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
) The ratchet runs the new spec and t27b passes it (one byte per char of the lexeme since ac3f5ae), so the ledger names it. The other UNLISTED rows in the run are master drift for the ledger steward and are not touched here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10793
What
The string lexer now lowers
\xNN(exactly two hex digits, either case) to one byte. Before,"\xD0\x96"stayed eight characters and the compiler said nothing, which blocked UTF-8 ideas for the Cyrillic idea factory (t27#10788). A\xthat is not followed by two hex digits keeps its four characters.The lexeme holds a byte as the char of the same value (U+0000..U+00FF), so every backend that writes a string must write that char back as ONE byte:
\xNN.\xswallows the hex digits after it).&strcannot hold a lone byte above 0x7F, so valid UTF-8 is written with\u{..}escapes and anything else becomescompile_error!("string literal is not valid UTF-8").Side effect worth knowing: the old Zig and Rust output double-encoded literals that already held real non-ASCII text (for example the em dash in
specs/crons/trinity-brain-ci.t27came out as mojibake). That is fixed by the same change, and it moves the gen hashes of 28 specs, so their seals are refreshed here (.trinity/seals/*, tool-written).Files
bootstrap/src/compiler.rs: lexer arm,zig_escape(raw, octal), newrust_str_literal, four call sites; the deadrecover_to_stmt_boundaryis deleted so the foreign-code budget nets to -17 lines.bootstrap/src/lex_conform.rs: one conformance row.bootstrap/stage0/FROZEN_HASH: moved for the compiler.rs change.specs/compiler/string_escapes.t27(new): 7 tests, including"\xD0\x96".len == 2with bytes 208 and 150.specs/tri/utils/terminal.t27: comment updated.compiler_StringEscapes.json, refreshedTriLogging/TriTerminaland 28 more.Verified on the Railway lab (nothing run on the Mac)
t27c test-reportonspecs/compiler/string_escapes.t27: 7/7 pass, 0 vacuous (the literals go through atext_ofcall so the asserts run at run time).t27c lex-conform: 35/35.t27c seal --verifyon every touched spec: all hashes MATCH.t27c suite --repo-root .: the set of failing checks (254) is identical to the merge-base binary's; the suite already reports "ACCEPTABLE: no" on the base because of existing failures."\xD0\x96": Zig\xd0\x96, C\320\226, Verilog octal, Rust\u{416}; invalid UTF-8 givescompile_error!.Not done
\xNNtext).specs/port/trinity/src/tri/tri_init.t27writes"\x1b[36m"and now emits the ESC byte; it has no seal and its gen-verilog failure is older than this change.🤖 Generated with Claude Code