The run-scan shim reported exit 0 on a file it never analysed; and a switch arm that ate 14 tests #5271

Re-filed from #2710. An attempt claimed that issue on 2026-09-24 and left an empty branch, and a dispatch row that has spent its retry ceiling keeps its issue for good - so no bee could take it again, however much the brief improved. The work below is carried over unchanged.

The instrument was unsound

formal/zig_run_scan.py compiled each spec through comptime { _ = @import(f); }. That shim was introduced in #2700's iteration to stop the shim's own test block inflating the assertion count by one per spec, and it does that -- but a comptime reference does not force full analysis.

For specs/lsp/language.t27, 14 tests in the emitted file:

comptime shim -> exit 0, "All 0 tests passed"
test shim     -> 26 own-file errors

A shim that can answer "fine, nothing to run" about a file it never analysed is not an instrument. Both zig_run_scan.py and check_one_spec.py now use test { _ = @import(f); } and subtract the shim's own test.

Re-measured, and the published figure survives:

The only spec affected was counted as passing with zero tests, so it contributed nothing to the assertion total. 888 was right -- but that could not be known without checking, and the same defect on a spec with tests would have been silent.

A switch arm that ate a whole file

.module => "module", in symbol_kind_to_string made the parser abandon that function, and recovery then consumed the rest of the file: 14 real behaviour tests, 12 invariants and 5 benches never reached the output. The only whole-file kill of this class in 497 specs.

Two sites had to change together, verified by an agent before either was written: the switch-arm pattern at parse_switch_expr and the enum-member name in parse_enum_body. With the arm alone the emitted Zig still fails with enum 'SymbolKind' has no member named 'module', because the member is dropped independently.

specs/lsp/language.t27 now emits 14 tests where it emitted 0. It still does not compile -- 26 type errors underneath, pub const LANG_T27 : [3]u8 = "t27" and its neighbours -- but those were always there, behind a wall.

Measured: ast-check 290 valid / 594 errors unchanged, 0 specs worse, parse gate holds at 105 events / 0 lost.

An audit of every TokenKind::Ident-only name position found 34 sites, and counted the corpus against each: 23 struct field names, 8 enum members, 5 struct-literal fields, 1 switch arm, 0 bare switch arms. The remaining sites are open.

The enum rewrite: refuted, by execution

The hypothesis was that const X = struct { enum : [A, B] } unambiguously means an enum, so the emitter could rewrite it. An adversarial agent refuted it and did not stop at argument:

• specs/tools/schema.t27:46 -- ParameterSchema is an ordinary record with a legitimate field named enum whose bracket list is a type, not a variant list.
• 27 use sites, including five specs whose tests pass today by naming these declarations as structs: @typeInfo(X).@"struct".fields.len, @FieldType(X, "Ascending"), std.meta.fields(X).len.
• It applied the proposed rewrite to specs/tri/sort/sort.t27 and turned a 2/2-passing module into two hard compile errors.

Those pinning tests were written by my own agents last iteration. Pinning a declaration cements its defects -- worth stating plainly, because the same trade will recur.

The census also found ten distinct shapes, not two. A safe subset exists and is not done here: the enum_type marker (4 declarations) and exit_codes (1) emit invalid Zig today, so nothing can depend on them, and their variant names are present in the source.

Original file list

• specs/lsp/language.t27
• specs/tools/schema.t27
• specs/tri/sort/sort.t27

The instruments

Every one of these reads and prints; none of them writes. Run them from the repository root, with <spec> replaced by the file in ## Boundary:

.t27 is not Rust and not Zig, and a bee fluent in either writes one by
accident. Measured 2026-09-20: a bee filled eight bodies in
specs/file/watcher.t27 with return Ok(());, Err(FileError::WatcherNotFound)
and for i in 0..watchers.length. The parse ratchet refused the whole file --
Unexpected token in expression: RParen, line 123 -- and a spec that does not
parse generates nothing, so every test it already carried stopped running.

At the top level the parser accepts exactly these eight forms, with an optional
pub, and no others:

const   var   fn   enum   struct   test   invariant   bench

There is no trait, no impl, no type X = ..., no generics, no macro. What
a bee reaches for from another language, and what happens:

• Rust's result sugar (Ok(()), Err(E::V), ?) is not a construct here.
• Rust macros (println!, format!, vec!) are not constructs here.
• A range loop (for i in 0..n) is Rust and Zig, not this language.
• Zig builtins (catch unreachable, @intCast, @constCast) are not constructs here.
• Cross-module reuse (use other::fn;) parses and then generates a comment and an unqualified call, so the Zig fails with use of undeclared identifier (use a::b; generates no import, so no spec can reuse a function from another spec #4298). It does not work yet.

t27c parse <spec> answers in one line whether what you wrote is the language.
Run it before you report, every time.

• t27c spec-status <spec> - the compiler's one-word verdict: IMPLEMENTED, PARTIAL, UNWRITTEN, NOPARSE, NOFN. Exit code is 0 whatever it says, so read the word.
• t27c symbols <spec> - every name the file declares, with its kind. Answers "does this already exist here?" before you add it.
• t27c outline <spec> - per function: its locals, what it calls, what it returns. The contract you are implementing against.
• t27c coverage <spec> - which functions have a test and which do not. The issue asks for tests; this is how you check you wrote them.
• t27c lint <spec> - style and shape warnings. It OVER-REPORTS has no test or invariant: measured 2026-09-20 over 60 specs it printed 677 of those where coverage found 190 untested functions, disagreeing on 59 of the 60 - it warns about bit_to_trit_pair in specs/base/ternary_encoding.t27, which test bit_to_trit_pair_zero calls on line 249. Read it as a hint; coverage is the answer.
• t27c typecheck <spec> - types, before generation. Prints Typecheck OK (0 errors, 0 warnings) or the errors.
• t27c test-report <spec> - builds this spec and runs its own tests, which is what the oracle does. BLOCKED means the generated Zig does not compile, with the error beside it.
• python3 tools/dupe_scan.py --name <function> - where that function already lives, if it does. 576 of 4021 bodies here are byte-identical copies.

The rest of the toolbelt is in docs/BEE_TOOLBELT.md.

Queen-runnable restatement (2026-10-05)

Measured on master 13e212a: two of the three Boundary files are settled. specs/tools/schema.t27 keeps its enum_values field, and specs/tri/sort/sort.t27 declares SortOrder as a real enum(u8) with a passing test. The enum_type marker shape is gone from specs/, and specs/tri/utils/exit_codes.t27 passes 3 of 3. The third file went backwards. specs/lsp/language.t27 is now a 14-line stub: a use line, an empty main and // TODO: Implement the LSP language specification. t27c spec-status calls it UNWRITTEN, and it has 0 tests (the merge of #4498 on 2026-10-04 replaced 525 lines). The 14 tests this issue reported were never on master in a form that compiled; the last copy, on branch fix/reject-vibee-specs (commit 8a5f0ea), is NOPARSE. The remaining work is a language.t27 that parses, generates and passes its own tests.

Boundary

• specs/lsp/language.t27

User Scenarios & Testing

• Given the stub, when a bee writes the spec, then it declares the language-ID constants (t27, zig, python, javascript, typescript, markdown, json, yaml) as str values rather than fixed [N]u8 arrays initialised from string literals, which is the type error this issue measured.
• Given an extension, when language_id_from_extension is called with "t27", "zig", "py" and an unknown extension, then it returns "t27", "zig", "python" and a stated fallback, and a test checks each case.
• Given the SymbolKind enum including a module member, when symbol_kind_to_string is called on each member, then it returns that member's name, and a test checks module in particular (the switch arm that once consumed the whole file).
• Given is_t27_extension and is_t27_language, when they are called on matching and non-matching inputs, then tests check both answers.

Requirements

• FR-001: specs/lsp/language.t27 MUST parse, typecheck and pass its own tests under t27c test-report, with no BLOCKED result.
• FR-002: The spec MUST carry at least 10 test blocks with concrete expected values, and no TODO, placeholder or always-true body.
• FR-003: The change MUST NOT add or modify non-t27 hand-written code (specs/policy/own_language.t27); only the .t27 file and any t27c-generated gen/ output may change.

Success Criteria

• t27c spec-status specs/lsp/language.t27 prints IMPLEMENTED
• t27c test-report specs/lsp/language.t27 does not print BLOCKED
• grep -cE '^[[:space:]]*test ' specs/lsp/language.t27 prints at least 10
• grep -c 'TODO' specs/lsp/language.t27 prints 0
• grep -c 'module' specs/lsp/language.t27 prints at least 2