You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Nothing executes the code a spec generates, so correctness is unmeasured #3903
Nothing in the pipeline has ever executed the code a spec generates.
A spec is accepted when a bee asserts its acceptance criteria are met and the
Queen agrees. Every criterion the review could judge mechanically — t27c parse, parse-complete, typecheck, empty-body count — asks whether the spec PARSES.
None asks whether what it generates WORKS. queen food counts empty function
bodies and reports the number as progress, which scores a body full of wrong
code above an empty one.
What that has cost
Measured 2026-09-16 by generating all 946 specs to Zig and running zig test:
541 pass — compile and pass their own tests
384 do not compile at all
21 compile, run, and hit @panic("not yet implemented")
That distribution was unknown and unmeasurable before the measurement existed.
Acceptance criteria
tools/oracle/run.sh generates every spec into a tree mirroring specs/ and
runs each under zig test
The oracle reports the file each error is IN, not the file under test
tools/oracle/baseline.tsv records the passing set and the run fails when a
spec that used to pass stops passing
The run does not fail merely because a spec has never passed
No ## Boundary on purpose: the work is already done in #3882. A boundary here would let a bee claim it and reimplement what is already in review.
Activity