Skip to content

test: cover every bundled route in the current routing corpus - #228

Merged
wheakerd merged 1 commit into
mainfrom
feature/routing-corpus-coverage
Oct 30, 2026
Merged

wheakerd merged 1 commit into
mainfrom
feature/routing-corpus-coverage

Conversation

@wheakerd

Copy link
Copy Markdown
Owner

Scope

Refs #217. The current routing corpus evals/routing-v6/current.jsonl had one case each for clarify-intent, delegate-simple-task, task-planning, agents-architect, optimize-agent-usage, review-axiom-task, confirm-external-action, traceable-git-submit, and reversible-system-change, and the fixed per-route workload under evals/routing/ has no case at all for five routes. This change adds thirty-four schema v6 cases to the current corpus, outside both benchmark profiles, so that every bundled route has a positive, a near-miss no-route control, and a cross-route control, with Chinese positives for task-planning and local-web-search. Case IDs are new; no existing case changed, so the contract version stays 6 and the records bound to the v6 profiles stay verifiable. Benchmark membership for the new cases and the changed expectations for the routes the next installed-runtime release narrows belong to that release's successor contract.

The evaluation guide records the addition and the decision that the per-route workload stays frozen as the routing-context budget's fixed workload, with its retired optimize-codex-usage name mapped at validation time. The current-corpus tests take the new totals and no longer pin the number of verify-completion cases. Repository policy revision 115 records the change; the plugin version and runtime contract digest are unchanged.

Affected files and surfaces

  • Shared files: None
  • Codex-specific files: None
  • Claude Code-specific files: None
  • Documentation or validation files: evals/routing-v6/current.jsonl (34 added cases); evals/README.md; tests/test_current_routing.py, tests/test_routing_evals.py, tests/test_validator_isolation.py (totals); CHANGELOG.md, evidence/repository-policy-revisions-v1.json, evidence/runtime-identity.json, evidence/release-status.json, docs/runtime-identity.md (revision 115)

Routing and authorization impact

  • Route-selection impact: None. Evaluation cases describe expected selection; they change no skill text.
  • Action-authorization impact: None. Every case keeps mutationAuthorized false.
  • New or changed stop conditions: None.

Documentation

evals/README.md (Current Contracts and Contract changes) describes the added coverage and the frozen fixed workload. Installation and hook commands are unchanged.

Validation

Command Result
python3 -B scripts/check-publication.py Passed on this exact tree: 174 black-box routing cases, 66 fixed host benchmark cases, 153 required files, 30 immutable external action and image pins
python3 -B -m unittest discover -s tests -p 'test_*.py' OK in a disposable clone of this tree: 311 tests
git diff --check Passed
Additional targeted checks The enforced offline route contract agrees with every new clarify-intent case; the keyword model returns no route for eleven other new cases whose expectations rest on the descriptions, which the static check does not enforce. Per-route coverage of the current corpus after the change: every route has at least three cases, at least one near-miss control that forbids it, and at least one cross-route control. ASCII scan of the changed Markdown, JSON, and Python files: passed.

Cross-platform parity

  • Shared skill changes were reviewed against Codex and Claude Code, or this change does not affect shared skills.
  • Manifest, marketplace, and hook differences remain intentional and documented.
  • Optional host-native validation results include the host/tool version, or are marked unavailable.

Unrelated work

  • I reviewed the final diff and did not reset, hide, stage, or rewrite unrelated work.
  • I did not add generated caches, disposable validation copies, local maintenance notes, or tool output.
  • I called out every affected file and any routing, authorization, documentation, or parity impact above.

Remote baseline: origin/main @ ad19af1

Scope:
Five bundled routes (clarify-intent, delegate-simple-task, task-planning, continuous-progress, local-web-search) had no near-miss or cross-route case anywhere, and six routes had a single current case. Thirty-four schema v6 cases join evals/routing-v6/current.jsonl outside both benchmark profiles, so that every route now has a positive, a near-miss no-route control, and a cross-route control, with Chinese positives for task-planning and local-web-search. The clarify-intent positives state a literal either/or choice whose alternatives differ in authorization, the one ambiguity form the offline model recognizes, so the enforced offline contract still agrees with every clarification case. The evaluation guide records the addition, states that the per-route workload under evals/routing/ stays frozen as the routing-context budget's fixed workload, and documents the retired optimize-codex-usage name mapping. The current-corpus tests take the new totals and no longer pin the number of verify-completion cases. Repository policy revision 115 records the change for Issue #217 with an unchanged plugin version and runtime digest.

Validation:
python3 -B scripts/check-publication.py: passed on the exact tree (174 black-box routing cases, 153 required files, 30 immutable action pins)
python3 -B -m unittest discover -s tests -p 'test_*.py': 311 tests OK in a disposable clone of the tree
offline route model: agrees with every new clarify-intent case; the unenforced disagreements on eleven other new cases are keyword-model limits, not corpus defects
git diff --check: clean; ASCII scan of the changed Markdown, JSON, and Python files: passed
@wheakerd
wheakerd merged commit 0e01f16 into main Oct 30, 2026
10 checks passed
@wheakerd
wheakerd deleted the feature/routing-corpus-coverage branch October 30, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant