Repository navigation
test: cover every bundled route in the current routing corpus - #228
Merged
Merged
Conversation
Remote baseline: origin/main @ ad19af1 Scope: Five bundled routes (clarify-intent, delegate-simple-task, task-planning, continuous-progress, local-web-search) had no near-miss or cross-route case anywhere, and six routes had a single current case. Thirty-four schema v6 cases join evals/routing-v6/current.jsonl outside both benchmark profiles, so that every route now has a positive, a near-miss no-route control, and a cross-route control, with Chinese positives for task-planning and local-web-search. The clarify-intent positives state a literal either/or choice whose alternatives differ in authorization, the one ambiguity form the offline model recognizes, so the enforced offline contract still agrees with every clarification case. The evaluation guide records the addition, states that the per-route workload under evals/routing/ stays frozen as the routing-context budget's fixed workload, and documents the retired optimize-codex-usage name mapping. The current-corpus tests take the new totals and no longer pin the number of verify-completion cases. Repository policy revision 115 records the change for Issue #217 with an unchanged plugin version and runtime digest. Validation: python3 -B scripts/check-publication.py: passed on the exact tree (174 black-box routing cases, 153 required files, 30 immutable action pins) python3 -B -m unittest discover -s tests -p 'test_*.py': 311 tests OK in a disposable clone of the tree offline route model: agrees with every new clarify-intent case; the unenforced disagreements on eleven other new cases are keyword-model limits, not corpus defects git diff --check: clean; ASCII scan of the changed Markdown, JSON, and Python files: passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Refs #217. The current routing corpus
evals/routing-v6/current.jsonlhad one case each forclarify-intent,delegate-simple-task,task-planning,agents-architect,optimize-agent-usage,review-axiom-task,confirm-external-action,traceable-git-submit, andreversible-system-change, and the fixed per-route workload underevals/routing/has no case at all for five routes. This change adds thirty-four schema v6 cases to the current corpus, outside both benchmark profiles, so that every bundled route has a positive, a near-miss no-route control, and a cross-route control, with Chinese positives fortask-planningandlocal-web-search. Case IDs are new; no existing case changed, so the contract version stays 6 and the records bound to the v6 profiles stay verifiable. Benchmark membership for the new cases and the changed expectations for the routes the next installed-runtime release narrows belong to that release's successor contract.The evaluation guide records the addition and the decision that the per-route workload stays frozen as the routing-context budget's fixed workload, with its retired
optimize-codex-usagename mapped at validation time. The current-corpus tests take the new totals and no longer pin the number ofverify-completioncases. Repository policy revision 115 records the change; the plugin version and runtime contract digest are unchanged.Affected files and surfaces
evals/routing-v6/current.jsonl(34 added cases);evals/README.md;tests/test_current_routing.py,tests/test_routing_evals.py,tests/test_validator_isolation.py(totals);CHANGELOG.md,evidence/repository-policy-revisions-v1.json,evidence/runtime-identity.json,evidence/release-status.json,docs/runtime-identity.md(revision 115)Routing and authorization impact
mutationAuthorizedfalse.Documentation
evals/README.md(Current Contracts and Contract changes) describes the added coverage and the frozen fixed workload. Installation and hook commands are unchanged.Validation
python3 -B scripts/check-publication.pypython3 -B -m unittest discover -s tests -p 'test_*.py'git diff --checkclarify-intentcase; the keyword model returns no route for eleven other new cases whose expectations rest on the descriptions, which the static check does not enforce. Per-route coverage of the current corpus after the change: every route has at least three cases, at least one near-miss control that forbids it, and at least one cross-route control. ASCII scan of the changed Markdown, JSON, and Python files: passed.Cross-platform parity
Unrelated work