Repository navigation
evals: measure the bundled skill descriptions in the routing-context budget - #229
Merged
Merged
Conversation
…budget Remote baseline: origin/main @ 0e01f16 Scope: The routing-context budget measured only the always-loaded gate, while the fourteen bundled descriptions, which each host lists on every turn as its selection surface, total 7,250 bytes and were unmeasured. Schema v4 records, selected from 0.22.1, add a descriptionSet with each skill's UTF-8 byte and character counts, the totals, and the labeled token estimate, plus a descriptionComparison against the nearest earlier record that carries a description set under the same 256-byte or 5% review threshold. The 0.22.1 record migrates to schema v4 as the description baseline with a null comparison; the validator reproduces the set from the frontmatter the hosts read, rejects drift, and keeps every earlier record on its own schema. scripts/measure-routing-context.py reports the description set when it measures the gate. The budget guide and the runtime-changes policy record each host's verified hook-output limit as a dated constraint: Codex limits a model-visible hook-output message to roughly 2,500 tokens by default, and Claude Code caps plain SessionStart stdout at 10,000 characters, both verified on 2026-10-30 against the official hooks documentation, with a 1.3 multiplier on the byte estimate for a conservative reading. Repository policy revision 116 records the change for Issue #218 with an unchanged plugin version and runtime digest. Validation: python3 -B scripts/check-publication.py: passed on the exact tree (153 required files, 30 immutable action pins) python3 -B scripts/measure-routing-context.py --check: passed (8 lifecycle scenarios); the gate measurement reports 14 descriptions, 7,250 bytes python3 -B -m unittest discover -s tests -p 'test_*.py': 312 tests OK in a disposable clone of the tree git diff --check: clean; ASCII scan of the changed Markdown and JSON files: passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Refs #218. The routing-context budget measured only the always-loaded gate (5,569 bytes), while the fourteen bundled skill descriptions, which each host lists on every turn as the selection surface, total 7,250 bytes and were not measured anywhere. Records from 0.22.1 now use schema v4, which adds a
descriptionSet(each skill's UTF-8 byte and character counts, the totals, and the same labeled token estimate) and adescriptionComparisonagainst the nearest earlier record that carries a description set, under the same 256-byte or 5% review threshold as the gate. The 0.22.1 record migrates to schema v4 as the description baseline with a null comparison; its gate metrics, lifecycle scenarios, and host evidence are unchanged. The validator reproduces the set from the frontmatter the hosts read and rejects drift; earlier records keep their own schemas. The measurement script reports the description set when it measures the gate.The budget guide and the runtime-changes policy also record each host's hook-output limit as a dated, verified constraint: Codex limits a model-visible hook-output message to roughly 2,500 tokens by default (raised through
additionalContextLimit), and Claude Code caps plainSessionStartstdout at 10,000 characters and replaces longer output with a file reference. Both were verified on 2026-10-30 against the official hooks documentation, read one page at a time. A 1.3 multiplier on the byte estimate gives a conservative token reading, under which the Codex default is the binding limit for the gate.Repository policy revision 116 records the change; the plugin version and runtime contract digest are unchanged.
Affected files and surfaces
axiom_validation/context_budget.py(schema v4 selection, description measurement, shape and comparison checks);scripts/measure-routing-context.py;evals/context-budget/schema-v4.json(new);evals/context-budget/results/v0.22.1.json(migrated to v4 with the description baseline);evals/context-budget/README.md;docs/maintainers/runtime-changes.md;tests/test_context_budget.py;CHANGELOG.md,evidence/repository-policy-revisions-v1.json,evidence/runtime-identity.json,evidence/release-status.json,docs/runtime-identity.md(revision 116)Routing and authorization impact
Documentation
evals/context-budget/README.md(Measurement Boundary; Byte Boundary And Record Ownership) anddocs/maintainers/runtime-changes.md(Always-Loaded Context) describe the description set and the host limits. Installation and hook commands are unchanged.Validation
python3 -B scripts/check-publication.pypython3 -B -m unittest discover -s tests -p 'test_*.py'git diff --checkpython3 -B scripts/measure-routing-context.py --check: passed;python3 -B scripts/measure-routing-context.py: reports the gate plus 14 descriptions totalling 7,250 bytes, equal to the recorded set. ASCII scan of the changed Markdown and JSON files: passed.Cross-platform parity
Unrelated work