Skip to content

evals: measure the bundled skill descriptions in the routing-context budget - #229

Merged
wheakerd merged 1 commit into
mainfrom
feature/context-budget-descriptions
Oct 30, 2026
Merged

wheakerd merged 1 commit into
mainfrom
feature/context-budget-descriptions

Conversation

@wheakerd

Copy link
Copy Markdown
Owner

Scope

Refs #218. The routing-context budget measured only the always-loaded gate (5,569 bytes), while the fourteen bundled skill descriptions, which each host lists on every turn as the selection surface, total 7,250 bytes and were not measured anywhere. Records from 0.22.1 now use schema v4, which adds a descriptionSet (each skill's UTF-8 byte and character counts, the totals, and the same labeled token estimate) and a descriptionComparison against the nearest earlier record that carries a description set, under the same 256-byte or 5% review threshold as the gate. The 0.22.1 record migrates to schema v4 as the description baseline with a null comparison; its gate metrics, lifecycle scenarios, and host evidence are unchanged. The validator reproduces the set from the frontmatter the hosts read and rejects drift; earlier records keep their own schemas. The measurement script reports the description set when it measures the gate.

The budget guide and the runtime-changes policy also record each host's hook-output limit as a dated, verified constraint: Codex limits a model-visible hook-output message to roughly 2,500 tokens by default (raised through additionalContextLimit), and Claude Code caps plain SessionStart stdout at 10,000 characters and replaces longer output with a file reference. Both were verified on 2026-10-30 against the official hooks documentation, read one page at a time. A 1.3 multiplier on the byte estimate gives a conservative token reading, under which the Codex default is the binding limit for the gate.

Repository policy revision 116 records the change; the plugin version and runtime contract digest are unchanged.

Affected files and surfaces

  • Shared files: None
  • Codex-specific files: None
  • Claude Code-specific files: None
  • Documentation or validation files: axiom_validation/context_budget.py (schema v4 selection, description measurement, shape and comparison checks); scripts/measure-routing-context.py; evals/context-budget/schema-v4.json (new); evals/context-budget/results/v0.22.1.json (migrated to v4 with the description baseline); evals/context-budget/README.md; docs/maintainers/runtime-changes.md; tests/test_context_budget.py; CHANGELOG.md, evidence/repository-policy-revisions-v1.json, evidence/runtime-identity.json, evidence/release-status.json, docs/runtime-identity.md (revision 116)

Routing and authorization impact

  • Route-selection impact: None. No skill text changes.
  • Action-authorization impact: None.
  • New or changed stop conditions: None.

Documentation

evals/context-budget/README.md (Measurement Boundary; Byte Boundary And Record Ownership) and docs/maintainers/runtime-changes.md (Always-Loaded Context) describe the description set and the host limits. Installation and hook commands are unchanged.

Validation

Command Result
python3 -B scripts/check-publication.py Passed on this exact tree: 153 required files, 168 Markdown files, 8 routing-context lifecycle scenarios, 30 immutable external action and image pins
python3 -B -m unittest discover -s tests -p 'test_*.py' OK in a disposable clone of this tree: 312 tests
git diff --check Passed
Additional targeted checks python3 -B scripts/measure-routing-context.py --check: passed; python3 -B scripts/measure-routing-context.py: reports the gate plus 14 descriptions totalling 7,250 bytes, equal to the recorded set. ASCII scan of the changed Markdown and JSON files: passed.

Cross-platform parity

  • Shared skill changes were reviewed against Codex and Claude Code, or this change does not affect shared skills.
  • Manifest, marketplace, and hook differences remain intentional and documented.
  • Optional host-native validation results include the host/tool version, or are marked unavailable.

Unrelated work

  • I reviewed the final diff and did not reset, hide, stage, or rewrite unrelated work.
  • I did not add generated caches, disposable validation copies, local maintenance notes, or tool output.
  • I called out every affected file and any routing, authorization, documentation, or parity impact above.

…budget

Remote baseline: origin/main @ 0e01f16

Scope:
The routing-context budget measured only the always-loaded gate, while the fourteen bundled descriptions, which each host lists on every turn as its selection surface, total 7,250 bytes and were unmeasured. Schema v4 records, selected from 0.22.1, add a descriptionSet with each skill's UTF-8 byte and character counts, the totals, and the labeled token estimate, plus a descriptionComparison against the nearest earlier record that carries a description set under the same 256-byte or 5% review threshold. The 0.22.1 record migrates to schema v4 as the description baseline with a null comparison; the validator reproduces the set from the frontmatter the hosts read, rejects drift, and keeps every earlier record on its own schema. scripts/measure-routing-context.py reports the description set when it measures the gate. The budget guide and the runtime-changes policy record each host's verified hook-output limit as a dated constraint: Codex limits a model-visible hook-output message to roughly 2,500 tokens by default, and Claude Code caps plain SessionStart stdout at 10,000 characters, both verified on 2026-10-30 against the official hooks documentation, with a 1.3 multiplier on the byte estimate for a conservative reading. Repository policy revision 116 records the change for Issue #218 with an unchanged plugin version and runtime digest.

Validation:
python3 -B scripts/check-publication.py: passed on the exact tree (153 required files, 30 immutable action pins)
python3 -B scripts/measure-routing-context.py --check: passed (8 lifecycle scenarios); the gate measurement reports 14 descriptions, 7,250 bytes
python3 -B -m unittest discover -s tests -p 'test_*.py': 312 tests OK in a disposable clone of the tree
git diff --check: clean; ASCII scan of the changed Markdown and JSON files: passed
@wheakerd
wheakerd merged commit 4fb5271 into main Oct 30, 2026
10 checks passed
@wheakerd
wheakerd deleted the feature/context-budget-descriptions branch October 30, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant