fix(ci): make the catalog sync self-healing and unblock the pi 0.85 test suite - #95
Conversation
The sync regenerated the catalog but left manual effort overrides in place, while tests/test-models.ts fails as soon as an override duplicates upstream efforts. The scheduled workflow therefore stayed red on every run since 2026-09-04, its test and format steps failed, and create-pull-request was skipped, so the automation could never propose the fix it exists for. Following the documented procedure by hand did not reach a green state either: with all five overrides removed the map is empty and its non-empty assertion failed, which made the override-free end state unreachable. That assertion is dropped, and the pruned file is now formatted and carried by the sync pull request.
pi 0.85.0 started streaming Anthropic Messages through the SDK, which appends ?beta=true to /v1/messages. The mock compared the exact URL, answered 404, and failed the pi end-to-end suite; CI installs pi@latest unpinned, so every pull request has been red since then, including the catalog fixes for the pinned-CLI issue.
README, the refresh-model-catalog skill, and the changelog still told contributors to delete obsolete overrides by hand.
Prettier wraps an override whose model id is longer than the print width, so the effort array spans the following lines. Removing only the entry line left those level lines behind as invalid syntax in a generated file the sync commits. Consume the wrapped lines up to the closing bracket, and cover the shape with a regression test.
Accept single-quoted or spaced entry syntax, locate the override map after its declaration so a comment containing '= {' cannot corrupt the header, and run the catalog check on pull requests that touch the overrides file or the documented version.
|
Thanks for fixing both blockers. I pushed a focused hardening commit: pruning now uses the existing TypeScript parser and only touches MODEL_EFFORT_OVERRIDES, preserving adjacent declarations and ignoring block-comment entries. Both new regression tests failed before the change and pass afterward (17/17 metadata-check tests). Full npm test passes locally, including real pi mock E2E; OMP is unavailable locally and must pass CI. Replayed the scheduled sync against command-code@1.54.0 in a detached worktree: sync, format, typecheck, model tests, format check and diff check all pass. A second sync produces a byte-identical diff (idempotent). Waiting for CI before merge. |
|
CI exposed another current-host regression: OMP 18.1.22 selects its built-in Command Code model before extension registration. At session_start the registry correctly contains commandcode-custom and the configured endpoint, but the active selection still uses openai-completions and the public endpoint, bypassing our transport. Added a narrowly guarded rebind to the registered model. Reproduced the failure locally with an external-network-blocking Bun preload, then verified the existing full OMP regression suite passes after the fix (including auth sources and generate fallback). Full npm test now passes with both real Pi and OMP required, plus format/diff checks. Waiting for updated CI. |
|
Shipped in |
Why
The daily catalog sync has failed on every scheduled run since 2026-09-04 (8 consecutive failures on
main, SHAadea658), so the automation could never propose the catalog update it exists for. Investigating that turned up two independent blockers in that loop, and a third one that has been failing every pull request.1. The sync could not satisfy its own guard, and the documented end state was unreachable
tests/test-models.tsrequires that a manual override be deleted by hand as soon as upstream publishes its own efforts:command-code@1.53.1now ships efforts for all fivemeta/muse-spark-*entries, so the sync wrote the catalog, then failed that test, thennpm run format:check, andcreate-pull-requestwas skipped — the workflow was red and blocked its own pull request.Following the documented procedure by hand does not reach a green state either: once the five overrides are gone,
MODEL_EFFORT_OVERRIDESis empty and itsassert.ok(...length > 0)fails. The end state was unreachable, not just unautomated.Fixing the automation is also what actually resolves #90: with the catalog synced to
command-code@1.53.1,deepseek/deepseek-v4.1-flashreportsthinking: yesinpi --list-modelswith nomodels.jsonoverride at all, because the CLI bundle now declares bothreasoning:!0and its efforts. Verified against a real pi binary with an isolatedHOME.2. The pi end-to-end suite broke against pi 0.85, failing every PR
pi@0.85.0started streaming Anthropic Messages through the SDK, which appends?beta=trueto/v1/messages. The mock compared the exact URL, answered404, andtests/test-pi-local.mjs:944failed. CI installspi@latestunpinned, so this fails onmaintoday and on every open PR — for example thenpm teststep of #91 fails with the sameAssertionError [ERR_ASSERTION]: 404 Not foundat the same line.What changed
.github/scripts/check-commandcode-model-metadata.ts— the sync prunes manual overrides that upstream now publishes. It rewrites single entry lines and preserves comments, ordering, and still-needed entries; an emptied map collapses to= {}soformat:checkstays green. Idempotent: a second run removes nothing..github/workflows/model-metadata.yml— format andadd-pathsnow includesrc/commandcode-catalog-overrides.ts, which the sync rewrites.tests/test-models.ts— drop the assertion that made the override-free end state unreachable. The guard itself stays: the test still fails while an override duplicates upstream efforts.tests/test-pi-local.mjs— match mock routes on the pathname, so both/v1/messagesand/v1/messages?beta=trueare served.tests/test-model-metadata-check.ts— cover pruning: only obsolete entries are dropped, comments and remaining entries survive, an emptied map collapses, commented-out entries are ignored.This PR deliberately contains no catalog data, so it does not conflict with the catalog-refresh PRs (#86, #93) or the override PRs (#91, #92). It does make #91/#92 unnecessary for
deepseek-v4.1-flash: once the catalog is synced, the model is reasoning-capable on its own.Verification
npm test— full suite green, includingtest-pi-local(PASS) andtest-omp-compat(PASS).test-pi-localfails on unmodifiedmainand passes here.sync:commandcode-catalog→format→typecheck→test:models→format:check→git diff --checkall pass. Before this change, the same sequence failed attest:models.HOMEand nomodels.json:deepseek/deepseek-v4.1-flash,meta/muse-spark-1.3andMiniMaxAI/MiniMax-M3all listthinking: yes.Notes for the maintainer
gpt-6-astraandminimax/minimax-m3-freeexist in the CLI bundle and inmodels.mdbut are absent fromGET /provider/v1/models(69 models, neither id present). No catalog sync can make them selectable in pi; that looks like an upstream Provider API gap worth reporting separately. It is also why amodels.jsonmodelOverridesentry for them has no effect — unknown model ids are ignored.x-command-code-versionis still the pinned catalog version (1.44.0), now 9 CLI releases behind, because one constant drives both the capability snapshot and the request header. Worth decoupling separately.reasoning:!0whenever it seesreasoningEfforts:[. I verified this across both the1.44.0snapshot and1.53.1. That is why this PR does not changesrc/models.ts.