Skip to content

fix(ci): make the catalog sync self-healing and unblock the pi 0.85 test suite - #95

Merged
patlux merged 7 commits into
patlux:mainfrom
pierreraby:fix/catalog-sync-self-heal
Sep 14, 2026
Merged

patlux merged 7 commits into
patlux:mainfrom
pierreraby:fix/catalog-sync-self-heal

Conversation

@pierreraby

Copy link
Copy Markdown
Contributor

Why

The daily catalog sync has failed on every scheduled run since 2026-09-04 (8 consecutive failures on main, SHA adea658), so the automation could never propose the catalog update it exists for. Investigating that turned up two independent blockers in that loop, and a third one that has been failing every pull request.

1. The sync could not satisfy its own guard, and the documented end state was unreachable

tests/test-models.ts requires that a manual override be deleted by hand as soon as upstream publishes its own efforts:

meta/muse-spark-1.1 now has upstream efforts; drop the manual override

command-code@1.53.1 now ships efforts for all five meta/muse-spark-* entries, so the sync wrote the catalog, then failed that test, then npm run format:check, and create-pull-request was skipped — the workflow was red and blocked its own pull request.

Following the documented procedure by hand does not reach a green state either: once the five overrides are gone, MODEL_EFFORT_OVERRIDES is empty and its assert.ok(...length > 0) fails. The end state was unreachable, not just unautomated.

Fixing the automation is also what actually resolves #90: with the catalog synced to command-code@1.53.1, deepseek/deepseek-v4.1-flash reports thinking: yes in pi --list-models with no models.json override at all, because the CLI bundle now declares both reasoning:!0 and its efforts. Verified against a real pi binary with an isolated HOME.

2. The pi end-to-end suite broke against pi 0.85, failing every PR

pi@0.85.0 started streaming Anthropic Messages through the SDK, which appends ?beta=true to /v1/messages. The mock compared the exact URL, answered 404, and tests/test-pi-local.mjs:944 failed. CI installs pi@latest unpinned, so this fails on main today and on every open PR — for example the npm test step of #91 fails with the same AssertionError [ERR_ASSERTION]: 404 Not found at the same line.

What changed

  • .github/scripts/check-commandcode-model-metadata.ts — the sync prunes manual overrides that upstream now publishes. It rewrites single entry lines and preserves comments, ordering, and still-needed entries; an emptied map collapses to = {} so format:check stays green. Idempotent: a second run removes nothing.
  • .github/workflows/model-metadata.yml — format and add-paths now include src/commandcode-catalog-overrides.ts, which the sync rewrites.
  • tests/test-models.ts — drop the assertion that made the override-free end state unreachable. The guard itself stays: the test still fails while an override duplicates upstream efforts.
  • tests/test-pi-local.mjs — match mock routes on the pathname, so both /v1/messages and /v1/messages?beta=true are served.
  • tests/test-model-metadata-check.ts — cover pruning: only obsolete entries are dropped, comments and remaining entries survive, an emptied map collapses, commented-out entries are ignored.
  • README / skill / CHANGELOG — describe the automatic pruning instead of the manual step.

This PR deliberately contains no catalog data, so it does not conflict with the catalog-refresh PRs (#86, #93) or the override PRs (#91, #92). It does make #91/#92 unnecessary for deepseek-v4.1-flash: once the catalog is synced, the model is reasoning-capable on its own.

Verification

  • npm test — full suite green, including test-pi-local (PASS) and test-omp-compat (PASS). test-pi-local fails on unmodified main and passes here.
  • Replayed the exact scheduled workflow on a clean checkout: sync:commandcode-catalogformattypechecktest:modelsformat:checkgit diff --check all pass. Before this change, the same sequence failed at test:models.
  • Ran the sync twice to confirm idempotence.
  • Confirmed the Catalog pinned to CLI 1.44.0: models added upstream (deepseek-v4.1-flash, etc.) can never have reasoning enabled #90 fix end to end against a real pi binary with an isolated HOME and no models.json: deepseek/deepseek-v4.1-flash, meta/muse-spark-1.3 and MiniMaxAI/MiniMax-M3 all list thinking: yes.

Notes for the maintainer

  • The catalog data itself is intentionally left to the automated PR (or chore: refresh all models from command-code@1.53.0 #86/feat(models): add DeepSeek V4.1 Flash (catalog sync to 1.53.0 + pricing) #93) to keep this reviewable and conflict-free.
  • gpt-6-astra and minimax/minimax-m3-free exist in the CLI bundle and in models.md but are absent from GET /provider/v1/models (69 models, neither id present). No catalog sync can make them selectable in pi; that looks like an upstream Provider API gap worth reporting separately. It is also why a models.json modelOverrides entry for them has no effect — unknown model ids are ignored.
  • x-command-code-version is still the pinned catalog version (1.44.0), now 9 CLI releases behind, because one constant drives both the capability snapshot and the request header. Worth decoupling separately.
  • The reasoning flag and the effort levels come from two different upstream artifacts, but they cannot disagree in practice: the generator emits reasoning:!0 whenever it sees reasoningEfforts:[. I verified this across both the 1.44.0 snapshot and 1.53.1. That is why this PR does not change src/models.ts.

The sync regenerated the catalog but left manual effort overrides in place, while tests/test-models.ts fails as soon as an override duplicates upstream efforts. The scheduled workflow therefore stayed red on every run since 2026-09-04, its test and format steps failed, and create-pull-request was skipped, so the automation could never propose the fix it exists for.

Following the documented procedure by hand did not reach a green state either: with all five overrides removed the map is empty and its non-empty assertion failed, which made the override-free end state unreachable. That assertion is dropped, and the pruned file is now formatted and carried by the sync pull request.
pi 0.85.0 started streaming Anthropic Messages through the SDK, which appends ?beta=true to /v1/messages. The mock compared the exact URL, answered 404, and failed the pi end-to-end suite; CI installs pi@latest unpinned, so every pull request has been red since then, including the catalog fixes for the pinned-CLI issue.
README, the refresh-model-catalog skill, and the changelog still told contributors to delete obsolete overrides by hand.
Prettier wraps an override whose model id is longer than the print width, so the effort array spans the following lines. Removing only the entry line left those level lines behind as invalid syntax in a generated file the sync commits. Consume the wrapped lines up to the closing bracket, and cover the shape with a regression test.
Accept single-quoted or spaced entry syntax, locate the override map after its declaration so a comment containing '= {' cannot corrupt the header, and run the catalog check on pull requests that touch the overrides file or the documented version.
@patlux

patlux commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Thanks for fixing both blockers. I pushed a focused hardening commit: pruning now uses the existing TypeScript parser and only touches MODEL_EFFORT_OVERRIDES, preserving adjacent declarations and ignoring block-comment entries. Both new regression tests failed before the change and pass afterward (17/17 metadata-check tests). Full npm test passes locally, including real pi mock E2E; OMP is unavailable locally and must pass CI. Replayed the scheduled sync against command-code@1.54.0 in a detached worktree: sync, format, typecheck, model tests, format check and diff check all pass. A second sync produces a byte-identical diff (idempotent). Waiting for CI before merge.

@patlux

patlux commented Sep 14, 2026

Copy link
Copy Markdown
Owner

CI exposed another current-host regression: OMP 18.1.22 selects its built-in Command Code model before extension registration. At session_start the registry correctly contains commandcode-custom and the configured endpoint, but the active selection still uses openai-completions and the public endpoint, bypassing our transport. Added a narrowly guarded rebind to the registered model. Reproduced the failure locally with an external-network-blocking Bun preload, then verified the existing full OMP regression suite passes after the fix (including auth sources and generate fallback). Full npm test now passes with both real Pi and OMP required, plus format/diff checks. Waiting for updated CI.

@patlux
patlux merged commit 33f25ca into patlux:main Sep 14, 2026
15 checks passed
@patlux

patlux commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Shipped in pi-commandcode-provider@0.7.0: https://github.com/patlux/pi-commandcode-provider/releases/tag/v0.7.0 . Public npm package model discovery and GOAT live smoke verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Catalog pinned to CLI 1.44.0: models added upstream (deepseek-v4.1-flash, etc.) can never have reasoning enabled

2 participants