feat(seats): agent-CLI compatibility matrix

.github/workflows/seats-compat-nightly.yml

@@ -0,0 +1,89 @@
+name: Seats compatibility nightly
+
+# Runs each pinned agent CLI (packages/seats/compat/targets.json) through the
+# compatibility suite against a local fake provider: no model provider, no
+# real key. A failing check is a result, not a job failure; the job fails
+# only when the matrix itself could not be produced.
+
+on:
+  schedule:
+    - cron: '23 9 * * *'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: seats-compat-nightly
+  cancel-in-progress: false
+
+jobs:
+  matrix:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      # The runner context is not available in job-level env, so the
+      # job-local paths are set here for every later step.
+      - name: Choose job-local paths
+        run: |
+          echo "COMPAT_PREFIX=$RUNNER_TEMP/compat-bin" >> "$GITHUB_ENV"
+          echo "COMPAT_OUT=$RUNNER_TEMP/compat-out" >> "$GITHUB_ENV"
+
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      # Aider (aider-chat) supports python 3.10 to 3.12 only.
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      # Job-local prefix only: release assets are sha256-checked before they
+      # are unpacked, npm packages install with --ignore-scripts, pip packages
+      # install from wheels into a venv under the prefix. A CLI that
+      # fails to install shows up in the matrix as "binary not installed".
+      - name: Install pinned CLIs
+        continue-on-error: true
+        run: node packages/seats/compat/install.js --targets packages/seats/compat/targets.json --prefix "$COMPAT_PREFIX"
+
+      - name: Run the compatibility matrix
+        run: node packages/seats/compat/run.js --targets packages/seats/compat/targets.json --prefix "$COMPAT_PREFIX" --out "$COMPAT_OUT"
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: seats-compat
+          path: ${{ env.COMPAT_OUT }}
+          if-no-files-found: error
+          retention-days: 30
+
+  publish:
+    # Disabled on purpose: publishing needs a maintainer to enable GitHub Pages
+    # for this repository (Settings → Pages → Source: GitHub Actions). Flip this
+    # to `if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'`
+    # once Pages is on.
+    if: false
+    needs: matrix
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deploy.outputs.page_url }}
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: seats-compat
+          path: site
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+      - id: deploy
+        uses: actions/deploy-pages@v4

packages/seats/README.md

@@ -24,3 +24,24 @@ It is private and is not published on its own. `@commonlyai/cli` depends on it a
 The extraction plan is D62, extract-in-place.
 
 **Run record.** The trace store writes each run as a versioned, hash-chained JSONL record. It covers who granted which agent what access for which task, who approved each call, what was attempted, and the result. A tool outside this package can read, verify and export it to OTLP. The format is specified in [`docs/record-v1.md`](docs/record-v1.md), and a generated sample is in [`docs/examples/`](docs/examples/).
+
+## Compatibility matrix
+
+`compat/` runs each pinned CLI in `compat/targets.json` through a suite of checks (`compat/checks.js`) against the local fake provider in `test-support/`, and writes `results.json` plus a self-contained `index.html`:
+
+```
+node compat/install.js --targets compat/targets.json --prefix <dir>   # pinned binaries, job-local
+node compat/run.js --targets compat/targets.json --prefix <dir> --out <dir>
+```
+
+A target is data: `{ cli, version, adapter: 'module#export', binary?, enabled, requires?, probeOptions?, limits? }`. The export is an adapter or a factory taking `options`. A target whose adapter module is absent skips every check. `.github/workflows/seats-compat-nightly.yml` runs it nightly and uploads the output.
+
+- `probeOptions` builds a second instance of a factory export for the checks that read the agent itself. The ACP targets pass `{ watchProvider: false }`, so `retry-status` sees what the agent streams while it retries a 5xx, with no provider watch ending the turn first. `retry-status-surfaced` then asks whether the adapter's trace carried those reports.
+- `binary` names one install recipe: `release` (a per-platform tarball with its sha256), `npm` (installed with `--ignore-scripts`), or `pip` (an exact `name==version`, installed from wheels into a venv under the prefix; the interpreter is `python3` or `COMPAT_PYTHON`).
+- CLIs that run one turn per process in their own headless mode (copilot, qwen-code, cline, aider) share `src/adapters/headless.js`: a `launch` that returns argv and env, a `read` per stdout line, and the process group, deadline, cancellation and trace done once.
+- A CLI that cannot be pointed at the run's OpenAI- or Anthropic-compatible provider (gemini, cursor, amp) still gets a row. Its adapter (`src/adapters/provider-unsupported.js`) detects the pinned binary and fails every turn as `provider_unsupported` with the reason, and every turn check reports that failure, never a skip.
+- `limits: { <checkId>: { note, fixedIn? } }` marks a failure that belongs to that CLI version, not to the adapter. The row keeps `status: 'fail'` and gains `limit`; the page shows it as a limit with the version that fixes it, and `totals.fail` counts only failures no limit explains. A declared limit that stops reproducing shows as a pass that says so.
+
+### Upstream bugs
+
+`compat/upstream.json` lists each known upstream bug with what we do about it: `{ id, cli, versions, bug, status, workaround: { name, ref, where, tests[] }, matrix?: { target, check }, references[] }`. We do not wait on upstream, so `status` is `"worked around"` or `"open"`, and `compat/upstream.js` enforces the first: a row reads as worked around only when its workaround names the pushed commit it lives on (`ref`, a full sha) and at least one test there that proves it. A row that claims more is shown as open, with the reason. An open row names its planned workaround. Upstream issue links go in `references`, never in place of a workaround. A row with `matrix` shows what that check did on this run beside the claim. `run.js` reads the file next to the targets, or the one `--upstream` names.