Repository navigation
test: pin the clock in three tests that raced the real one on Windows #74129
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| branches: [main] | |
| permissions: | |
| contents: read | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| # PRs cancel superseded revisions: only a PR's tip commit needs a verdict. | |
| # main does not, so let an in-flight run finish instead of killing it. | |
| # This does NOT produce a verdict per commit. The group still permits one | |
| # pending run, and GitHub evicts that pending run when a newer one queues | |
| # (a documented GitHub concurrency behavior). With a ~19min run and a | |
| # ~1.4min median merge gap, main gets periodic verdicts instead of none. | |
| cancel-in-progress: ${{ github.event_name == 'pull_request' }} | |
| # There is deliberately no BACKEND_DESELECTS here any more. Eleven files used to be | |
| # deselected from EVERY backend pytest invocation because they drive a real | |
| # `git`/`gh`/`pytest` through `sandboxed_spawn_argv`, and a GH runner denies | |
| # `unshare(CLONE_NEWNS)` (EPERM) so `wrap_argv` fails closed. That kept 608 tests -- | |
| # including the suites guarding the ops autonomy gate -- off every pull request. | |
| # | |
| # They no longer need it: each one that requires a sandbox carries its own | |
| # `skipif(not userns_available())`, so on a runner it skips with a named reason and on | |
| # a capable host it runs for real. MEASURED against a faithful local simulation of the | |
| # runner (`unshare(CLONE_NEWUSER)` ok, `CLONE_NEWNS` EPERM, no user session bus for the | |
| # cgroup scope, no gh token): 608 passed, 85 skipped, 0 failed. | |
| # | |
| # A whole-file deselect is the wrong tool for a missing host capability, and this is | |
| # why: it is invisible in the run output, it takes the file's OTHER tests with it, and | |
| # nothing goes red when the reason expires. A capability guard says which capability, | |
| # on which test, in the report. If a suite here starts needing one again, add the guard | |
| # -- not a deselect. | |
| jobs: | |
| # ── Surface detection (issue #1556) ────────────────────────────────── | |
| # Most tests only exercise their own surface, so a single-surface diff can | |
| # skip the opposite suite's *provably* single-surface files. The selector | |
| # (scripts/ci-surface-tests.py) is deny-by-default: every cross-surface | |
| # parity guard -- and anything it cannot classify -- stays in the must-run | |
| # set, so a heuristic miss costs CI time, never a skipped guard. | |
| # | |
| # only_backend / only_frontend are computed here ONCE. `backend` is a | |
| # catch-all, so every changed file is exactly one of frontend / meta / | |
| # backend: a diff that touches two buckets (or meta) leaves BOTH flags false | |
| # and the full matrix runs. One exception: temp-screenshots/** evidence | |
| # media counts toward NO bucket, so a screenshots-only diff also leaves both | |
| # flags false and runs the full matrix (fails safe). | |
| changes: | |
| name: Detect changed surface | |
| runs-on: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) && format('codebuild-kirocrew-gha-linux-{0}-{1}', github.run_id, github.run_attempt) || 'ubuntu-latest' }} | |
| # 15, not 5: the job's own work is seconds, but its `fetch-depth: 0` | |
| # checkout (paths-filter and the leaf resolver diff against the merge base | |
| # with local git) is a full-history fetch of a large repository, and on a | |
| # slow hosted runner that fetch alone has run past 5 minutes and had the | |
| # job CANCELLED mid-checkout. Every test lane `needs` this job, so that one | |
| # cancel skipped the whole matrix and failed Coverage Gate with nothing to | |
| # read (#13111). The budget covers a slow fetch; a hung one still ends. | |
| timeout-minutes: 15 | |
| # dorny/paths-filter runs with `token: ''` (below), so it detects changes | |
| # with local git instead of the REST API and needs no pull-requests:read. | |
| # Job-level permissions REPLACE the top-level grant, so contents:read is | |
| # restated. | |
| permissions: | |
| contents: read | |
| outputs: | |
| only_backend: ${{ steps.decide.outputs.only_backend }} | |
| only_frontend: ${{ steps.decide.outputs.only_frontend }} | |
| linux_packaging: ${{ steps.filter.outputs.linux_packaging }} | |
| leaf_tests: ${{ steps.leaf.outputs.leaf_tests }} | |
| leaf_targets: ${{ steps.leaf.outputs.leaf_targets }} | |
| leaf_gates: ${{ steps.leaf.outputs.leaf_gates }} | |
| # Central runner labels: eligible repository/event/actor combinations | |
| # use CodeBuild; all other runs use the corresponding hosted OS. | |
| # Linux consumers retain a fallback when this job never reports: | |
| # `${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}`. | |
| # if this job fails or is cancelled the output is empty, and a job that | |
| # runs anyway (`coverage-gate` fails closed with `if: always()`) must still | |
| # get a runner rather than error out before its own checks execute. | |
| linux_runner: ${{ steps.runner.outputs.linux_runner }} | |
| linux_runner_large: ${{ steps.runner.outputs.linux_runner_large }} | |
| windows_runner: ${{ steps.windows-runner.outputs.windows_runner }} | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| # The leaf-test selector diffs against the PR base, so the base commit | |
| # must be present. Paid once here rather than on every shard job. | |
| fetch-depth: 0 | |
| persist-credentials: false | |
| - uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 | |
| id: filter | |
| with: | |
| # An empty token switches the action to LOCAL git change detection on | |
| # pull_request events (documented on the `token` input; the default is | |
| # the shared installation token). The pinned v4.0.3 build then runs | |
| # getChanges(pull_request.base.sha, HEAD): a plain two-dot | |
| # `git diff` -- the checkout above already has the base commit | |
| # (fetch-depth: 0), so no fetch and no API call happen. This matters | |
| # because the REST list-files call spends the per-hour installation | |
| # quota shared by EVERY workflow in this repo, and at dozens of | |
| # ci.yml runs per hour it exhausted the quota and failed this job | |
| # repo-wide (#10434). With an empty token the `base` input stays | |
| # empty on purpose: the action falls back to the PR base SHA from | |
| # the event payload. Both diff endpoints are frozen when the run is | |
| # created (base.sha from the payload, HEAD at the event's merge | |
| # commit), so a base-branch move after that changes nothing. The | |
| # result is the NET change the merge introduces relative to base.sha: | |
| # a PR-listed file whose identical change already landed in base | |
| # drops out, which is correct for surface detection -- that surface | |
| # is unchanged relative to base -- and any wider change set still | |
| # pushes toward the full matrix. push events never consult `token`, | |
| # so main builds are unchanged. | |
| token: '' | |
| # 'some-with-excludes': a file matches a filter when it matches at | |
| # least one pattern AND none of the negated ones. That lets `backend` | |
| # be a true CATCH-ALL ("every changed file that is not frontend, not | |
| # meta, and not ignored evidence media") instead of an allowlist that | |
| # would have to be complete to be safe. Every changed file therefore | |
| # lands in exactly one bucket -- or none, for temp-screenshots/** | |
| # evidence -- which is what keeps "only_X means only X changed" true | |
| # for real files: an unrecognised path (docs/**, *.md, a new root | |
| # file) counts as backend rather than silently riding along under a | |
| # narrowed suite. | |
| # NOTE: this quantifier requires >= v4.0.3 (added upstream in PR #322); | |
| # v4.0.2 rejects the input and throws, so do not lower the pin. | |
| predicate-quantifier: 'some-with-excludes' | |
| filters: | | |
| frontend: | |
| - 'website/**' | |
| meta: | |
| - '.github/**' | |
| - 'scripts/**' | |
| backend: | |
| - '**' | |
| - '!website/**' | |
| - '!.github/**' | |
| - '!scripts/**' | |
| # temp-screenshots/** is never-packaged, never-imported evidence | |
| # media (gitignored; PR evidence travels as a GitHub attachment), | |
| # so it must not drag a frontend-only PR into the | |
| # full backend matrix (#8027). Excluding it keeps the catch-all | |
| # safety property: any unrecognised REAL file still lands in | |
| # backend, a screenshots+frontend diff stays only_frontend=true, | |
| # and a screenshots-ONLY diff matches no bucket so both flags | |
| # stay false and the FULL matrix runs (fails safe). | |
| - '!temp-screenshots/**' | |
| # Independent of the three buckets above: it answers a different | |
| # question ("could this diff change what a Linux package installs?") | |
| # and so may overlap them freely. | |
| linux_packaging: | |
| - 'website/electron/package.json' | |
| - 'website/electron/auto-update.js' | |
| - 'website/electron/bundle-location.js' | |
| - 'packaging/build-desktop.sh' | |
| - 'scripts/stamp-distribution.sh' | |
| - '.github/workflows/build-desktop.yml' | |
| - name: Decide surface | |
| id: decide | |
| env: | |
| FRONTEND: ${{ steps.filter.outputs.frontend }} | |
| BACKEND: ${{ steps.filter.outputs.backend }} | |
| META: ${{ steps.filter.outputs.meta }} | |
| # Escape hatch: the ci-full-run label forces the full matrix. NOTE: | |
| # this workflow does not listen for the `labeled` event (that would | |
| # re-run everything on every bot label), so the label takes effect on | |
| # the NEXT push, not the instant it is applied. | |
| FULL_RUN: ${{ contains(github.event.pull_request.labels.*.name, 'ci-full-run') }} | |
| run: | | |
| # A surface is "only" when it changed and NO other bucket did. | |
| # `backend` is a catch-all, so an unrecognised path counts as backend | |
| # and cannot ride along under a narrowed suite. | |
| only_backend=false; only_frontend=false | |
| if [ "$FULL_RUN" != "true" ] && [ "$META" != "true" ]; then | |
| if [ "$BACKEND" = "true" ] && [ "$FRONTEND" != "true" ]; then only_backend=true; fi | |
| if [ "$FRONTEND" = "true" ] && [ "$BACKEND" != "true" ]; then only_frontend=true; fi | |
| fi | |
| echo "only_backend=$only_backend" >> "$GITHUB_OUTPUT" | |
| echo "only_frontend=$only_frontend" >> "$GITHUB_OUTPUT" | |
| echo "backend=$BACKEND frontend=$FRONTEND meta=$META full_run=$FULL_RUN -> only_backend=$only_backend only_frontend=$only_frontend" | |
| # A diff that touches ONLY leaf test modules cannot change a production code | |
| # path, and cannot change another test file's outcome either -- the selector | |
| # escalates on conftest, shared helpers, fixture trees, and any test module | |
| # some other file imports. So the sharded suite is replaced by the changed | |
| # files plus the gates that read the test/ tree as data, run several times. | |
| # | |
| # This exists because a one-line flaky-test fix used to wait ~30 minutes for | |
| # the full matrix, which is what makes a flake block every open PR. Repeats | |
| # matter more than breadth here: a flaky test run once in isolation passes. | |
| # | |
| # Every failure path below yields the FULL suite. `ci-full-run` still forces | |
| # the whole matrix when a fix depends on suite-wide interleaving. | |
| - name: Decide leaf-test scope | |
| id: leaf | |
| env: | |
| FULL_RUN: ${{ contains(github.event.pull_request.labels.*.name, 'ci-full-run') }} | |
| BASE_SHA: ${{ github.event.pull_request.base.sha }} | |
| run: | | |
| set -uo pipefail | |
| leaf_tests=false | |
| # BOTH must be pre-initialized: the heredoc below expands each one | |
| # unconditionally, while they are assigned only in the `else` branch. A | |
| # push build (no PR base) and a `ci-full-run` PR both take the `if` | |
| # branch, so an uninitialized variable there dies on `set -u` and takes | |
| # the whole `changes` job -- and every job that needs it -- with it, | |
| # breaking the very full-matrix escape hatch this branch exists to serve. | |
| targets="" | |
| gates="" | |
| if [ "$FULL_RUN" = "true" ] || [ -z "${BASE_SHA:-}" ]; then | |
| echo "Full suite: ci-full-run label set, or no PR base (push build)." | |
| else | |
| rc=0 | |
| targets="$(SCOPED_TESTS_BASE_REF="$BASE_SHA" \ | |
| python3 scripts/leaf_test_scope.py --targets)" || rc=$? | |
| gates="$(SCOPED_TESTS_BASE_REF="$BASE_SHA" \ | |
| python3 scripts/leaf_test_scope.py --gates)" || true | |
| if [ "$rc" = "0" ] && [ -n "$targets" ]; then | |
| leaf_tests=true | |
| echo "Leaf-test diff: $(printf '%s\n' "$targets" | grep -c .) target file(s)." | |
| printf '%s\n' "$targets" | sed 's/^/ - /' | |
| else | |
| echo "Full suite: not a leaf-test-only diff (selector rc=$rc)." | |
| gates="" | |
| fi | |
| fi | |
| echo "leaf_tests=$leaf_tests" >> "$GITHUB_OUTPUT" | |
| { | |
| echo "leaf_targets<<LEAF_EOF" | |
| printf '%s\n' "$targets" | |
| echo "LEAF_EOF" | |
| echo "leaf_gates<<GATE_EOF" | |
| printf '%s\n' "$gates" | |
| echo "GATE_EOF" | |
| } >> "$GITHUB_OUTPUT" | |
| # Same-repository push/PR AND an actor admitted by the fleet webhook. | |
| # Missing CODEBUILD_ACTOR_IDS fails toward hosted, never a queued fleet | |
| # job the webhook cannot accept. The webhook remains an independent fence. | |
| - name: Pick the Linux runner label | |
| id: runner | |
| env: | |
| ELIGIBLE: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) }} | |
| LABEL: codebuild-kirocrew-gha-linux-${{ github.run_id }}-${{ github.run_attempt }} | |
| run: | | |
| if [ "$ELIGIBLE" = "true" ]; then | |
| echo "linux_runner=$LABEL" >> "$GITHUB_OUTPUT" | |
| echo "linux_runner_large=$LABEL instance-size:large" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "linux_runner=ubuntu-latest" >> "$GITHUB_OUTPUT" | |
| echo "linux_runner_large=ubuntu-latest" >> "$GITHUB_OUTPUT" | |
| fi | |
| - name: Pick the Windows runner label | |
| id: windows-runner | |
| env: | |
| ELIGIBLE: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) }} | |
| LABEL: codebuild-kirocrew-gha-windows-${{ github.run_id }}-${{ github.run_attempt }} | |
| run: | | |
| if [ "$ELIGIBLE" = "true" ]; then | |
| echo "windows_runner=$LABEL instance-size:large" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "windows_runner=windows-latest" >> "$GITHUB_OUTPUT" | |
| fi | |
| # Build the Linux desktop packages and install them for real, but ONLY when the | |
| # diff could change what a package installs. The check is expensive (a PBS | |
| # interpreter, a full pip closure, then electron-builder three times), and it | |
| # verifies things no cheaper gate can: dependency names that must exist in the | |
| # target distro's repositories, the generated .desktop entry, and the maintainer | |
| # scripts. Same trade the macOS-packaging path gate already makes. | |
| # | |
| # A packaging change that lands without this having run is the failure mode this | |
| # exists to prevent: a package that builds green and refuses to install. | |
| linux-packaging: | |
| name: Linux Packaging (build + smoke-install) | |
| needs: [changes, await-fast-gate] | |
| if: ${{ needs.changes.outputs.linux_packaging == 'true' }} | |
| runs-on: ubuntu-22.04 | |
| timeout-minutes: 45 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - name: Install uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| # Builds all three Linux formats from one backend tree, re-stamping the | |
| # beacon distribution between electron-builder invocations. | |
| - name: Build the Linux desktop packages | |
| env: | |
| CSC_IDENTITY_AUTO_DISCOVERY: "false" | |
| run: bash packaging/build-desktop.sh | |
| - name: Install the packages in their target distros | |
| run: ./scripts/smoke-linux-packages.sh website/electron/dist | |
| # ── Fast gate barrier (see .github/workflows/fast-gate.yml) ────────── | |
| # The eleven cheap gates live in their own workflow so the fork reviewers can | |
| # key on them, which means ci.yml cannot reach them with `needs:` -- a needs | |
| # edge cannot cross a workflow file. This job is that edge: it reads the Fast | |
| # Gate run for this exact commit and fails closed, and every heavy job needs | |
| # it. One extra ~1-minute job buys the whole matrix the right to not start. | |
| # | |
| # Fails closed on purpose, in all three ways it can go wrong: a Fast Gate run | |
| # that never appears, one that never completes inside the budget, and one whose | |
| # conclusion is a decided non-success all exit non-zero. `action_required` is | |
| # not one of them: GitHub reports a fork run awaiting maintainer approval that | |
| # way, so it is polled as pending. A barrier that passed when it could | |
| # not read its subject would be worse than no barrier, because the matrix would | |
| # run anyway and the log would claim it was cleared to. | |
| await-fast-gate: | |
| name: Await Fast Gate | |
| runs-on: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) && format('codebuild-kirocrew-gha-linux-{0}-{1}', github.run_id, github.run_attempt) || 'ubuntu-latest' }} | |
| timeout-minutes: 15 | |
| permissions: | |
| # actions:read is what lets this read another workflow's runs. Job-level | |
| # permissions REPLACE the top-level grant, so contents:read is restated. | |
| actions: read | |
| contents: read | |
| steps: | |
| - name: Wait for the Fast Gate verdict on this commit | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| REPO: ${{ github.repository }} | |
| # The head sha, not github.sha: on a pull_request event github.sha is | |
| # the ephemeral merge commit, and Fast Gate's own run is keyed to the | |
| # same head the PR reports. Matching on `event` too keeps a push run on | |
| # the same commit from answering for a pull_request run. | |
| SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} | |
| EVENT: ${{ github.event_name }} | |
| # A head SHA is NOT a unique key for a run. Two pull requests can carry | |
| # the same head commit -- a shared branch, a cherry-pick, a fork whose | |
| # branch was pushed to a second PR -- and each one gets its own Fast Gate | |
| # run on that commit. Keyed on the SHA alone, this job could read the | |
| # OTHER PR's verdict and release its own matrix on a gate that never ran | |
| # against this PR's base. The head branch plus the head repository is | |
| # what makes the triple unique, so both are matched below and then | |
| # re-verified on the run that is selected. | |
| BRANCH: ${{ github.event_name == 'pull_request' && github.head_ref || github.ref_name }} | |
| HEAD_REPO: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name || github.repository }} | |
| run: | | |
| set -euo pipefail | |
| # Both workflows start from the same webhook, so Fast Gate's run may not | |
| # be visible for the first few seconds. APPEAR_BUDGET covers that race; | |
| # after it, a missing run is a real misconfiguration and fails. | |
| APPEAR_BUDGET=180 | |
| TOTAL_BUDGET=720 | |
| started="$(date +%s)" | |
| seen=false | |
| while :; do | |
| elapsed=$(( $(date +%s) - started )) | |
| runs="$(gh api --method GET \ | |
| "repos/$REPO/actions/workflows/fast-gate.yml/runs" \ | |
| -f "head_sha=$SHA" -f "event=$EVENT" -f "branch=$BRANCH" \ | |
| -f per_page=100 2>/dev/null || true)" | |
| # `branch=` is passed to the API as a narrowing hint, but the match is | |
| # re-asserted here: a filter the server silently ignores would hand | |
| # back another PR's run, and selecting max_by(.id) out of an unfiltered | |
| # list would pick the NEWEST such run -- the most likely one to be a | |
| # different PR's. Filter first, collapse second. | |
| latest="$(printf '%s' "$runs" \ | |
| | jq -c --arg branch "$BRANCH" --arg repo "$HEAD_REPO" \ | |
| '[(.workflow_runs // [])[] | |
| | select(.head_branch == $branch) | |
| | select(.head_repository.full_name == $repo)] | |
| | if length == 0 then null else max_by(.id) end' \ | |
| 2>/dev/null || echo null)" | |
| if [ "$latest" = "null" ] || [ -z "$latest" ]; then | |
| if [ "$elapsed" -ge "$APPEAR_BUDGET" ]; then | |
| echo "::error::No Fast Gate run found for $SHA on $HEAD_REPO@$BRANCH" \ | |
| "($EVENT) after ${elapsed}s. The gates cannot be confirmed, so the" \ | |
| "matrix is not cleared to run." | |
| exit 1 | |
| fi | |
| sleep 10 | |
| continue | |
| fi | |
| status="$(printf '%s' "$latest" | jq -r '.status')" | |
| conclusion="$(printf '%s' "$latest" | jq -r '.conclusion // ""')" | |
| url="$(printf '%s' "$latest" | jq -r '.html_url')" | |
| if [ "$seen" = false ]; then | |
| echo "Fast Gate run: $url" | |
| seen=true | |
| fi | |
| if [ "$status" = "completed" ]; then | |
| case "$conclusion" in | |
| success) | |
| echo "Fast Gate passed. Releasing the heavy jobs." | |
| exit 0 | |
| ;; | |
| action_required) | |
| # A fork run still awaiting maintainer approval is reported as | |
| # completed/action_required. That is pending, not decided, so keep | |
| # polling under TOTAL_BUDGET -- otherwise which of the two pending | |
| # runs a maintainer approves first decides whether the matrix runs. | |
| status="awaiting maintainer approval" | |
| ;; | |
| *) | |
| echo "::error::Fast Gate concluded '$conclusion' ($url)." \ | |
| "Fix the gate it reports; the heavy jobs are skipped on purpose." | |
| exit 1 | |
| ;; | |
| esac | |
| fi | |
| if [ "$elapsed" -ge "$TOTAL_BUDGET" ]; then | |
| echo "::error::Fast Gate still '$status' after ${elapsed}s ($url)." \ | |
| "Failing closed rather than starting the matrix unverified." | |
| exit 1 | |
| fi | |
| sleep 10 | |
| done | |
| backend-lint: | |
| name: Backend Lint & Type Check | |
| needs: [changes, await-fast-gate] | |
| # black/flake8/isort/mypy read no file under website/, so a frontend-only | |
| # diff cannot change this verdict. Gated on only_frontend (not on a | |
| # positive "python changed" allowlist) so that anything the selector could | |
| # not classify as frontend -- docs, a new root file, meta -- still runs it. | |
| if: needs.changes.outputs.only_frontend != 'true' | |
| # Large alone did not fix black's process-pool failure. Bound its workers | |
| # below as well, on hosted and fleet alike; preserve every lint/type gate. | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| # The job's floor on today's hosted runners is ~14-15 min (black --check | |
| # over the whole tree alone takes 7-8 min, flake8 ~2 min, mypy cold ~3 min), | |
| # so 15 cancelled mypy mid-run on green PRs. 25 is the budget frontend-test | |
| # and bundle-size already run on, and leaves headroom for runner variance. | |
| timeout-minutes: 25 | |
| strategy: | |
| matrix: | |
| python-version: ["3.12"] | |
| steps: | |
| # Full history so the black gate can compute THIS change's own diff. A | |
| # shallow clone truncates parent info and leaves no base ref, and the gate | |
| # then falls back to judging the whole tree -- which reddens the PR for | |
| # files the base branch merged after the baseline was taken. depth 2 was | |
| # tried first and was not enough; the gate now prints which scope it used, | |
| # so a regression here is visible in the log instead of silent. | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| fetch-depth: 0 | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # This repo has no uv.lock (pip/pyproject-managed), so point the | |
| # cache key at the files that define the dependency set. Without | |
| # this, setup-uv defaults to globbing **/uv.lock, finds nothing, | |
| # and errors: "No file matched to [**/uv.lock]". | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| # uv resolves/installs the heavy deps far faster than pip and caches | |
| # across runs. --system targets the setup-python interpreter (no venv); | |
| # --group pulls [dependency-groups].dev from pyproject. | |
| run: uv pip install --system -e ".[voice]" --group dev | |
| # Python vulnerability visibility. The repository had none: the step this | |
| # replaces ran an unpinned `pip install pip-audit`, and pip-audit is now | |
| # pinned in the dev group installed immediately above, so the audit needs no | |
| # install of its own and two runs a week apart audit with the same tool. | |
| # | |
| # continue-on-error: this REPORTS on day one. Widening dependency auditing | |
| # and making it block in the same change is how a gate ends up switched off | |
| # by whoever it first stopped, and pip-audit reaches the network, so a | |
| # registry blip would otherwise fail unrelated PRs. What makes this | |
| # different from the advisory step it replaces is the baseline: the script | |
| # compares the count against python-audit-baseline.json and says so when it | |
| # rises, which turns a standing wall of known advisories into a number that | |
| # only moves when someone moves it. Dropping continue-on-error is the | |
| # follow-up once a count is recorded and has held steady. | |
| - name: Python dependency audit (pinned, report-only) | |
| continue-on-error: true | |
| run: python3 scripts/check_python_audit.py | |
| # black IS enforced, behind a shrinking baseline. A bare `black --check` | |
| # cannot be turned on here: 1,420 files under src/ and test/ are not | |
| # black-clean, so the bulk pass it waits on is ~95,800 lines and has been | |
| # "pending" long enough for the old estimate in this comment to drift by 3x. | |
| # Leaving it off meanwhile was the worse failure: AGENTS.md tells every | |
| # contributor to run `black src/kiro_crew test` before committing, and | |
| # following that instruction reformats those 1,420 files on top of their | |
| # change. The baseline makes the gate real for everything else, so new code | |
| # is formatted without any PR being forced to carry unrelated churn. | |
| - name: Check formatting (black, baselined) | |
| # Every runner recycles black workers: one file per child, at most eight, | |
| # capped at 2 GiB each on Linux. Hosted `ubuntu-latest` needs this too: | |
| # the compiled wheel retains memory per worker and the VM has no cgroup | |
| # cap, so a native pool takes the whole 16 GB runner down mid-step (exit | |
| # 143). Recycling costs ~1.6x native wall time on four cores. | |
| # Keep cgroup evidence: per-process limits do not bound the whole job. | |
| run: python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.py | |
| # Text-mode subprocess calls must pin their decode encoding: without one | |
| # the child's output decodes with the Windows ANSI code page (#5249). | |
| # Self-test first: it plants one probe per rule family, so a typo that | |
| # silently disables a rule fails here instead of shipping green. | |
| - name: Check subprocess text-mode encoding (gate, baselined) | |
| run: | | |
| python3 scripts/check_subprocess_encoding.py --test | |
| python3 scripts/check_subprocess_encoding.py | |
| # A committed ACP frame fixture carries what the WIRE carried. The class that | |
| # gets through a credential scrub is an ENUMERATION -- a provider catalog, a | |
| # command inventory -- because it reads as product data while describing what | |
| # the recording host HAS. Today's leaks are carried in the baseline as | |
| # pre-existing and it is shrink-only, per file AND per marker class, so a new | |
| # fixture cannot inherit an exemption. | |
| - name: Check ACP frame fixtures for host data (gate, baselined) | |
| run: python3 scripts/check_acp_frame_host_data.py | |
| # Application code must reach the agent backend through kiro_crew.agent_sdk, | |
| # not by importing kiro_crew.acp (or kiro_crew.providers, which re-exports | |
| # the same ACP shapes and would otherwise be a laundering channel). Today's | |
| # edges are carried in the baseline as pre-existing and it is shrink-only. Design | |
| # record: docs/request-for-change/rfc-crew-agent-sdk-boundary.md. | |
| # Self-test first, same reason as the gate above. | |
| - name: Check agent-SDK import boundary (gate, baselined) | |
| run: | | |
| python3 scripts/check_agent_sdk_boundary.py --test | |
| python3 scripts/check_agent_sdk_boundary.py | |
| # Blocking IO inside an `async def` stalls every session's turn, and a | |
| # stall past dashboard.loop_stall_exit_after_secs (25s) makes the watchdog | |
| # kill the gateway — the user sees a restart for no apparent reason and | |
| # loses every in-flight turn (#1572). #3057's audit found the sites | |
| # cluster where nothing was watching: ~70 on-loop store.db.execute() calls | |
| # in dashboard/handlers/knowledge.py against zero in handlers/memory.py | |
| # next door. Baselined for the same reason as the gates above — the | |
| # known set is recorded as legacy so it can only shrink, while a NEW call | |
| # (or one on a line this change adds) fails here. | |
| - name: Check for blocking IO inside async def (gate, baselined) | |
| run: | | |
| # Self-test first: it plants one probe per rule family, so a rule that | |
| # silently stops matching fails here instead of shipping green. | |
| python3 scripts/check_sync_io_in_async.py --test | |
| python3 scripts/check_sync_io_in_async.py | |
| # A secret-bearing file must be locked down BEFORE any content reaches it: | |
| # a post-publish lockdown leaves the payload at its final path under the | |
| # inherited DACL, which on Windows is not covered by atomic_write's POSIX | |
| # mode at all. #5307 converted the last seven such writers; this keeps a | |
| # new one from reintroducing the shape. Known-unconverted sites live in | |
| # KNOWN_UNCONVERTED in the script, keyed to an open issue, and that list is | |
| # shrink-only -- a paid-off entry fails the gate until it is deleted. | |
| - name: Check lockdown-before-publish (secret writers) | |
| run: python3 scripts/check_lockdown_before_publish.py | |
| # isort/flake8/mypy below are BLOCKING (config in pyproject.toml + setup.cfg). | |
| # | |
| # The two repo-root modules are named explicitly: `src/kiro_crew test` misses | |
| # them, and between them they hold the host-protection floor for all three | |
| # testpaths -- the isolation fixtures and the xdist worker budget -- so they | |
| # are the last files that should go unlinted. | |
| - name: Check imports (isort) | |
| run: isort --check-only src/kiro_crew test conftest.py xdist_budget.py | |
| - name: Lint (flake8) | |
| run: flake8 src/kiro_crew test conftest.py xdist_budget.py | |
| - name: Type check (mypy) | |
| run: mypy src/kiro_crew/ | |
| backend-test: | |
| name: Backend Tests | |
| needs: [changes, await-fast-gate] | |
| # All eight shards use large compute and the measured non-root boundary. | |
| # Forks, non-admitted actors and absent resolver output stay hosted. | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| # Keep the existing wall-clock budget; individual tests still time out at | |
| # 120 seconds. File sharding reduces repeated collection, not test coverage. | |
| timeout-minutes: 60 | |
| strategy: | |
| # Don't cancel sibling shards when one fails -- we want the full failure | |
| # picture across the suite, not just the first shard to break. | |
| fail-fast: false | |
| matrix: | |
| python-version: ["3.12"] | |
| # Eight runners reduce per-shard work at the cost of more runner slots. | |
| # Whole files are assigned before import; -n auto parallelizes within | |
| # each shard. To change the count, update this list and SHARD_COUNT. | |
| group: [1, 2, 3, 4, 5, 6, 7, 8] | |
| env: | |
| SHARD_COUNT: 8 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # This repo has no uv.lock (pip/pyproject-managed), so point the | |
| # cache key at the files that define the dependency set. Without | |
| # this, setup-uv defaults to globbing **/uv.lock, finds nothing, | |
| # and errors: "No file matched to [**/uv.lock]". | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| # Setup/install stays under the runner identity. Select test scope also | |
| # stays there because it writes the runner-owned GITHUB_OUTPUT file. | |
| - uses: ./.github/actions/run-as-runner | |
| - name: Select test scope | |
| id: scope | |
| env: | |
| ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }} | |
| LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }} | |
| LEAF_TARGETS: ${{ needs.changes.outputs.leaf_targets }} | |
| TARGETS_FILE: ${{ runner.temp }}/backend-targets.txt | |
| # A frontend-only diff cannot affect a backend test that never touches | |
| # the frontend surface, so we run only the files the selector could NOT | |
| # prove single-surface (every parity guard, plus anything unclassified). | |
| # Every failure path here falls back to the FULL suite on purpose -- | |
| # never trade a missed test for a faster run. | |
| run: | | |
| # Leaf-test verdict wins: it is the narrower and more specific reduction, | |
| # and it is computed once in the `changes` job. `reduced=true` rides along | |
| # so every existing coverage guard keyed on it keeps working unchanged -- | |
| # a subset's coverage is not comparable to the repo floor either way. | |
| if [ "$LEAF_TESTS" = "true" ] && [ -n "${LEAF_TARGETS//[[:space:]]/}" ]; then | |
| printf '%s\n' "$LEAF_TARGETS" | grep . > "$TARGETS_FILE" || true | |
| if [ -s "$TARGETS_FILE" ]; then | |
| echo "reduced=true" >> "$GITHUB_OUTPUT" | |
| echo "leaf=true" >> "$GITHUB_OUTPUT" | |
| echo "Leaf-test scope: $(wc -l < "$TARGETS_FILE") file(s), repeated." | |
| exit 0 | |
| fi | |
| echo "::warning::leaf targets arrived empty -- running the FULL backend suite." | |
| fi | |
| if [ "$ONLY_FRONTEND" != "true" ]; then | |
| echo "reduced=false" >> "$GITHUB_OUTPUT" | |
| echo "Full backend suite (diff is not frontend-only)." | |
| exit 0 | |
| fi | |
| rc=0 | |
| python3 scripts/ci-surface-tests.py --surface backend --summary > "$TARGETS_FILE" || rc=$? | |
| if [ "$rc" != "0" ] || [ ! -s "$TARGETS_FILE" ]; then | |
| echo "::warning::surface selector unusable (rc=$rc) -- running the FULL backend suite." | |
| echo "reduced=false" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| echo "reduced=true" >> "$GITHUB_OUTPUT" | |
| echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface backend file(s)." | |
| - name: Run tests (shard ${{ matrix.group }}/${{ env.SHARD_COUNT }}) | |
| shell: /usr/local/bin/ci-shell {0} | |
| # scripts.ci_file_shards assigns each file to one shard before pytest | |
| # imports it. Discovery and platform ignores remain pytest's own; files | |
| # are never promoted to explicit arguments just to partition the suite. | |
| # Hash assignment is stable, not duration-balanced. Every file's tests | |
| # and parameters stay together; xdist still uses loadgroup within a job. | |
| # | |
| # Coverage runs on every shard: since the floor moved to >= 3.12 the | |
| # matrix has a single Python, so there is no --no-cov lane left. Each | |
| # shard emits a partial data file merged later by the coverage-combine | |
| # job. Keep the package-name boundary: a whole source-directory selector | |
| # also measures vendored and standalone scripts outside that boundary. | |
| # The two explicit directories contain already-measured source whose | |
| # synthetic/app-local module names otherwise make tracing import-order dependent. | |
| env: | |
| PYTHONPATH: ${{ github.workspace }} | |
| REDUCED: ${{ steps.scope.outputs.reduced }} | |
| LEAF: ${{ steps.scope.outputs.leaf }} | |
| LEAF_REPEAT: "3" | |
| LEAF_GATES: ${{ needs.changes.outputs.leaf_gates }} | |
| TARGETS_FILE: ${{ runner.temp }}/backend-targets.txt | |
| run: | | |
| # Reuse payload-free progress on full, reduced and repeated leaf runs. | |
| # A worker crash stays red; do not replace it and lose its failure report. | |
| PROGRESS=(-p scripts.ci_pytest_progress --ci-progress-dir "$RUNNER_TEMP/pytest-progress" --max-worker-restart=0) | |
| # Take the `python` problem matcher registered by actions/setup-python | |
| # out of scope before pytest writes a line. Its pattern is a traceback | |
| # frame followed by `raise SomeError('msg')`, matched anywhere in the | |
| # log -- including inside pytest's warnings summary. MEASURED on the six | |
| # jobs in #7296: it matched a warning traceback every time and a pytest | |
| # failure not once, so every red was annotated `Event loop is closed` | |
| # at asyncio/base_events.py's line 545 while the tests that actually | |
| # failed were named in no annotation at all, and four unrelated PRs | |
| # were misdiagnosed as an event-loop teardown flake. conftest.py's | |
| # `pytest_terminal_summary` now emits the annotations from pytest's own | |
| # report objects; this keeps a louder wrong answer off the check run. | |
| # Not `add-problem-matchers: false`: that input does not exist on this | |
| # action pin, and passing it only adds an "Unexpected input(s)" warning. | |
| echo "::remove-matcher owner=python::" | |
| # Reduced scope (frontend-only diff): run just the cross-surface | |
| # files, still sharded, with coverage off -- a subset's coverage is | |
| # not comparable to the repo floor, so Coverage Combine/Gate skip the | |
| # backend lane for this run (see coverage-gate). | |
| TARGETS=() | |
| if [ "$REDUCED" = "true" ]; then | |
| while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then TARGETS+=("$line"); fi; done < "$TARGETS_FILE" | |
| fi | |
| # Leaf-test scope: run the whole narrow set on every matrix job, NOT | |
| # split across shards -- 14 files do not shard usefully, and the point | |
| # is repeats. Each shard job is then an independent roll on its own | |
| # runner, which is what a flake fix actually needs. Measured | |
| # on this checkout: one pass over the narrow set is ~38s (910 tests) | |
| # against ~29 minutes for the slowest full shard. | |
| if [ "$LEAF" = "true" ]; then | |
| # The corpus gates run ONCE: they are deterministic content assertions, | |
| # so repeating them buys nothing and -- at ~3 minutes a pass over ~158 | |
| # files -- would dominate the run and undo the reduction. Only the | |
| # CHANGED files repeat, because that is where a flake lives. | |
| GATES=() | |
| while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then GATES+=("$line"); fi; done <<< "$LEAF_GATES" | |
| CHANGED=() | |
| for target in ${TARGETS[@]+"${TARGETS[@]}"}; do | |
| keep=1 | |
| for gate in ${GATES[@]+"${GATES[@]}"}; do | |
| if [ "$target" = "$gate" ]; then keep=0; break; fi | |
| done | |
| if [ "$keep" = "1" ]; then CHANGED+=("$target"); fi | |
| done | |
| if [ "${#CHANGED[@]}" -eq 0 ]; then | |
| echo "::error::leaf run resolved no changed files -- refusing to report green." | |
| exit 1 | |
| fi | |
| if [ "${#GATES[@]}" -gt 0 ]; then | |
| echo "::group::leaf corpus gates (${#GATES[@]} file(s), once)" | |
| pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -q -n auto --timeout=120 --no-cov -m "not ipv6_required" ${GATES[@]+"${GATES[@]}"} | |
| echo "::endgroup::" | |
| fi | |
| for pass in $(seq 1 "$LEAF_REPEAT"); do | |
| echo "::group::leaf changed files, pass $pass/$LEAF_REPEAT" | |
| pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -q -n auto --timeout=120 --no-cov -m "not ipv6_required" ${CHANGED[@]+"${CHANGED[@]}"} | |
| echo "::endgroup::" | |
| done | |
| exit 0 | |
| fi | |
| if [ "$REDUCED" = "true" ]; then | |
| python -m pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -p scripts.ci_file_shards -q -n auto --timeout=120 -m "not ipv6_required" \ | |
| --file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \ | |
| --no-cov \ | |
| ${TARGETS[@]+"${TARGETS[@]}"} | |
| exit 0 | |
| fi | |
| python -m pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -p scripts.ci_file_shards -q -n auto --timeout=120 -m "not ipv6_required" \ | |
| --file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \ | |
| --cov=kiro_crew --cov=sage_lib \ | |
| --cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging \ | |
| --cov=src/kiro_crew/apps/builtins/code_review_sage/tests | |
| # The shard union plus backend-test-ipv6 retains the full test population. | |
| # The eleven sandbox files -- the two sandbox suites, the two ops git/gh | |
| # suites, and the seven auto_improvement suites -- | |
| # each carry `skipif(not userns_available())` on the tests that need a namespace | |
| # sandbox, so on this runner (`unshare(CLONE_NEWNS)` -> EPERM) those skip with a | |
| # named reason and the other ~523 run. See the note above the `jobs:` key. | |
| - name: Upload Linux pytest progress | |
| if: ${{ always() }} | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: linux-pytest-progress-${{ matrix.group }} | |
| path: ${{ runner.temp }}/pytest-progress/*.jsonl | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| - name: Stage shard coverage data | |
| shell: /usr/local/bin/ci-shell {0} | |
| if: steps.scope.outputs.reduced != 'true' | |
| # Fold any xdist parallel data files into a single .coverage (no-op if | |
| # pytest-cov already combined this shard), then name it per-shard so the | |
| # combine job can merge all shards with `coverage combine`. | |
| run: | | |
| coverage combine 2>/dev/null || true | |
| test -f .coverage || { echo "::error::shard ${{ matrix.group }} produced no coverage data"; exit 1; } | |
| mv .coverage ".coverage.${{ matrix.group }}" | |
| - name: Upload shard coverage | |
| if: steps.scope.outputs.reduced != 'true' | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: coverage-shard-${{ matrix.group }} | |
| path: .coverage.${{ matrix.group }} | |
| # upload-artifact v4 excludes dotfiles unless this is set. | |
| include-hidden-files: true | |
| if-no-files-found: error | |
| # Windows regression line: the backend supports Windows natively via | |
| # kiro_crew.platform_compat (see docs/guides/windows-install.md for the | |
| # per-feature status), but until this job existed nothing in CI held | |
| # that line -- a POSIX-only regression would land silently. Single | |
| # Python (3.12: the preferred Windows runtime; numpy 1.x ships no 3.13 | |
| # Windows wheel), trace-free (--no-cov: the coverage-combine job merges | |
| # ubuntu shards only). | |
| # | |
| # Whole-file shards, assigned before import using the same plugin as POSIX. | |
| # Eight runners share the work; the per-job timeout remains unchanged. | |
| # File hashing avoids full-suite item collection in every worker, but does not | |
| # claim equal durations across files or platforms. | |
| backend-test-windows: | |
| name: Backend Tests (Windows) | |
| needs: [changes, await-fast-gate] | |
| # All eight shards share the actor-gated large label; absent output is hosted. | |
| runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }} | |
| # Retain the existing Windows budget and coverage-free run. | |
| timeout-minutes: 60 | |
| strategy: | |
| # Full failure picture across shards, same as backend-test. | |
| fail-fast: false | |
| matrix: | |
| group: [1, 2, 3, 4, 5, 6, 7, 8] | |
| env: | |
| SHARD_COUNT: 8 | |
| # windows-latest subprocess spawn + filesystem latency is slow and | |
| # highly variable; the md_notebook git-ops default (30s) intermittently | |
| # times out on plain `git add` here (#1673). Give git invocations a | |
| # wider budget on this job only. | |
| MDNB_GIT_TIMEOUT_SEC: 120 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: ./.github/actions/setup-windows-tests | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # Same no-uv.lock caveat as backend-test: key the cache on the | |
| # files that define the dependency set. | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - name: Verify Windows test prerequisites | |
| if: runner.environment == 'self-hosted' | |
| shell: bash | |
| run: python .github/actions/setup-windows-tests/probe.py | |
| # No surface selector here, unlike the POSIX backend-test job. Narrowing | |
| # this job to the cross-surface subset on a frontend-only diff was | |
| # measured as a PESSIMISATION: 4 shards x ~18 min = 70 job-min for the | |
| # subset, against 62-67 job-min for the whole suite (runs 33412618648 vs | |
| # 33406565382). `-n auto` already saturates each runner, so a smaller | |
| # file list does not shorten a shard, and the cross-surface files are not | |
| # the cheap ones. Running everything is both cheaper and strictly more | |
| # coverage, so there is nothing to select. | |
| - name: Run tests (Windows shard ${{ matrix.group }}/${{ env.SHARD_COUNT }}) | |
| # bash (Git Bash on windows-latest), not pwsh: the invocation shares | |
| # the POSIX job's line-continuation and env-interpolation syntax. | |
| # | |
| # --max-worker-restart=0 overrides setup.cfg's =2 for THIS job only. | |
| # pytest-timeout has no SIGALRM on Windows, so it falls back to the | |
| # thread method, which cannot fail a single test -- it terminates the | |
| # whole worker. Every per-test timeout here therefore lands as | |
| # "node down: Not properly terminated", and xdist then enters its | |
| # node-replacement path, where #2803 documented a broken invariant (a | |
| # node in assigned_work but absent from registered_collections). That | |
| # path either dies with INTERNALERROR or, as in #4227, never completes | |
| # the session and burns the full 40-minute cap. Refusing the | |
| # replacement keeps us out of it: the first dead worker ends the run | |
| # promptly with the victim named. setup.cfg keeps =2 deliberately, to | |
| # absorb a one-off memory-pressure crash on a developer machine; a CI | |
| # runner has no such need. | |
| # | |
| # Measured on windows-latest-equivalent pins (pytest 9.0.3, xdist | |
| # 3.5.0, pytest-timeout 2.2.0) with one test forced past the timeout: | |
| # =2 hung indefinitely with no summary and no victim named, and a | |
| # single slow test took down TWO workers because the crashed item is | |
| # re-dispatched to a live one. =0 exited 1 in 12s and named the test. | |
| # The trade-off is deliberate: refusing the replacement abandons the | |
| # work still pending on that shard, so a killed shard reports fewer | |
| # tests than it collected. That is the right trade for CI -- the job | |
| # is red either way, and a named red in 12 minutes beats an | |
| # uninterpretable one at the 40-minute cap. | |
| shell: bash | |
| env: | |
| PYTHONPATH: ${{ github.workspace }} | |
| run: | | |
| # Same as backend-test (#7296): the `python` matcher annotates any | |
| # traceback in the log, including one printed inside a warning, and | |
| # never matches a pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| python -m pytest -p scripts.ci_pytest_progress -p scripts.ci_file_shards \ | |
| --ci-progress-dir "$RUNNER_TEMP/pytest-progress" \ | |
| -q -n auto --timeout=180 --no-cov --max-worker-restart=0 -m "not ipv6_required" \ | |
| --file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} | |
| # Ordinary shards plus backend-test-ipv6 retain every Windows test. | |
| # Windows has no namespace sandbox, so `userns_available()` is False and every | |
| # sandbox-dependent test skips. Residual POSIX assumptions in | |
| # test_dogfood_learnings / test_pr_watchers stay tracked as node ids in | |
| # test/windows-expected-failures.txt, which is the mechanism a whole-file deselect | |
| # was standing in for. | |
| - name: Upload Windows pytest progress | |
| # Job timeout can skip even always() steps; periodic CI_PROGRESS lines | |
| # retain partial evidence in the Actions log without relying on this upload. | |
| if: ${{ always() }} | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: windows-pytest-progress-${{ matrix.group }} | |
| path: ${{ runner.temp }}/pytest-progress/*.jsonl | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| # Native IPv6 loopback is required here, not enabled on the shared fleet. | |
| # Keep both OS contracts, including Windows' real IPv6 listener discovery. | |
| backend-test-ipv6: | |
| name: Backend Tests (IPv6, ${{ matrix.os }}) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ matrix.os }} | |
| timeout-minutes: 15 | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| os: [ubuntu-latest, windows-latest] | |
| defaults: | |
| run: | |
| shell: bash | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e . --group dev | |
| - name: Run required IPv6 tests | |
| env: | |
| ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }} | |
| LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }} | |
| run: | | |
| echo "::remove-matcher owner=python::" | |
| COVERAGE=(--no-cov) | |
| if [ "$RUNNER_OS" = "Linux" ] && [ "$ONLY_FRONTEND" != "true" ] && [ "$LEAF_TESTS" != "true" ]; then | |
| COVERAGE=(--cov=kiro_crew --cov=sage_lib | |
| --cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging | |
| --cov=src/kiro_crew/apps/builtins/code_review_sage/tests) | |
| fi | |
| # The bounded IPv6 population always runs, even in reduced scope. | |
| # Leaf runs repeat it three times; ordinary corpus gates stay once. | |
| REPEAT=1 | |
| if [ "$LEAF_TESTS" = "true" ]; then REPEAT=3; fi | |
| for pass in $(seq 1 "$REPEAT"); do | |
| python -m pytest -q -n0 --timeout=120 -m ipv6_required \ | |
| ${COVERAGE[@]+"${COVERAGE[@]}"} --junitxml="$RUNNER_TEMP/ipv6.xml" \ | |
| test/test_meetings_providers.py test/test_platform_compat.py | |
| python - <<'PY' | |
| import os | |
| from pathlib import Path | |
| from defusedxml import ElementTree | |
| report = Path(os.environ["RUNNER_TEMP"]) / "ipv6.xml" | |
| cases = ElementTree.parse(report).findall(".//testcase") | |
| assert len(cases) == 5, f"Expected five native IPv6 cases, got {len(cases)}" | |
| skipped = [case.attrib["name"] for case in cases if case.find("skipped") is not None] | |
| expected_skips = [] if os.environ["RUNNER_OS"] == "Windows" else [ | |
| "test_windows_finds_real_ipv6_loopback_listener" | |
| ] | |
| assert skipped == expected_skips, f"Unexpected IPv6 skips: {skipped}" | |
| assert not any(case.find("failure") is not None or case.find("error") is not None for case in cases) | |
| PY | |
| done | |
| - name: Stage IPv6 coverage data | |
| if: runner.os == 'Linux' && needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true' | |
| run: | | |
| test -f .coverage || { echo "::error::IPv6 lane produced no coverage data"; exit 1; } | |
| mv .coverage .coverage.ipv6 | |
| - name: Upload IPv6 coverage | |
| if: runner.os == 'Linux' && needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true' | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: coverage-ipv6 | |
| path: .coverage.ipv6 | |
| include-hidden-files: true | |
| if-no-files-found: error | |
| # Some container kernels omit a dead flock acquirer's PID from their records. | |
| # The orphan test there proves the honest unknown-owner refusal only after | |
| # independently observing the live owner and the later kernel evidence. This | |
| # hosted VM runs the same two tests with KIROCREW_LOCK_OWNER_STRICT=1, which | |
| # refuses that branch: the dead parent must be named exactly, on the runner's | |
| # ordinary filesystem, with no job container and no designated lock root. The | |
| # ordinary shards keep running both tests in their own environment. | |
| backend-test-kernel-lock-owner: | |
| name: Backend Tests (Kernel Lock Owner, hosted) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 15 | |
| defaults: | |
| run: | |
| shell: bash | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e . --group dev | |
| - name: Run strict kernel lock owner tests | |
| env: | |
| KIROCREW_LOCK_OWNER_STRICT: "1" | |
| ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }} | |
| LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }} | |
| run: | | |
| echo "::remove-matcher owner=python::" | |
| test -z "${KIROCREW_LOCK_TEST_ROOT:-}" | |
| test "$(stat -f -c %T /proc)" = proc | |
| COVERAGE=(--no-cov) | |
| if [ "$ONLY_FRONTEND" != "true" ] && [ "$LEAF_TESTS" != "true" ]; then | |
| COVERAGE=(--cov=kiro_crew --cov=sage_lib | |
| --cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging | |
| --cov=src/kiro_crew/apps/builtins/code_review_sage/tests) | |
| fi | |
| # Both strict cases always run, even in reduced scope. Leaf runs | |
| # repeat them three times; ordinary corpus gates stay once. | |
| REPEAT=1 | |
| if [ "$LEAF_TESTS" = "true" ]; then REPEAT=3; fi | |
| for pass in $(seq 1 "$REPEAT"); do | |
| python -m pytest -q -n0 --timeout=120 \ | |
| ${COVERAGE[@]+"${COVERAGE[@]}"} --junitxml="$RUNNER_TEMP/kernel-lock-owner.xml" \ | |
| "test/test_gateway_lock.py::TestLockHolder::test_live_holder_is_named_and_alive" \ | |
| "test/test_gateway_lock_diagnosis.py::test_flock_is_held_by_a_fork_orphan" | |
| python - <<'PY' | |
| import os | |
| from pathlib import Path | |
| from defusedxml import ElementTree | |
| report = Path(os.environ["RUNNER_TEMP"]) / "kernel-lock-owner.xml" | |
| cases = ElementTree.parse(report).findall(".//testcase") | |
| names = sorted(f"{case.attrib['classname']}::{case.attrib['name']}" for case in cases) | |
| assert names == [ | |
| "test.test_gateway_lock.TestLockHolder::test_live_holder_is_named_and_alive", | |
| "test.test_gateway_lock_diagnosis::test_flock_is_held_by_a_fork_orphan", | |
| ], f"Unexpected strict kernel lock owner cases: {names}" | |
| for case in cases: | |
| for outcome in ("skipped", "failure", "error"): | |
| assert case.find(outcome) is None, f"{case.attrib['name']} {outcome}" | |
| PY | |
| done | |
| - name: Stage kernel lock owner coverage data | |
| if: needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true' | |
| run: | | |
| test -f .coverage || { echo "::error::kernel lock owner lane produced no coverage data"; exit 1; } | |
| mv .coverage .coverage.kernel-lock-owner | |
| - name: Upload kernel lock owner coverage | |
| if: needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true' | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: coverage-kernel-lock-owner | |
| path: .coverage.kernel-lock-owner | |
| include-hidden-files: true | |
| if-no-files-found: error | |
| # `kirocrew pod` on Windows, booted for real under Task Scheduler. | |
| # | |
| # A SEPARATE JOB for the same reason the fail-closed canary below is one: a | |
| # step inside backend-test-windows would run in every shard and pay for | |
| # one real gateway boot per shard to make one assertion. It is also the only Windows | |
| # job that needs a built `.venv` INSIDE the checkout, because a pod boots the | |
| # worktree's own binary rather than the control plane's -- the shards install | |
| # with `uv pip install --system` and create no venv at all, which is why | |
| # test_pod_windows_boot.py is listed in test/windows-collect-ignore.txt and | |
| # named explicitly here instead. | |
| # | |
| # What only this job can establish: that the runner user can create a | |
| # scheduled task at all, that `/SC ONCE /ST 00:00` does not auto-fire, that the | |
| # generated .cmd wrapper's quoting is something cmd.exe accepts, that the | |
| # gateway answers over TCP (Windows has no AF_UNIX, so the pod's private | |
| # dashboard socket does not exist there), and that `pod down` really reaps both | |
| # the task and the process tree. Every one of those is faked in the sharded | |
| # test_pod_windows.py, which is what makes this job worth its minutes. | |
| # | |
| # LABEL OPT-IN: a pull request carrying the `ci:pod-scenarios` label ALSO runs | |
| # the full 55-test pod scenario suite (test/e2e/scenarios/) here, against a | |
| # real Vite-built SPA, with the same invocation and count pin as nightly.yml's | |
| # pod-scenarios leg. That is how a current revision gets hosted Windows | |
| # evidence for the suite without dispatching the release-publishing nightly and | |
| # without charging every PR for a Node toolchain, an SPA build, a wheel build | |
| # and a second pod bring-up. Every step the label adds carries the same | |
| # `if: env.POD_SCENARIOS == 'true'`; unlabelled PRs and pushes to main run the | |
| # original boot-only shape, byte for byte. The workflow does not listen for | |
| # `labeled`, so the label must be on the PR BEFORE the push that should run it, | |
| # and a re-run of an earlier run keeps that run's original event payload. | |
| # test/test_pod_scenario_matrix.py pins both shapes. | |
| pod-boot-windows: | |
| name: Pod Boot Canary (Windows) | |
| needs: [await-fast-gate] | |
| # Hosted, and it cannot move. The Windows fleet is a CONTAINER | |
| # (`ContainerAdministrator`), where `schtasks /Create` exits 1 on its own | |
| # `/ST is earlier than current time` warning -- so the runner user cannot | |
| # create a scheduled task, which is the single fact this canary exists to | |
| # establish. Measured there: all 3 tests fail at the probe with | |
| # `WindowsTaskError ... schtasks /Create rc=1`, before any pod boots. | |
| runs-on: windows-latest | |
| # A cold boot pays for the venv build, interpreter start, config seeding and | |
| # the health wait. The test's own bound is 180s; this is the job's. The | |
| # label opt-in measured 76s (boot canary) + 205s (55 scenarios) plus the SPA | |
| # build on a hosted runner, well inside the same ceiling. | |
| timeout-minutes: 30 | |
| env: | |
| # `true` only for a pull_request event whose labels include the opt-in; | |
| # anything else (push to main, an unlabelled PR) is the string `false`. | |
| POD_SCENARIOS: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci:pod-scenarios') }} | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # Same no-uv.lock caveat as backend-test: key the cache on the files | |
| # that define the dependency set. | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| # Label opt-in only: the scenario suite's pod serves the built SPA, so the | |
| # frontend toolchain is a prerequisite there. Same pin and cache shape as | |
| # the `e2e` job. Skipped on the default boot-only run. | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| if: env.POD_SCENARIOS == 'true' | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| # The CONTROL plane: the interpreter that runs pytest and drives the pod | |
| # verbs. Deliberately separate from the payload below, which is the real | |
| # split the pod design rests on. | |
| - name: Install dependencies (control plane) | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| # Label opt-in only: the wheel scenario runs `python -m build --wheel` with | |
| # THIS interpreter (sys.executable) and, under KIROCREW_E2E_REQUIRE=1, fails | |
| # rather than skips without it. `build` is not in the dev group on purpose. | |
| - name: Install the wheel scenario's build tool (label opt-in) | |
| if: env.POD_SCENARIOS == 'true' | |
| run: uv pip install --system build==1.3.0 | |
| # The PAYLOAD: the worktree's own `.venv\Scripts\kirocrew.exe`, which is | |
| # what the scheduled task actually boots. Without this the test FAILS | |
| # naming the missing venv rather than skipping, so a broken step here | |
| # cannot leave the canary silently green. | |
| - name: Build the checkout's own venv (the binary the pod boots) | |
| run: | | |
| uv venv .venv | |
| uv pip install --python .venv/Scripts/python.exe -e "." | |
| # `pod up` refuses a checkout with no served SPA bundle, which is correct | |
| # product behaviour and not what this canary is testing (Task Scheduler | |
| # supervision is). The runner is disposable, so the one-file stand-in is | |
| # placed by the WORKFLOW rather than by the test: a test that wrote into | |
| # the checkout could be killed by its own timeout before cleaning up. | |
| - name: Place a one-file SPA stand-in for `pod up` | |
| if: env.POD_SCENARIOS != 'true' | |
| shell: bash | |
| run: | | |
| mkdir -p src/kiro_crew/static/dist | |
| [ -f src/kiro_crew/static/dist/index.html ] || \ | |
| printf '<!doctype html><title>pod boot canary</title>' > src/kiro_crew/static/dist/index.html | |
| # Label opt-in only: the scenario suite drives the served dashboard and | |
| # packages it into the wheel it installs, so the bundle is the real Vite | |
| # build, staged exactly as the `e2e` job and nightly.yml stage it. It | |
| # replaces the stand-in above, so the boot canary serves the same bundle. | |
| - name: Build and stage the SPA bundle (label opt-in) | |
| if: env.POD_SCENARIOS == 'true' | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| cd website | |
| npm ci --no-audit --no-fund | |
| npm run build | |
| cd .. | |
| rm -rf src/kiro_crew/static/dist | |
| mkdir -p src/kiro_crew/static | |
| cp -R website/dist src/kiro_crew/static/dist | |
| test -f src/kiro_crew/static/dist/index.html | |
| - name: Assert a real pod boots and is reclaimed (not skipped) | |
| shell: bash | |
| env: | |
| # The module's own opt-in: without it the canary skips itself, and the | |
| # grep below turns that skip into a failure. | |
| KIROCREW_E2E_POD_WINDOWS: "1" | |
| run: | | |
| set -o pipefail | |
| # Same as backend-test (#7296): the `python` matcher annotates any | |
| # traceback in the log, including one printed inside a warning, and | |
| # never matches a pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| # The whole opt-in module: the creation probe, the create/run/delete | |
| # round trip, and the full pod boot. These are the only tests in the | |
| # tree that drive the real Task Scheduler, which is why they live in | |
| # one module that is skipped without KIROCREW_E2E_POD_WINDOWS. | |
| pytest -v -n0 --no-cov --timeout=600 \ | |
| test/test_pod_windows_boot.py \ | |
| | tee pod-boot.log | |
| # The canary. `pytest` exits 0 on a fully skipped module, so the exit | |
| # code alone cannot tell "a pod booted and was reclaimed" from "the | |
| # test never ran". Grepping the exact count is what makes this honest; | |
| # raise the number when you add a test to this module. | |
| # ANCHORED on purpose: `grep -q "3 passed"` is a substring match, so it | |
| # also HITs "13 passed" and "23 passed" — measured. A pin that accepts a | |
| # larger number pins nothing when a test is added, which is the direction | |
| # a count pin exists to catch. `^3 passed` cannot be used: pytest's | |
| # summary line starts with `=` padding. | |
| grep -qE '(^|[^0-9])3 passed' pod-boot.log || { | |
| echo "::error::the Windows pod boot canary did not report 3 passed (skipped, not collected, or a test was added or renamed without updating this step)" | |
| exit 1 | |
| } | |
| # Label opt-in only: the full scenario suite, on the prerequisites the | |
| # canary just used, with the SAME invocation and anchored count as | |
| # nightly.yml's pod-scenarios leg so a green here is evidence for that | |
| # matrix entry at THIS revision. `-o addopts=` drops `-n auto` (one pod for | |
| # the whole session, and a killed xdist worker on Windows aborts the run), | |
| # `--timeout=600` is the per-test cap (the wheel scenario carries its own | |
| # wider mark), and `--basetemp` pins the plane under runner.temp so a failed | |
| # run's pod logs land where the upload step below can find them. | |
| # KIROCREW_E2E_REQUIRE=1 turns every precondition skip into a failure; the | |
| # anchored `55 passed` grep is the second lock, for the same reason as the | |
| # canary's: pytest exits 0 on a collected-but-skipped suite. | |
| - name: Run the full pod scenario suite (label opt-in, 55 tests, not skipped) | |
| if: env.POD_SCENARIOS == 'true' | |
| shell: bash | |
| env: | |
| KIROCREW_E2E_SCENARIOS: "1" | |
| KIROCREW_E2E_REQUIRE: "1" | |
| run: | | |
| set -uo pipefail | |
| echo "::remove-matcher owner=python::" | |
| pytest -v -p no:cacheprovider -o addopts= -n0 --timeout=600 \ | |
| --basetemp "${RUNNER_TEMP}/pod-scenarios-tmp" \ | |
| test/e2e/scenarios/ 2>&1 | tee pod-scenarios.log | |
| status="${PIPESTATUS[0]}" | |
| echo "pytest exit: ${status}" | |
| # Six user-flow scenarios and 49 fixture-isolation checks must execute. | |
| # Anchored so 155 passed cannot satisfy the 55-test contract; raise the | |
| # number here AND in nightly.yml when you add a scenario. | |
| if ! grep -qE '(^|[^0-9])55 passed' pod-scenarios.log; then | |
| echo "::error::the pod scenario suite did not report 55 passed on windows-latest; a skipped or collapsed suite is not a pass" | |
| exit 1 | |
| fi | |
| exit "${status}" | |
| # On the default run only pod-boot.log exists; the scenario globs match | |
| # nothing and `if-no-files-found: ignore` keeps that from being a second | |
| # failure. On a labelled run the pod's own log files are the only place a | |
| # boot refusal is legible, and the plane root is the --basetemp above. | |
| - name: Upload the pod boot log | |
| if: failure() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: pod-boot-windows | |
| path: | | |
| pod-boot.log | |
| pod-scenarios.log | |
| ${{ runner.temp }}/pod-scenarios-tmp/kce2e-*/a/** | |
| ${{ runner.temp }}/pod-scenarios-tmp/kce2e-*/h/**/*.log | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| # The Windows fail-closed sandbox opt-in contract, asserted to have RUN. | |
| # | |
| # PR #8117 made `wrap_argv`'s Windows Kiro delegation depend on | |
| # `kiro_internal_sandbox_enabled()`, which reads a settings file a fresh Windows | |
| # install does not have. Delegation stopped, the no-backend policy fail-closed, | |
| # and the ACP backend never started. The suite that pinned that exact branch, | |
| # test_sandbox_argv.py, is line 2 of test/windows-collect-ignore.txt -- so on | |
| # windows-latest it was never collected, and in the Linux shards it reached the | |
| # branch only by monkeypatching sys.platform to "win32" AND patching the | |
| # settings probe to return True, hardcoding the one answer a fresh Windows host | |
| # cannot give. | |
| # | |
| # A SEPARATE JOB rather than a step in backend-test-windows: that job is a | |
| # 4-shard matrix, so a canary step there would run four times and pay the | |
| # gateway boot four times over for one assertion. It also overrides | |
| # --max-worker-restart for xdist reasons that do not apply to an -n0 run. | |
| # | |
| # Run by NODE ID with `-v -n0` and grep the pass count, the same shape as the | |
| # macOS peer-identity canary below and for the same reason: `pytest -q` does not | |
| # name passing tests and a skip exits 0, so a canary that quietly stopped | |
| # running (a changed skipif, a collect-ignore entry, a renamed node) would leave | |
| # this job green while the contract it proves went unverified. That is precisely | |
| # the blindness that let test_sandbox_argv.py sit unrun inside conftest's | |
| # Windows collect_ignore. | |
| # | |
| # The file is deliberately NOT in test/windows-collect-ignore.txt, so the | |
| # backend-test-windows shards collect it too. This job is the assertion that it | |
| # RAN, not the only place it runs. | |
| backend-test-windows-fail-closed: | |
| name: Windows Fail-Closed Sandbox Contract | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }} | |
| # Keep the real gateway/ACP contract and its existing timeout unchanged. | |
| timeout-minutes: 25 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: ./.github/actions/setup-windows-tests | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # Same no-uv.lock caveat as backend-test: key the cache on the files | |
| # that define the dependency set. | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - name: Verify Windows test prerequisites | |
| if: runner.environment == 'self-hosted' | |
| shell: bash | |
| run: python .github/actions/setup-windows-tests/probe.py | |
| # A fresh windows-latest image has no ~/.kiro/settings/amazon-internal.json, | |
| # which IS the precondition under test. Assert it from the workflow as well | |
| # as from the fixture: if a future runner image or an earlier step ever | |
| # created it, the suite would SKIP with a clear reason and the grep below | |
| # would fail -- but this step names the cause on the run page instead of | |
| # leaving it to be read out of a pytest skip line. | |
| - name: Assert the host carries no kiro internal-sandbox settings file | |
| shell: pwsh | |
| run: | | |
| $settings = Join-Path $env:USERPROFILE ".kiro\settings\amazon-internal.json" | |
| if (Test-Path -LiteralPath $settings) { | |
| Write-Host "::error::$settings exists on this runner; the fresh-install precondition this job pins cannot be observed" | |
| exit 1 | |
| } | |
| Write-Host "no $settings on this host, as a fresh Windows install would be" | |
| - name: Assert the Windows fail-closed contract runs (not skipped) | |
| shell: bash | |
| run: | | |
| set -o pipefail | |
| # Same as backend-test (#7296): the `python` matcher annotates any | |
| # traceback in the log, including one printed inside a warning, and | |
| # never matches a pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| pytest -v -n0 --no-cov --timeout=900 \ | |
| test/test_windows_fail_closed_optin.py \ | |
| | tee canary.log | |
| # Anchored: an unanchored "7 passed" also matches "17 passed" / "27 | |
| # passed". See the pod-boot canary above for the measurement. | |
| grep -qE '(^|[^0-9])7 passed' canary.log || { | |
| echo "::error::the Windows fail-closed sandbox contract did not fully PASS (skipped, not collected, or a test was added/removed without updating this count)" | |
| exit 1 | |
| } | |
| # The macOS pytest lane is NOT here any more. It lives in | |
| # .github/workflows/platform-tests.yml, which nightly.yml calls, and it gates the | |
| # nightly PUBLISH jobs there. | |
| # | |
| # It was moved for one MEASURED reason, and the reason is queueing, not runtime. | |
| # On three consecutive green PR runs (34866269260, 34864945056, 34863753125) the | |
| # macOS jobs WAITED 176, 190 and 213 minutes for a macos-15 runner and then ran | |
| # for 26-33. Everything else in this workflow finished at 78-98 minutes; the runs | |
| # themselves took 248-268. So ~64 percent of a pull request's CI wall clock was | |
| # macOS queue time, and "PR Readiness" -- this repository's only required check -- | |
| # is triggered by `workflow_run` on THIS workflow's completion, so that queue sat | |
| # directly on the merge button. Over the same window `Backend Tests (macOS)` | |
| # failed 0 times and was CANCELLED 129 times, mostly still queued when a newer | |
| # push superseded the run: the lane was charging every pull request ~2.8 hours and | |
| # usually returning no verdict at all. | |
| # | |
| # Linux does not have this problem because it does not use GitHub's hosted pool: | |
| # `changes` routes it to this repository's CodeBuild runners. There is no | |
| # self-hosted macOS equivalent, so the queue is not something this workflow can | |
| # tune -- only something it can stop paying per pull request. | |
| # | |
| # NOTHING on the pull_request path may instantiate a macOS runner, and that is | |
| # asserted, not hoped: a `continue-on-error` macOS job would still hold this | |
| # workflow's completion and therefore still hold readiness, so an advisory lane | |
| # cannot live in this file. The opt-in real-Darwin run is its own workflow | |
| # (macos-on-demand.yml, which calls platform-tests.yml), outside readiness' | |
| # lane list. | |
| # test_macos_platform_tests_gate.py pins all of it. | |
| # The job where Linux mount-namespace tests EXECUTE rather than skip. They need | |
| # unprivileged user namespaces (unshare NEWNS), which ubuntu-24.04 restricts via | |
| # AppArmor -- the same restriction codex-review.yml clears for its bwrap sandbox. | |
| # Without this job the ~/.kiro/crew sensitive-path keystone has no CI coverage at all, | |
| # even though branch protection treats breaking it as a blocking offence. | |
| # | |
| # The shards COLLECT all eleven of these files (nothing is deselected any more), but the | |
| # `skipif(not userns_available())` tests inside SIX of them skip there -- so this job is | |
| # the only place those 85 assertions execute, the ~/.kiro/crew sensitive-path keystone | |
| # among them. The other five carry no such guard and run fully in the shards too; they | |
| # are listed here because running them against a REAL sandbox is a different test than | |
| # running them against a stubbed one, and this is the only lane that can. | |
| # | |
| # The list is hand-maintained, so `test_coverage_omit_contract.py` asserts that every | |
| # file carrying the guard appears on the argv below. Without that, a twelfth guarded | |
| # file would skip in every shard, join no lane, and be silently green -- which is | |
| # exactly how these nine came to execute nowhere. | |
| # | |
| # The nine ops/auto_improvement suites were rejected from this job once, as "~6 minutes | |
| # against a real git" and "a poor fit for a 20-minute job". That cost was the launcher's | |
| # pre-exec hardlink scan arming on every spawn (1.5s per sandboxed spawn), since fixed. | |
| # MEASURED afterwards under this job's own environment -- real sandbox, no gh token, no | |
| # user session bus -- all eleven: 694 passed, 1 skipped, 38s. | |
| # | |
| backend-test-sandbox: | |
| name: Backend Tests (namespace sandbox) | |
| needs: [await-fast-gate] | |
| runs-on: ubuntu-24.04 | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| fetch-depth: 2 | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - name: Enable unprivileged user namespaces | |
| run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 | |
| - name: Verify the namespace sandbox is actually available | |
| # Fail LOUDLY if the runner image ever stops allowing this, rather than | |
| # letting the suite silently skip and the gate go green having asserted | |
| # nothing -- the same failure mode the deselects created. | |
| run: unshare --mount --map-root-user true | |
| - name: Verify Kiro Crew's own probe agrees with the kernel | |
| # The probe is stricter than the bare `unshare` above: it also performs the | |
| # launcher's propagation mount inside the new namespace, because a host | |
| # that grants the namespaces and refuses that mount cannot build the | |
| # sandbox. A probe that wrongly said "no sandbox" here would make every | |
| # guarded test below skip and this job go green having asserted nothing, | |
| # so the probe's verdict is asserted on its own, with its reason. | |
| run: | | |
| python3 -c "from kiro_crew.sandbox import userns_available, unavailable_reason; assert userns_available(), unavailable_reason()" | |
| - name: Run sandbox-dependent tests | |
| # `--no-cov` deliberately. This is the only lane that reaches these lines, so | |
| # measuring it is what would let setup.cfg drop the omit for the six guarded test | |
| # files -- but that needs this job's data uploaded as an artifact and | |
| # `coverage-combine` taught to consume it, which couples the coverage gate to a | |
| # job it does not depend on today. Separate change; the omit carries the pointer. | |
| run: | | |
| # Same as backend-test (#7296): the `python` matcher annotates any | |
| # traceback in the log, including one printed inside a warning, and | |
| # never matches a pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| pytest -q --timeout=180 --no-cov \ | |
| test/test_script_hooks.py \ | |
| test/test_cron_script.py \ | |
| src/kiro_crew/apps/builtins/ops_mission_control/tests/test_ledger_sync_git.py \ | |
| src/kiro_crew/apps/builtins/ops_mission_control/tests/test_schedule_file.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_dogfood_learnings.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_lint_findings.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_github_profile.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_profile_capture.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_runner.py \ | |
| src/kiro_crew/apps/builtins/auto_improvement/tests/test_perf_track_propose.py | |
| - name: Require real V1 vector acceleration regressions | |
| timeout-minutes: 5 | |
| # Keep optional accelerator dependencies out of the base shards and | |
| # type-check environment. Reuse installed test dependencies read-only. | |
| run: | | |
| uv venv --system-site-packages "$RUNNER_TEMP/member-faiss-venv" | |
| uv pip install --python "$RUNNER_TEMP/member-faiss-venv/bin/python" "faiss-cpu==1.12.0" | |
| "$RUNNER_TEMP/member-faiss-venv/bin/python" -m pytest -v -n0 --no-cov --timeout=60 \ | |
| test/test_memory_record_metadata.py::test_bulk_episode_edit_invalidates_live_and_saved_faiss_vectors \ | |
| test/test_memory_record_metadata.py::test_another_process_vector_edit_cannot_reuse_old_faiss_scores_after_restart \ | |
| test/test_memory_v1_acceleration.py::test_v1_forgotten_row_keeps_resident_numpy_scoring \ | |
| test/test_memory_v1_acceleration.py::test_v1_faiss_overfetches_past_forgotten_top_hits \ | |
| --junitxml="$RUNNER_TEMP/member-faiss-results.xml" | |
| python - <<'PY' | |
| import os | |
| import xml.etree.ElementTree as ET | |
| from pathlib import Path | |
| report = Path(os.environ["RUNNER_TEMP"]) / "member-faiss-results.xml" | |
| cases = list(ET.parse(report).iter("testcase")) | |
| expected = { | |
| name + "[" + version + "]" | |
| for name in ( | |
| "test_bulk_episode_edit_invalidates_live_and_saved_faiss_vectors", | |
| "test_another_process_vector_edit_cannot_reuse_old_faiss_scores_after_restart", | |
| ) | |
| for version in ("v1", "v2") | |
| } | |
| expected.update( | |
| { | |
| "test_v1_forgotten_row_keeps_resident_numpy_scoring", | |
| "test_v1_faiss_overfetches_past_forgotten_top_hits", | |
| } | |
| ) | |
| assert len(cases) == 6 and {case.attrib["name"] for case in cases} == expected | |
| assert all(case.find(tag) is None for case in cases for tag in ("skipped", "failure", "error")) | |
| PY | |
| - name: Upload FAISS regression evidence | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: member-faiss-regressions | |
| path: ${{ runner.temp }}/member-faiss-results.xml | |
| if-no-files-found: warn | |
| - name: Select optional member recall evidence | |
| id: member-benchmark | |
| if: github.event_name == 'pull_request' | |
| run: | | |
| CHANGED="$(git diff --name-only HEAD^1 HEAD)" | |
| if printf '%s\n' "$CHANGED" | grep -Eq '^(src/kiro_crew/(.*memory.*|embeddings\.py|context\.py|eval/bench/.*)|scripts/ci-member-memory-benchmark\.py|\.github/workflows/ci\.yml)$'; then | |
| echo "selected=true" >> "$GITHUB_OUTPUT" | |
| fi | |
| - name: Locate the pinned member benchmark model | |
| id: member-benchmark-model | |
| if: steps.member-benchmark.outputs.selected == 'true' | |
| continue-on-error: true | |
| env: | |
| KIROCREW_HOME: ${{ runner.temp }}/member-memory-benchmark-home | |
| run: | | |
| python - <<'PY' >> "$GITHUB_OUTPUT" | |
| from kiro_crew import embeddings | |
| print(f"digest={embeddings._GGUF_SHA256}") | |
| print(f"path={embeddings.default_model_path().parent}") | |
| PY | |
| - name: Cache the pinned member benchmark model | |
| if: steps.member-benchmark-model.outputs.digest != '' | |
| continue-on-error: true | |
| uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | |
| with: | |
| path: ${{ steps.member-benchmark-model.outputs.path }} | |
| key: gguf-${{ steps.member-benchmark-model.outputs.digest }} | |
| - name: Measure member recall with the real pinned Qwen model | |
| if: steps.member-benchmark.outputs.selected == 'true' | |
| continue-on-error: true | |
| timeout-minutes: 11 | |
| # Model availability is optional; retrieval scores are observations. | |
| # Completed structural measurements are enforced by the next step. | |
| env: | |
| KIROCREW_HOME: ${{ runner.temp }}/member-memory-benchmark-home | |
| KIROCREW_SKIP_MODEL_DOWNLOAD: "" | |
| MEMBER_BENCHMARK_OUTPUT: ${{ runner.temp }}/member-memory-benchmark-evidence | |
| PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} | |
| run: | | |
| mkdir -p "$MEMBER_BENCHMARK_OUTPUT" | |
| set -o pipefail | |
| if timeout --kill-after=15s 600s python scripts/ci-member-memory-benchmark.py \ | |
| 2>&1 | tee "$MEMBER_BENCHMARK_OUTPUT/run.log"; then | |
| echo "Member recall measurement completed; inspect the artifact's scores, bounds and provenance." >> "$GITHUB_STEP_SUMMARY" | |
| else | |
| echo "::warning::Member recall measurement did not finish cleanly; inspect the benchmark artifact and structural validation." | |
| echo "Read provenance for availability versus completed structural failures; no cached report is substituted." >> "$GITHUB_STEP_SUMMARY" | |
| exit 1 | |
| fi | |
| - name: Enforce completed member recall structural checks | |
| if: always() && steps.member-benchmark.outputs.selected == 'true' | |
| env: | |
| MEMBER_BENCHMARK_OUTPUT: ${{ runner.temp }}/member-memory-benchmark-evidence | |
| run: python scripts/ci-member-memory-benchmark.py --validate-structures | |
| - name: Upload current member recall evidence | |
| if: always() && steps.member-benchmark.outputs.selected == 'true' | |
| continue-on-error: true | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: member-recall-qwen-${{ github.sha }} | |
| path: ${{ runner.temp }}/member-memory-benchmark-evidence/ | |
| retention-days: 30 | |
| if-no-files-found: warn | |
| backend-test-crew-container: | |
| name: Backend Tests (crew container) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 20 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/requirements.txt | |
| # Two installs, deliberately not merged into one resolution. | |
| # | |
| # `-e .` and the dev group are what every other backend lane installs, and this | |
| # suite needs them for a reason that is not obvious: no container test imports | |
| # `kiro_crew`, but the repo-root conftest's autouse isolation fixtures do | |
| # (`importlib.import_module("kiro_crew.apps")`), so a venv without the package | |
| # errors at test SETUP rather than at collection. Measured: 233 errors, every | |
| # one a ModuleNotFoundError on a transitive app dependency. | |
| # | |
| # The image's own runtime pins come from `container/requirements.txt` and are | |
| # installed ONLY here. That file's header forbids them becoming dependencies of | |
| # the application ("the app's backend hooks are imported into every owner's | |
| # gateway process, and a web framework there buys nothing"), which is exactly | |
| # why this cannot be folded into the `backend-test` shards: their environment IS | |
| # the application's. | |
| # | |
| # Not `container/requirements-dev.txt`, which is the file the review named. It | |
| # carries `-r requirements.txt`, pytest and pytest-asyncio and nothing else, | |
| # while this rootdir's addopts pass `-n auto --dist loadgroup | |
| # --max-worker-restart=2 --timeout=120`; with no xdist and no pytest-timeout | |
| # installed that is an argument error, not a test run. Its `pytest==8.3.4` also | |
| # contradicts the dev group's `9.0.3` that the rootdir conftest is written | |
| # against, so the two sets cannot share one resolution. The image build still | |
| # uses that file; CI installs the runtime half plus this repo's own toolchain. | |
| - name: Install dependencies | |
| run: | | |
| uv pip install --system -e "." --group dev | |
| uv pip install --system -r \ | |
| src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/requirements.txt | |
| # CREW_CONTAINER_TESTS_REQUIRED is the point of this job, not a detail of it. | |
| # The suite's conftest answers a missing image dependency with | |
| # `collect_ignore_glob`, which takes the WHOLE suite out of collection while the | |
| # run still reports success. Under this variable every reason it would decline to | |
| # collect becomes a hard error, and the collection is additionally checked against | |
| # the tree: every module that defines tests must yield at least one, and the total | |
| # must clear a floor. So a dependency rename reds THIS lane instead of quietly | |
| # restoring a lane that reports success while collecting nothing. | |
| # | |
| # `-n0`: the suite forks real process trees and binds real ports, so xdist | |
| # workers would race each other for both. It runs in about ten seconds serially, | |
| # which is less than the workers cost to start. | |
| # | |
| # `--no-cov` for the same reason as the sibling sandbox lane: this data is not | |
| # part of the coverage gate's combine, and half-reported coverage is worse than | |
| # none. | |
| - name: Run the crew container suite | |
| env: | |
| CREW_CONTAINER_TESTS_REQUIRED: "1" | |
| run: | | |
| # As in backend-test (#7296): the `python` matcher annotates any traceback | |
| # in the log, including one printed inside a warning, and never matches a | |
| # pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| pytest -q -n0 --timeout=180 --no-cov \ | |
| src/kiro_crew/apps/builtins/aws_control/crew/runtime/container_tests/ | |
| # ── Real-adapter contract lane (the measurements no other runner makes) ── | |
| # The codex and opencode projections rest on facts about the ADAPTER, not about | |
| # Crew: which `mcpServers` element shape a `session/new` accepts, whether the | |
| # named child is really launched, what environment reaches it, and which | |
| # transports are refused outright. Those cannot be asserted from Crew's side, so | |
| # they are measured against a real adapter -- and every guarded test skipped on | |
| # every runner, because no runner had one installed. A guard that never runs | |
| # where drift arrives is the one place drift is invisible: a codex-acp release | |
| # that reshaped the element would break every codex session's MCP surface with | |
| # nothing red to say so. | |
| # | |
| # Per pull request rather than nightly. The facts change when the adapter revs, | |
| # but what makes them load-bearing changes when Crew's own projection changes, | |
| # and this lane is what tells the author of that change. The whole job is ~5 | |
| # minutes with the install cached. | |
| real-adapter-contract: | |
| name: Real Adapter Contract Tests | |
| needs: [changes, await-fast-gate] | |
| # Skipped on a frontend-only diff, like every other backend lane: nothing the | |
| # frontend surface can change moves an adapter contract, and the install is a | |
| # registry download this job should not make to prove that. Deliberately NOT a | |
| # narrower path filter -- what makes these facts load-bearing is any change to | |
| # Crew's projection, and the selector cannot prove which backend file that is. | |
| if: needs.changes.outputs.only_frontend != 'true' | |
| # On the fleet, under the same non-root boundary as the backend shards: the | |
| # adapters are spawned through Crew's own session code, which refuses | |
| # provider execution as root. Setup runs as the runner identity; the | |
| # contract tests and the junit read-back run through ci-shell. | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 30 | |
| env: | |
| # THE point of this job. Every guarded test skips where its adapter is absent | |
| # -- correct on a developer's box, and what kept these contracts unmeasured in | |
| # CI. Here the adapters are installed deliberately, so an absent one is a | |
| # FAILURE: without this a broken npm install produces a green job that | |
| # asserted nothing, which is the state the lane exists to end. | |
| # | |
| # This is the repository's EXISTING switch for "this job declared its | |
| # preconditions must hold" -- the E2E scenario suites, the Playwright gate and | |
| # the gateway-boot matrix all read the same name -- rather than a second | |
| # spelling of it. Each job sets it for its own module set; this job's set is | |
| # the `real_adapter` marker below. | |
| KIROCREW_E2E_REQUIRE: "1" | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| # No extras: these tests drive a subprocess adapter and a stub MCP server, | |
| # and the rootdir conftest needs the package importable. Neither reaches the | |
| # voice or desktop surfaces. | |
| - name: Install dependencies | |
| run: uv pip install --system -e "." --group dev | |
| # Node 24, the version every other lane in this file installs. codex-acp is | |
| # a Node stdio server and the codex guard also asserts `node` is on PATH. | |
| # Node 24, the version every other lane in this file installs. codex-acp is | |
| # a Node stdio server and the codex guard also asserts `node` is on PATH. | |
| # The npm cache is keyed on the adapters' OWN lockfile -- the repo pattern, | |
| # with a different lockfile -- so a warm run pays extraction only. | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: test/real_adapters/package-lock.json | |
| # From the COMMITTED lockfile, never an ad hoc `npm install pkg@X`. The | |
| # manifest in test/real_adapters/ pins one exact version per adapter, and | |
| # `npm ci` installs the closure that lockfile records -- top-level AND | |
| # transitive -- so a cache miss cannot change what this lane measures. A pin | |
| # alone does not do that: `npm install pkg@X` re-resolves the package's own | |
| # dependency ranges on every run (zizmor adhoc-packages). | |
| # | |
| # The same manifest is what test/real_adapter_gate.py reads for the measured | |
| # versions, and what Dependabot refreshes weekly. A bump therefore arrives as | |
| # its own PR, and THAT PR's run of this lane is where adapter drift shows up. | |
| # Nothing here holds a credential: the tests fabricate an API key inside a | |
| # throwaway CODEX_HOME, and session/new makes no model call. | |
| - name: Install the pinned adapters from the lockfile | |
| working-directory: test/real_adapters | |
| run: npm ci --no-audit --no-fund | |
| # ASSERTS the pin held, rather than trusting the install: an adapter whose | |
| # binary cannot start, or a manifest edited without its lockfile, would | |
| # otherwise be measured as if it were the pinned release. The expected | |
| # versions come from the manifest through the gate module -- the one reader | |
| # the tests use -- so this step and the assertions cannot name different | |
| # releases. | |
| - name: Assert the installed adapters are the pinned ones | |
| run: | | |
| python3 - <<'PY' | |
| import json | |
| import subprocess | |
| import sys | |
| sys.path.insert(0, "test") | |
| import real_adapter_gate as gate | |
| modules = gate.MANIFEST_DIR / "node_modules" | |
| manifest = modules / gate.CODEX_ACP_PACKAGE / "package.json" | |
| codex = json.loads(manifest.read_text(encoding="utf-8"))["version"] | |
| assert codex == gate.MEASURED_CODEX_ACP_VERSION, ( | |
| f"codex-acp {codex} installed, {gate.MEASURED_CODEX_ACP_VERSION} pinned" | |
| ) | |
| # The binary's own answer, not its manifest: this is also the proof the | |
| # installed executable starts on this runner. | |
| opencode = subprocess.run( | |
| [str(modules / ".bin" / "opencode"), "--version"], | |
| check=True, capture_output=True, encoding="utf-8", timeout=120, | |
| ).stdout.strip() | |
| assert opencode == gate.MEASURED_OPENCODE_VERSION, ( | |
| f"opencode {opencode!r} installed, {gate.MEASURED_OPENCODE_VERSION} pinned" | |
| ) | |
| print(f"codex-acp {codex}, opencode {opencode}: both the pinned release") | |
| PY | |
| - uses: ./.github/actions/run-as-runner | |
| - name: Run the real-adapter contract tests | |
| shell: /usr/local/bin/ci-shell {0} | |
| # Both resolution ladders read an explicit override first, so the lane | |
| # names the entry it installed instead of measuring whatever else a runner | |
| # image happens to ship. | |
| env: | |
| CODEX_ACP_BIN: ${{ github.workspace }}/test/real_adapters/node_modules/@agentclientprotocol/codex-acp/dist/index.js | |
| OPENCODE_BIN: ${{ github.workspace }}/test/real_adapters/node_modules/.bin/opencode | |
| run: | | |
| # As in backend-test: the `python` matcher annotates any traceback in the | |
| # log, including one printed inside a warning, and never matches a pytest | |
| # failure. | |
| echo "::remove-matcher owner=python::" | |
| pytest -v -n0 --no-cov --timeout=1200 -m real_adapter \ | |
| --junitxml="$RUNNER_TEMP/real-adapter-results.xml" \ | |
| test/test_codex_session_mcp.py \ | |
| test/test_opencode_session_mcp.py | |
| # The second half of the no-silent-skip rule, and the half that does not trust | |
| # the first. KIROCREW_E2E_REQUIRE turns an absent adapter into a failure; this | |
| # asserts on the REPORT that the selected cases ran. A variable that never | |
| # arrived, a marker selection that matched nothing, or a skip added inside a | |
| # test body all reach this step as a skipped or missing case rather than as a | |
| # green lane. | |
| # | |
| # FLOOR, not an inventory: a count can only be raised, so it cannot go stale | |
| # the way a copied list of names does, and its one job is to refuse a run that | |
| # measured FEWER than the contracts known to exist. Which tests those are is | |
| # the marker's business, and the parity test names the files. | |
| - name: Require every contract test to have executed | |
| if: always() | |
| env: | |
| MEASURED_CONTRACT_FLOOR: "4" | |
| run: | | |
| python3 - <<'PY' | |
| import os | |
| import xml.etree.ElementTree as ET | |
| from pathlib import Path | |
| report = Path(os.environ["RUNNER_TEMP"]) / "real-adapter-results.xml" | |
| cases = list(ET.parse(report).iter("testcase")) | |
| floor = int(os.environ["MEASURED_CONTRACT_FLOOR"]) | |
| names = sorted(case.attrib["name"] for case in cases) | |
| assert len(cases) >= floor, f"selected {len(cases)} of at least {floor}: {names}" | |
| for case in cases: | |
| for tag in ("skipped", "failure", "error"): | |
| found = case.find(tag) | |
| assert found is None, ( | |
| f"{case.attrib['name']} reported {tag}: " | |
| f"{found.attrib.get('message', '')[:400]}" | |
| ) | |
| print(f"{len(cases)} real-adapter contract tests executed, none skipped: {names}") | |
| PY | |
| coverage-combine: | |
| name: Coverage Combine | |
| needs: [backend-test, backend-test-ipv6, backend-test-kernel-lock-owner, changes] | |
| # A frontend-only diff runs the backend suite as a coverage-free subset, so | |
| # there are no shard artifacts to combine. Coverage Gate asserts this job is | |
| # exactly `skipped` in that case, and `success` otherwise. | |
| # | |
| # A leaf-test diff is the SAME situation for a different reason, and it needs | |
| # naming separately: it is a BACKEND diff, so `only_frontend` is false, and | |
| # without this clause the job would run and fail on artifacts the leaf lane | |
| # deliberately never uploaded. | |
| if: >- | |
| needs.changes.outputs.only_frontend != 'true' | |
| && needs.changes.outputs.leaf_tests != 'true' | |
| # CodeBuild-hosted runner (pilot); artifact-only job, default size. Rollback: `ubuntu-latest`. | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 15 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install coverage | |
| run: uv pip install --system coverage==7.10.6 | |
| - name: Download shard coverage data | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| pattern: coverage-shard-* | |
| merge-multiple: true | |
| - name: Download required IPv6 coverage data | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| name: coverage-ipv6 | |
| - name: Download required kernel lock owner coverage data | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| name: coverage-kernel-lock-owner | |
| - name: Combine into unified coverage.xml | |
| # The downloaded .coverage.<group> files are parallel data files; | |
| # `coverage combine` merges them. -o pins the output path (setup.cfg's | |
| # [coverage:xml] otherwise redirects it under build/coverage/). | |
| run: | | |
| test -f .coverage.ipv6 || { echo "::error::required IPv6 coverage data missing"; exit 1; } | |
| test -f .coverage.kernel-lock-owner || { echo "::error::required kernel lock owner coverage data missing"; exit 1; } | |
| coverage combine | |
| coverage xml -o coverage.xml | |
| coverage report | |
| - name: Upload combined coverage | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: coverage-backend | |
| path: coverage.xml | |
| # ── Coverage Gate ────────────────────────────────────────────────────── | |
| # Enforces minimum line-coverage thresholds. Rather than re-running the | |
| # entire backend + frontend suites (which duplicated ~30 min of work the | |
| # test jobs already did on a single serial runner), this job CONSUMES the | |
| # coverage artifacts those jobs already produced in this same run: | |
| # * backend-test (3.12 shards) -> coverage-combine -> coverage.xml | |
| # (artifact: coverage-backend) | |
| # * frontend-test -> build/coverage/cobertura-coverage.xml | |
| # (artifact: coverage-frontend) | |
| # Result: download + parse + threshold-check only -- runs in seconds. | |
| # | |
| # Fail-closed: if an expected artifact/report is missing the job errors | |
| # (download-artifact errors on a missing artifact; the report presence is | |
| # asserted below), so a broken coverage pipeline can never silently pass. | |
| # Lives here (not code-review.yml) so it can use `needs:` for a | |
| # deterministic same-run handoff instead of a cross-workflow artifact race. | |
| # The check name "Coverage Gate" is unchanged, so branch protection that | |
| # requires it is still satisfied. | |
| coverage-gate: | |
| name: Coverage Gate | |
| # CodeBuild-hosted runner (pilot); artifact-only job, default size. Rollback: `ubuntu-latest`. | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 10 | |
| needs: [coverage-combine, frontend-test, frontend-coverage-merge, backend-test, backend-test-ipv6, backend-test-kernel-lock-owner, changes] | |
| # Single source of truth for the floors -- referenced by the check steps, | |
| # their names, and the summary table so the three can never drift apart. | |
| # | |
| # RATCHET POLICY: keep each floor a little under what `main` actually | |
| # measures, so the gate blocks a real coverage regression without going red | |
| # on ordinary noise. Raise a floor only after main has held the higher | |
| # number for a while, and never above the LOWEST recent main measurement. | |
| # | |
| # Backend measured 90.57% on main (2026-08-18) -> floor 90% | |
| # Frontend measured 91.32-91.47% on main (2026-08-18) -> floor 90% | |
| # | |
| # Both were sampled across three consecutive successful main runs, because | |
| # the policy above is written against the LOWEST recent measurement and one | |
| # sample cannot establish that. | |
| # | |
| # The headroom is deliberately THIN, and asymmetrically so: frontend keeps | |
| # ~1.5pp, backend only ~0.6pp. At 0.6pp a single merge that adds a few | |
| # hundred uncovered statements reds the gate, so backend coverage now has to | |
| # be maintained with each change rather than recovered in later waves. That | |
| # is the intended trade: the two lanes are held at the round number they | |
| # actually reached. If backend does start bouncing off 90, the fix is to | |
| # lower THAT floor to 89 and say why -- not to weaken the gate's logic. | |
| # | |
| # Why not 85%, which is where this change started: the lanes crossed 90% | |
| # while it sat unmerged, and a floor 5-6pp under main protects almost | |
| # nothing. The point of the gate is to catch a regression, so the number has | |
| # to track what main actually holds. | |
| # | |
| # How backend got here, so the next person can judge the headroom: | |
| # 78.69% starting point | |
| # 80.15% #2090 stopped measuring test files CI never executes (the | |
| # BACKEND_DESELECTS set) -- an accounting fix, not new tests. | |
| # Since retired: CI runs those files, so they are measured again. | |
| # 81.50% #2103 added unit coverage for 7 pure-logic-ish modules | |
| # 81.94% #2115 added 11 dashboard/Slack/app-platform modules | |
| # 86.93% #2947 and 88.84% #3005, ranked by uncovered-statement MASS | |
| # rather than by percentage -- a 550-statement module at 17% is | |
| # worth more than a small one at 60% | |
| # 89.57% #3118, which converted only 58% of the statements it targeted | |
| # where #3005 converted more than it targeted: what remains sits | |
| # behind I/O, subprocess and network boundaries | |
| # 90.57% work merged since | |
| # | |
| # And frontend, which moved later and converts differently: | |
| # 60.46% 2026-08-07, before any dedicated coverage work | |
| # 83.38% the zero-coverage sweep (files with no test file at all) | |
| # 86.14% #2948 and #3014 | |
| # 88.29% #3132, where three surfaces starting at or near 0% produced 739 | |
| # of its 1,470 statements while ChatPage.tsx -- the single largest | |
| # target by uncovered count, 481 -- produced nothing measurable | |
| # 91.47% work merged since | |
| # Two lessons in those two lines, both of which cost a wave to learn: rank | |
| # candidates by how UNCOVERED a file is and not merely by its uncovered | |
| # count, because a file near 0% converts far better than a large page | |
| # already at 80%; and do not size a frontend wave off a backend one. | |
| env: | |
| BACKEND_MIN: 90 | |
| FRONTEND_MIN: 90 | |
| # The per-file floor, applied to BOTH lanes with a per-lane baseline of | |
| # the files that were already below it. A project-wide average lets a | |
| # well-covered large file pay for a bare small one, so it can be satisfied | |
| # without touching the files that carry the risk; a per-file floor removes | |
| # that subsidy. The rationale, the measured cost of this number, and the | |
| # three baseline rules live in scripts/check_per_file_coverage.py's | |
| # docstring -- deliberately not restated here, where it would go stale. | |
| PER_FILE_MIN: 80 | |
| # Always run so the required check emits a real verdict. Without | |
| # `if: always()`, a failing OR skipped dependency would SKIP this job, and | |
| # GitHub treats a skipped required check as SATISFIED -- silently defeating | |
| # the fail-closed guarantee (including the `if-no-files-found: error` upload | |
| # guard this workflow relies on). The first step below converts any | |
| # non-success upstream result into an explicit failure. | |
| if: ${{ always() }} | |
| steps: | |
| - name: Require upstream coverage jobs to have succeeded | |
| env: | |
| CC: ${{ needs.coverage-combine.result }} | |
| LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }} | |
| FE: ${{ needs.frontend-test.result }} | |
| FCM: ${{ needs.frontend-coverage-merge.result }} | |
| BE: ${{ needs.backend-test.result }} | |
| IPV6: ${{ needs.backend-test-ipv6.result }} | |
| KERNEL_LOCK_OWNER: ${{ needs.backend-test-kernel-lock-owner.result }} | |
| ONLY_BACKEND: ${{ needs.changes.outputs.only_backend }} | |
| ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }} | |
| run: | | |
| echo "coverage-combine=$CC frontend-test=$FE frontend-coverage-merge=$FCM backend-test=$BE only_backend=$ONLY_BACKEND only_frontend=$ONLY_FRONTEND" | |
| # Empty flags mean the surface-detection job never reported (it failed | |
| # or was cancelled) -- its verdict can't be trusted, so fail closed. | |
| if [ -z "$ONLY_BACKEND" ] || [ -z "$ONLY_FRONTEND" ]; then | |
| echo "::error::surface detection (changes) never reported -- failing closed." | |
| exit 1 | |
| fi | |
| # Both test jobs always RUN (only their scope narrows), so both must | |
| # always have succeeded -- a subset run that fails is still a failure. | |
| # backend-test is checked explicitly because on a frontend-only diff | |
| # coverage-combine is skipped by design and would otherwise be the only | |
| # signal, masking a failed reduced backend suite. | |
| if [ "$FE" != "success" ]; then | |
| echo "::error::frontend-test=$FE -- failing closed." | |
| exit 1 | |
| fi | |
| # Frontend coverage lane: frontend-coverage-merge unions the shard | |
| # blobs into the coverage-frontend artifact. Required to succeed, | |
| # EXCEPT on a backend-only diff where the frontend suite ran as a | |
| # coverage-free subset and the merge job is skipped by design. | |
| # Anything else (failure, cancellation, dependency-skip, or a run that | |
| # happened when it shouldn't have) fails closed. | |
| if [ "$ONLY_BACKEND" = "true" ]; then | |
| if [ "$FCM" != "skipped" ]; then | |
| echo "::error::frontend-coverage-merge=$FCM but this run was backend-only (expected exactly 'skipped') -- inconsistent." | |
| exit 1 | |
| fi | |
| elif [ "$FCM" != "success" ]; then | |
| echo "::error::frontend-coverage-merge=$FCM -- failing closed." | |
| exit 1 | |
| fi | |
| if [ "$IPV6" != "success" ]; then | |
| echo "::error::backend-test-ipv6=$IPV6 -- failing closed." | |
| exit 1 | |
| fi | |
| # The hosted strict lane is required in every scope, like the IPv6 | |
| # lane: it always runs, so anything but success (failure, cancellation, | |
| # skip, or an empty result) fails closed. | |
| if [ "$KERNEL_LOCK_OWNER" != "success" ]; then | |
| echo "::error::backend-test-kernel-lock-owner=$KERNEL_LOCK_OWNER -- failing closed." | |
| exit 1 | |
| fi | |
| if [ "$BE" != "success" ]; then | |
| echo "::error::backend-test=$BE -- failing closed." | |
| exit 1 | |
| fi | |
| # Backend coverage lane: required to succeed, EXCEPT on a frontend-only | |
| # diff where the backend suite ran as a coverage-free subset and | |
| # coverage-combine is therefore skipped by design. Anything else | |
| # (failure, cancellation, dependency-skip, or a run that happened when | |
| # it shouldn't have) fails closed. | |
| # A leaf-test run is coverage-free by design (the backend lane runs a | |
| # subset, whose coverage is not comparable to the repo floor), so | |
| # coverage-combine is skipped for it exactly as it is for a | |
| # frontend-only run. Checked BEFORE the frontend branch because the two | |
| # flags are independent and a leaf diff sets only this one. | |
| if [ "$LEAF_TESTS" = "true" ]; then | |
| if [ "$CC" != "skipped" ]; then | |
| echo "::error::coverage-combine=$CC but this run was a leaf-test run (expected exactly 'skipped') -- inconsistent." | |
| exit 1 | |
| fi | |
| echo "Leaf-test run: coverage-combine skipped by design; backend coverage not asserted." | |
| elif [ "$ONLY_FRONTEND" = "true" ]; then | |
| if [ "$CC" != "skipped" ]; then | |
| echo "::error::coverage-combine=$CC but this run was frontend-only (expected exactly 'skipped') -- inconsistent." | |
| exit 1 | |
| fi | |
| elif [ "$CC" != "success" ]; then | |
| echo "::error::coverage-combine=$CC -- failing closed." | |
| exit 1 | |
| fi | |
| # Checked out for the per-file gate: it needs scripts/check_per_file_coverage.py | |
| # and the two baselines from the tree. It sits BEFORE the artifact | |
| # downloads because actions/checkout cleans the workspace by default and | |
| # would otherwise delete them. | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| # The gate parses the Cobertura reports through defusedxml rather than the | |
| # stdlib parser, which resolves external entities -- and on a fork PR the | |
| # report is produced by contributor-controlled code. Installed here because | |
| # this job otherwise needs no Python dependencies. | |
| # | |
| # setup-python is a PREREQUISITE, not decoration: the runner image's /usr | |
| # interpreter is externally managed (PEP 668), so `uv pip install --system` | |
| # against it exits 2 with "externally managed". coverage-combine provisions | |
| # the same pair for the same reason. | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| - name: Install the per-file gate's XML parser | |
| run: uv pip install --system 'defusedxml>=0.7,<1' | |
| - name: Download backend coverage | |
| if: needs.coverage-combine.result == 'success' | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| name: coverage-backend | |
| path: backend-cov | |
| - name: Download frontend coverage | |
| if: needs.changes.outputs.only_backend != 'true' | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| name: coverage-frontend | |
| path: frontend-cov | |
| - name: Check backend coverage (min ${{ env.BACKEND_MIN }}%) | |
| if: needs.coverage-combine.result == 'success' | |
| run: | | |
| F=backend-cov/coverage.xml | |
| test -f "$F" || { echo "::error::coverage.xml missing from coverage-backend artifact"; exit 1; } | |
| # Compare the raw line-rate against the threshold; round only for | |
| # display (a round-then-compare would let 89.95%..89.99% pass 90%). | |
| python3 - "$F" "$BACKEND_MIN" Backend <<'PY' | |
| import sys, xml.etree.ElementTree as ET | |
| path, floor, label = sys.argv[1], float(sys.argv[2]), sys.argv[3] | |
| rate = float(ET.parse(path).getroot().attrib["line-rate"]) * 100 | |
| print(f"{label} coverage: {rate:.2f}%") | |
| if rate < floor: | |
| print(f"::error::{label} coverage {rate:.2f}% below {floor:.0f}% minimum") | |
| sys.exit(1) | |
| PY | |
| - name: Check frontend coverage (min ${{ env.FRONTEND_MIN }}%) | |
| if: needs.changes.outputs.only_backend != 'true' | |
| run: | | |
| F=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1) | |
| test -n "$F" || { echo "::error::cobertura report missing from coverage-frontend artifact"; exit 1; } | |
| python3 - "$F" "$FRONTEND_MIN" Frontend <<'PY' | |
| import sys, xml.etree.ElementTree as ET | |
| path, floor, label = sys.argv[1], float(sys.argv[2]), sys.argv[3] | |
| rate = float(ET.parse(path).getroot().attrib["line-rate"]) * 100 | |
| print(f"{label} coverage: {rate:.2f}%") | |
| if rate < floor: | |
| print(f"::error::{label} coverage {rate:.2f}% below {floor:.0f}% minimum") | |
| sys.exit(1) | |
| PY | |
| - name: Check per-file coverage, backend (floor ${{ env.PER_FILE_MIN }}%) | |
| # Gated on the same condition as the project floor: on a frontend-only | |
| # diff the backend suite ran as a coverage-free subset, and a subset's | |
| # per-file rates are not comparable to a baseline recorded on the full | |
| # suite -- enforcing them there would fail files whose tests never ran. | |
| if: needs.coverage-combine.result == 'success' | |
| run: | | |
| python3 scripts/check_per_file_coverage.py backend-cov/coverage.xml \ | |
| --label Backend --floor "$PER_FILE_MIN" \ | |
| --baseline .github/coverage-baselines/backend.txt | |
| - name: Check per-file coverage, frontend (floor ${{ env.PER_FILE_MIN }}%) | |
| if: needs.changes.outputs.only_backend != 'true' | |
| run: | | |
| F=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1) | |
| test -n "$F" || { echo "::error::cobertura report missing from coverage-frontend artifact"; exit 1; } | |
| python3 scripts/check_per_file_coverage.py "$F" \ | |
| --label Frontend --floor "$PER_FILE_MIN" \ | |
| --baseline .github/coverage-baselines/frontend.txt | |
| - name: Coverage summary | |
| if: always() | |
| run: | | |
| echo "## Coverage" >> "$GITHUB_STEP_SUMMARY" | |
| echo "| Component | Coverage | Min |" >> "$GITHUB_STEP_SUMMARY" | |
| echo "|-----------|----------|-----|" >> "$GITHUB_STEP_SUMMARY" | |
| BF=backend-cov/coverage.xml | |
| if [ -f "$BF" ]; then | |
| B=$(python3 -c "import xml.etree.ElementTree as ET; print(f\"{float(ET.parse('$BF').getroot().attrib['line-rate'])*100:.2f}\")") | |
| echo "| Backend | ${B}% | ${BACKEND_MIN}% |" >> "$GITHUB_STEP_SUMMARY" | |
| fi | |
| FF=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1) | |
| if [ -n "$FF" ]; then | |
| FR=$(python3 -c "import xml.etree.ElementTree as ET; print(f\"{float(ET.parse('$FF').getroot().attrib['line-rate'])*100:.2f}\")") | |
| echo "| Frontend | ${FR}% | ${FRONTEND_MIN}% |" >> "$GITHUB_STEP_SUMMARY" | |
| fi | |
| frontend-lint: | |
| name: Frontend Lint & Type Check | |
| needs: [changes, await-fast-gate] | |
| # eslint/tsc/i18n over website/** only; a backend-only diff leaves every | |
| # input byte-identical. only_backend is the catch-all side of the same | |
| # decision, so an unclassifiable path still runs this. | |
| if: needs.changes.outputs.only_backend != 'true' | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 20 | |
| steps: | |
| # persist-credentials retained: frontend-lint/Check for new hardcoded UI strings runs `git fetch origin` | |
| # against the same remote, through the shared diff-base resolver script. | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: Install dependencies | |
| working-directory: website | |
| run: npm ci --no-audit --no-fund | |
| - name: Type check | |
| working-directory: website | |
| # Spelled out rather than `npm run typecheck` so this gate cannot be | |
| # changed from package.json. The project is named explicitly: the root | |
| # tsconfig is `files: []` + project references, so a bare `tsc --noEmit` | |
| # there compiles an empty program. Not `tsc -b`: build mode judges | |
| # "up to date" from the project's own inputs only, so with the incremental | |
| # cache in tsconfig.app.json it would green-light a changed dependency | |
| # `.d.ts` without re-checking a single file. `-p` re-hashes the whole program. | |
| run: npx tsc -p tsconfig.app.json | |
| - name: Lint | |
| working-directory: website | |
| # Blocking, and the tree is at ZERO: the ceiling is a hard zero rather than | |
| # a stored baseline, so any warning a change introduces fails here. A hard | |
| # zero is the only ceiling that needs no bookkeeping — there is no count to | |
| # re-measure and no ceiling that can drift above the truth. | |
| # | |
| # NEVER raise this number. A warning admitted into slack never surfaces | |
| # again: it is indistinguishable from the pre-existing ones and nothing | |
| # downstream reports it. Fix the warning, or — if the code is genuinely | |
| # right — suppress that one line with | |
| # `// eslint-disable-next-line <rule> -- <why the code is correct>`, which | |
| # is reviewable in the diff where a raised ceiling is not. | |
| run: npx eslint src/ --max-warnings 0 | |
| - name: Copy/paste detection | |
| working-directory: website | |
| run: npx jscpd . | |
| - name: Check i18n extraction, key references and plurals | |
| working-directory: website | |
| env: | |
| # Two gates need a commit to diff against: `check-source-strings.mjs` compares | |
| # the catalogs against it to find NEWLY ADDED keys and CHANGED values, and | |
| # `check-i18n-strings.mjs` uses it to find literals on lines this branch wrote. | |
| # On a PR this is `base.sha` — the commit the MERGE REF was computed against — | |
| # NOT `origin/<base.ref>`. The tip is a moving target measured at step time while | |
| # the checked-out tree is a snapshot from job start, so anything that lands on | |
| # `main` in between appears only on the base side and is charged to every PR | |
| # running in that window. Observed: a PR whose run started 36s before #1223 | |
| # merged was billed 58 render findings for `WelcomeView` translations its head | |
| # simply did not have yet. `base.sha` makes the two sides consistent by | |
| # construction. On a push to main it is the commit the push replaced, so a merge | |
| # that only breaks in combination with another merge is still charged to a diff. Note `frontend-lint`'s | |
| # checkout is shallow, so it has to be fetched — see the step below. | |
| # `check-i18n-strings.mjs` falls back from the merge base to the base tip for | |
| # exactly that reason, and EXITS NON-ZERO if a configured ref cannot be | |
| # resolved rather than skipping itself green. | |
| I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }} | |
| # `npm run i18n:check` is a RUNNER (`website/scripts/i18n-check.mjs`), not an `&&` | |
| # chain: it executes all six scripts and reports the eleven checks they contain in | |
| # one table, because `&&` short-circuits and a PR only ever learned about its first | |
| # failing gate. `--check` on the two codemods is deliberately not `--dry-run`: | |
| # dry-run reports and exits 0, which would make this step decorative. An | |
| # unrecognised flag makes those scripts refuse to run rather than fall through to | |
| # their destructive conversion path. | |
| # | |
| # ONLY THREE KINDS OF CHECK CAN FAIL THIS STEP. | |
| # | |
| # DIFF-SCOPED, zero tolerance — a finding on a line you wrote is yours and there | |
| # is no number to raise: | |
| # [added-lines] a user-visible literal on a line THIS BRANCH WROTE. | |
| # [vs-base] a file you touched holds more than it did at the base. | |
| # [source-strings] new ENGLISH keys must be well-shaped copy. | |
| # [changed-values] catalog QA over values this branch added or changed. | |
| # | |
| # WHOLE-REPO HARD ZEROS — no ceiling to inherit, so a new one is always a diff: | |
| # [key-refs] a `t()` naming a key that does not exist. i18next returns a | |
| # missing key as its own fallback, so the symptom is a raw | |
| # dotted key rendered to a user. Necessarily whole-repo: a | |
| # reference and its catalog entry are edited independently, so | |
| # a catalog-only rename dangles references in files the PR | |
| # never opens. Needs no base ref, so it cannot skip itself. | |
| # [plurals] an i18nT-adjacent `{n} {t('item')}s` concatenation. | |
| # [pseudolocale] `en-XA.json` stale against its generator. | |
| # | |
| # WHOLE-REPO CEILING — fails only when the class GROWS: | |
| # [plurals-hardcoded] the same plural-glue defect with NO i18nT in it, e.g. | |
| # a fully hardcoded aria-label gluing `?'s':''` onto a | |
| # noun. The frozen sites are pinned at HARDCODED_CEILING | |
| # in i18n-plural-codemod.mjs and ratchet DOWN as they are | |
| # converted; a failure lists every site with file:line so | |
| # the added one is findable. | |
| # | |
| # EVERYTHING ELSE IS REPORT ONLY — [untranslated], [allcaps], [extractable] and | |
| # [dynamic-keys]. Each is a stored whole-repo total, written by whichever branch | |
| # measured it last, so another branch can push it past its number without touching | |
| # your files — and then the failure names no diff anyone can fix. Measured twice in | |
| # one day: `main` sat at 1120 against its own ALL-CAPS ceiling of 1118 within | |
| # minutes of it being set, and every open PR inherited the red. [dynamic-keys] was | |
| # the worst, failing when a file IMPROVED. The numbers are still printed; they do | |
| # not gate. See docs/ci/i18n-gates.md for the residual gap this leaves on [dynamic-keys] | |
| # specifically, and the qa `CEILINGS` / `deadKeys` / `glossary` ratchets that are | |
| # still exact in both directions. | |
| # | |
| # `check-i18n-strings.mjs` drives a SECOND eslint invocation with its own config. | |
| # `no-literal-string` at `mode: 'all'` reports thousands of findings, and folding | |
| # those into the budget above would make an i18n regression indistinguishable | |
| # from a new `no-explicit-any`. | |
| run: | | |
| # `actions/checkout` fetches depth 1, so the base branch is not present and | |
| # `git show origin/main:…` would fail. Fetch just that one ref. | |
| # | |
| # `set -e` on the chain is the point: `check-source-strings.mjs` SKIPS ITSELF | |
| # and exits 0 when the base ref is unreadable, so a failed fetch used to sail | |
| # past this step and silently disable the new-key gate for the whole PR -- | |
| # observed in practice, where the script printed "OK: skipped -- cannot read | |
| # website/src/i18n/locales/en.json at origin/main (shallow clone?)" and the | |
| # step still went green. A gate that cannot run must fail, not pass: an | |
| # unavailable base ref is a CI problem to fix, not a reason to stop checking. | |
| # Absolute path: this step's working-directory is `website`. | |
| I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \ | |
| "$I18N_BASE_REF" 'the new-key i18n gate')" || exit 1 | |
| export I18N_BASE_REF | |
| npm run i18n:check | |
| - name: Check the color utilities this change touches name real tokens | |
| # A Tailwind utility naming a token the theme never declared is not an | |
| # error anywhere: the class is simply not emitted, the element renders | |
| # with no color, and the build, tsc and eslint all stay silent. Diff- | |
| # scoped for the same reason as the focus-cue gate above: the tree | |
| # carried a backlog of these, so a whole-tree gate would charge it to | |
| # whoever pushed next. Literals a change touched are complete for | |
| # regression, and the whole-tree count is printed as a non-failing | |
| # report either way. | |
| working-directory: website | |
| env: | |
| PHANTOM_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }} | |
| run: | | |
| # Absolute path: this step's working-directory is `website`. | |
| PHANTOM_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \ | |
| "$PHANTOM_BASE_REF" 'the phantom-classes gate')" || exit 1 | |
| export PHANTOM_BASE_REF | |
| # Self-test first: it plants one probe per rule family, so a change | |
| # that breaks the lexer or promotes a non-class string to a class | |
| # fails here instead of shipping a gate that passes everything. | |
| npm run lint:phantom-classes -- --test | |
| npm run lint:phantom-classes | |
| - name: Check no file restyles more ui/ primitives than the baseline allows | |
| # `shadcn/no-restyle` stays off in eslint.config.js: the tree carries a | |
| # few hundred call sites that restyle ui/ primitives and the eslint gate | |
| # is a hard zero. Off means the backlog can grow silently, so this step | |
| # runs the rule through its own config and holds each file at the count | |
| # recorded in website/scripts/restyle-baseline.json — a rise or a new | |
| # file fails; a fall fails too until `--update-baseline` records it, so | |
| # progress is locked in rather than left to a log line. That flag never | |
| # raises a number. Whole-tree, no base ref: the baseline is committed, | |
| # so the gate is complete for regression without a diff. Not eslint's | |
| # own bulk suppressions: the Node API editors use ignores that file, and | |
| # the CLI loads it for every config, so the i18n eslint run would fail | |
| # on "unused" entries (both measured; see the script header). | |
| working-directory: website | |
| run: | | |
| # Self-test first: it plants one restyled primitive and asserts the | |
| # rule fires under this config, so a plugin or parser change that | |
| # silences the rule fails here instead of passing every count as 0. | |
| npm run lint:restyle-ratchet -- --test | |
| npm run lint:restyle-ratchet | |
| # Prove `npm ci` still works on the OLDEST Node the frontend claims to support. | |
| # Every other job here runs the .nvmrc toolchain (Node 24 / npm 11), and npm 11 | |
| # and npm 10 (Node 22) build different ideal trees for an OPTIONAL PEER | |
| # dependency: npm 10 wants a nested entry for it, npm 11 does not install it at | |
| # all. A lockfile regenerated under npm 11 therefore drops what npm 10 demands, | |
| # and `npm ci` dies with EUSAGE ("Missing: <pkg> from lock file") for everyone | |
| # on the declared floor while all of CI stays green. That happened twice to the | |
| # same dependency, the second time as a side effect of a change about something | |
| # else entirely. | |
| # | |
| # The specific divergence is gone -- an `overrides` entry now makes both majors | |
| # resolve that dependency identically, so neither can drop anything. This job | |
| # exists because that is a property of today's dependency tree, not a guarantee: | |
| # the next package with an unsatisfiable optional peer reintroduces the whole | |
| # class, and without a job on the floor nothing would notice. | |
| lockfile-engines-floor: | |
| name: Lockfile Installs On Declared Node Floor | |
| needs: [changes, await-fast-gate] | |
| # Reads website/package.json + website/package-lock.json and nothing else. | |
| if: needs.changes.outputs.only_backend != 'true' | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 10 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| # Pinned to the floor, NOT to `engines.node` via node-version-file: that | |
| # input resolves a range like ">=22" to the NEWEST matching release, which | |
| # would quietly re-test Node 24 and make this job vacuous. The literal is | |
| # held to engines.node statically instead, by | |
| # test_the_engines_floor_job_pins_the_declared_floor -- one derivation of | |
| # the floor, in the file that already governs every other node pin. | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 22 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: npm ci on the engines floor | |
| working-directory: website | |
| # A real install, not `--dry-run`: dry-run still fires the postinstall | |
| # patch script but leaves no node_modules for it to patch, so it fails | |
| # for a reason that has nothing to do with the lockfile. Installing for | |
| # real also makes the gate mean what its name says -- `npm ci` works on | |
| # the declared floor -- rather than only that resolution succeeded. | |
| # | |
| # The failure hint matters because this job is the ONLY one on the floor: | |
| # whoever trips it is on Node 24 and cannot reproduce the red locally, so | |
| # naming the reproduction turns an opaque failure into a self-servicing | |
| # one. | |
| run: | | |
| npm ci --no-audit --no-fund || { | |
| echo "::error::npm ci failed on Node $(node -v), the engines.node floor." \ | |
| "Every other job runs the .nvmrc toolchain (Node 24 / npm 11), so this can" \ | |
| "fail while they pass: the two npm majors resolve optional peer dependencies" \ | |
| "differently. Reproduce without installing Node 22:" \ | |
| "cd website && npx -y npm@10 ci. If it is a lockfile disagreement, prefer" \ | |
| "removing the divergence (an overrides entry both majors resolve the same way)" \ | |
| "over hand-adding whichever entry one major happens to want." | |
| exit 1 | |
| } | |
| cfn-lint: | |
| name: CloudFormation Lint | |
| needs: [changes, await-fast-gate] | |
| # The templates it lints live under src/kiro_crew/deploy/, which the | |
| # paths-filter counts as backend, so a frontend-only diff cannot touch one. | |
| if: needs.changes.outputs.only_frontend != 'true' | |
| # PILOT (first wave): runs on a CodeBuild-hosted runner (project | |
| # `kirocrew-gha-linux`, Ubuntu 22.04) instead of ubuntu-latest. The label is | |
| # chosen once in `changes` (forks and fork PRs get ubuntu-latest, because the | |
| # runner never serves them). Rollback: `ubuntu-latest`. Policy, image caveat | |
| # and exit condition: docs/ci/ci-and-reviews.md, "CodeBuild-hosted runner (pilot)". | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 10 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - name: Install cfn-lint | |
| # Pinned: an unpinned lint tool silently changes the gate's behaviour on | |
| # every upstream release. Bump deliberately. | |
| run: pip install "cfn-lint==1.22.3" | |
| - name: Lint deploy templates | |
| run: cfn-lint src/kiro_crew/deploy/skills/artifact-deploy/templates/*.yaml | |
| # The Electron shell (website/electron) has its own 460+ test node:test suite, | |
| # and until now NOTHING ran it on a PR — its only runner was ota-test.yml, | |
| # which is nightly/on-demand and called by no other workflow. So main-process | |
| # regressions (permission handling, packaging/entitlements contracts, the | |
| # updater IPC surface) could land green and surface a day later, or in a | |
| # shipped DMG. That is not hypothetical: the missing macOS microphone | |
| # entitlement reached users this way. This job is deliberately tiny and fast — | |
| # pure unit tests, no Electron binary download, no packaging. | |
| # NOT diff-gated, deliberately. The obvious gate (only_backend != 'true') | |
| # is wrong: test/packaging.test.js pins the macOS entitlement contract | |
| # against BOTH signing lanes, and one of them is | |
| # packaging/signing/Entitlements.entitlements -- outside website/, so the | |
| # paths-filter counts it as backend. A packaging-only diff would therefore | |
| # skip the job whose absence already shipped a broken bundle once (the | |
| # missing microphone entitlement above). Same for the | |
| # packaging/installer-assets/* references it pins. | |
| electron-test: | |
| name: Electron Shell Tests | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 10 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/electron/package-lock.json | |
| - name: Install dependencies | |
| working-directory: website/electron | |
| # Belt-and-braces: this pin currently downloads no Electron binary | |
| # anyway (electron 43's package ships no install script), so the var is | |
| # inert today — it is set so a future version that reintroduces the | |
| # ~100MB postinstall download cannot silently slow this job down. The | |
| # tests require() only main-process modules and the stdlib test runner, | |
| # so the binary is never needed. | |
| env: | |
| ELECTRON_SKIP_BINARY_DOWNLOAD: "1" | |
| run: npm ci --no-audit --no-fund | |
| - name: Unit tests | |
| working-directory: website/electron | |
| run: npm test | |
| # The Linux job above fakes realpath and Win32_Process. The Windows port-owner | |
| # identity (windows-port.js + gateway-stop.js) rests on two OS premises no | |
| # fake can prove -- realpathSync.native follows an NTFS junction, and a | |
| # process launched through a junction is matched however Win32_Process spells | |
| # its ExecutablePath -- so those test files also run once on a real Windows | |
| # host. No npm ci: the two modules under test require only the Node stdlib. | |
| electron-test-windows: | |
| name: Electron Shell Tests (Windows port identity) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }} | |
| timeout-minutes: 10 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| - name: Windows port-identity tests | |
| working-directory: website/electron | |
| run: node --test test/windows-port.test.js test/windows-port-junction.test.js test/windows-port-real-junction.test.js | |
| frontend-test: | |
| name: Frontend Tests | |
| needs: [changes, await-fast-gate] | |
| # CodeBuild-hosted runner (pilot, second wave). `instance-size:large` for the | |
| # vitest --coverage shards' memory; routing and rollback as for cfn-lint. See | |
| # docs/ci/ci-and-reviews.md, "CodeBuild-hosted runner (pilot)". | |
| # Rollback: `ubuntu-latest`. | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| # Per-shard budget. The whole suite was ~23 min on one runner; split four | |
| # ways each shard is well under this, and a hung shard fails fast. | |
| timeout-minutes: 25 | |
| strategy: | |
| # Don't cancel sibling shards when one fails -- we want the full failure | |
| # picture across the suite, not just the first shard to break. | |
| fail-fast: false | |
| matrix: | |
| # vitest splits by TEST FILE (not test case) across shards. 4 shards | |
| # cuts the ~950-file suite roughly fourfold and confines a fork-worker | |
| # crash to one shard instead of reddening the whole job. To change the | |
| # count, edit this list AND SHARD_COUNT together -- mind the ~20 | |
| # concurrent-job ceiling for public repos. | |
| shard: [1, 2, 3, 4] | |
| env: | |
| SHARD_COUNT: 4 | |
| steps: | |
| # persist-credentials retained: frontend-test/Run the diff-scoped vitest gates runs `git fetch origin` | |
| # against the same remote, through the shared diff-base resolver script. | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: Install dependencies | |
| working-directory: website | |
| run: npm ci --no-audit --no-fund | |
| - name: Select test scope | |
| id: scope | |
| env: | |
| ONLY_BACKEND: ${{ needs.changes.outputs.only_backend }} | |
| TARGETS_FILE: ${{ runner.temp }}/frontend-targets.txt | |
| # Mirror of the backend selector: a backend-only diff cannot affect a | |
| # spec that never touches the backend surface. Cross-surface specs (and | |
| # anything unclassified) still run. Any failure falls back to the FULL | |
| # suite rather than silently narrowing it. | |
| run: | | |
| if [ "$ONLY_BACKEND" != "true" ]; then | |
| echo "reduced=false" >> "$GITHUB_OUTPUT" | |
| echo "Full frontend suite (diff is not backend-only)." | |
| exit 0 | |
| fi | |
| rc=0 | |
| python3 scripts/ci-surface-tests.py --surface frontend --summary > "$TARGETS_FILE.all" || rc=$? | |
| if [ "$rc" != "0" ] || [ ! -s "$TARGETS_FILE.all" ]; then | |
| echo "::warning::surface selector unusable (rc=$rc) -- running the FULL frontend suite." | |
| echo "reduced=false" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| # vitest runs with cwd=website and only covers website/src; the | |
| # Electron shell specs belong to electron-test, which stays always-on. | |
| sed -n 's#^website/##p' "$TARGETS_FILE.all" | grep -v '^electron/' > "$TARGETS_FILE" || true | |
| if [ ! -s "$TARGETS_FILE" ]; then | |
| echo "::warning::no in-scope cross-surface specs resolved -- running the FULL frontend suite." | |
| echo "reduced=false" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| echo "reduced=true" >> "$GITHUB_OUTPUT" | |
| echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface spec(s)." | |
| - name: Unit tests (shard ${{ matrix.shard }}/${{ env.SHARD_COUNT }}) | |
| working-directory: website | |
| env: | |
| # Two vitest gates are DIFF-SCOPED rather than baseline-scoped — | |
| # `localeFormatting.test.ts`'s `[added-lines]` / `[vs-base]`. They read the | |
| # base commit out of git instead of storing a count, and they skip themselves | |
| # when this is empty. Without it they would skip on every PR, which is the | |
| # "gate nobody watched go red" failure mode their own docs warn about. | |
| # Same expression as `frontend-lint`: the base branch on a PR, the commit the | |
| # push replaced on a push to main. | |
| I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }} | |
| REDUCED: ${{ steps.scope.outputs.reduced }} | |
| TARGETS_FILE: ${{ runner.temp }}/frontend-targets.txt | |
| # `actions/checkout` fetches depth 1, so the base is not present and | |
| # `git diff`/`git show` against it would fail. Fetch just that one commit, and | |
| # fail the step if it cannot be fetched — the gates throw on an unresolvable | |
| # configured ref rather than skipping green, and this keeps the two consistent. | |
| run: | | |
| I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \ | |
| "$I18N_BASE_REF" 'the diff-scoped i18n test gates')" || exit 1 | |
| export I18N_BASE_REF | |
| if [ "$REDUCED" = "true" ]; then | |
| # Subset run (backend-only diff): coverage of a subset is not | |
| # comparable to the repo floor, so it is not collected and the | |
| # Coverage Gate skips the frontend lane for this run. Still sharded | |
| # and blob-reported so the merge job produces one unified result. | |
| TARGETS=() | |
| while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then TARGETS+=("$line"); fi; done < "$TARGETS_FILE" | |
| # Still sharded so the matrix step is uniform; a target set smaller | |
| # than SHARD_COUNT leaves some shards with no files, so | |
| # --passWithNoTests keeps an empty shard green (a legitimately empty | |
| # slice, not a lost suite). | |
| # blob alone prints no verdict (it only writes the artifact), so a | |
| # failing shard's log would end at "exit code 1" with no FAIL line | |
| # or summary; the second reporter keeps a human-readable verdict in | |
| # this job's own log. | |
| npx vitest run --reporter=blob --reporter=default --passWithNoTests \ | |
| --shard=${{ matrix.shard }}/"$SHARD_COUNT" ${TARGETS[@]+"${TARGETS[@]}"} | |
| else | |
| # --reporter=blob stores results + per-shard coverage under | |
| # website/.vitest-reports/; the frontend-coverage-merge job stitches | |
| # the four shards back into one report and the merged cobertura the | |
| # Coverage Gate reads. Coverage is emitted per shard so the merge | |
| # can union it -- a single shard covers only the files it ran. | |
| # blob alone prints no verdict (it only writes the artifact), so a | |
| # failing shard's log would end at "exit code 1" with no FAIL line | |
| # or summary; the second reporter keeps a human-readable verdict in | |
| # this job's own log. | |
| npx vitest run --coverage --reporter=blob --reporter=default \ | |
| --shard=${{ matrix.shard }}/"$SHARD_COUNT" | |
| fi | |
| - name: Upload shard blob report | |
| # Always upload (even a failed shard) so the merge job can render the | |
| # full picture; skipped only if the run itself was cancelled. The blob | |
| # carries this shard's results AND its coverage data; the merge job | |
| # unions them. .vitest-reports is a dot dir -> include-hidden-files. | |
| if: ${{ !cancelled() }} | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: frontend-blob-${{ matrix.shard }} | |
| path: website/.vitest-reports/* | |
| include-hidden-files: true | |
| if-no-files-found: error | |
| retention-days: 1 | |
| frontend-coverage-merge: | |
| name: Frontend Coverage Merge | |
| needs: [frontend-test, changes] | |
| # Runs whenever the sharded run collected coverage -- any diff that is not | |
| # backend-only. (A backend-only diff runs the frontend suite as a | |
| # coverage-free subset, so there is nothing to merge and the Coverage Gate | |
| # skips the frontend lane.) `!cancelled()` lets it still stitch a report | |
| # when a shard failed, so the failure surfaces in one place. | |
| # | |
| # The `!= 'skipped'` clause is what keeps that `!cancelled()` honest now that | |
| # the shards sit behind await-fast-gate: a red fast gate skips frontend-test, | |
| # and without this the merge would still run, find no coverage artifact, and | |
| # go red for a reason that has nothing to do with the gate the developer has | |
| # to fix. A skipped shard set has nothing to stitch; a FAILED one still does. | |
| if: >- | |
| ${{ !cancelled() | |
| && needs.frontend-test.result != 'skipped' | |
| && needs.changes.outputs.only_backend != 'true' }} | |
| # CodeBuild-hosted runner (pilot), default size: a merge is one process | |
| # re-reading four blobs, not a test run. Rollback: `ubuntu-latest`. | |
| runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }} | |
| timeout-minutes: 15 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: Install dependencies | |
| working-directory: website | |
| run: npm ci --no-audit --no-fund | |
| - name: Download shard blob reports | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| path: website/.vitest-reports | |
| pattern: frontend-blob-* | |
| merge-multiple: true | |
| - name: Normalize blob paths to this runner's workspace | |
| # A blob stores every path absolute, and vitest's coverage map is keyed | |
| # by those paths. Hosted runners all check out under the same root, so | |
| # the shards and this job agree by accident; the CodeBuild-hosted runner | |
| # gives every build its own root (/codebuild/output/src<random>/...), so | |
| # four shards arrive with four roots and the merge unions nothing: each | |
| # file is reported four times with one shard's numbers and the step | |
| # exits non-zero with no failing test (pilot runs 1 and 2, 2026-09-12). | |
| # This rewrites each blob's root to this checkout's; a no-op when they | |
| # already match, so hosted runs are unaffected. | |
| working-directory: website | |
| run: node "$GITHUB_WORKSPACE/.github/scripts/frontend-blob-normalize-paths.mjs" .vitest-reports | |
| - name: Merge reports and coverage | |
| id: merge | |
| working-directory: website | |
| # --merge-reports stitches the per-shard blobs back into one result set | |
| # WITHOUT re-running tests; --coverage unions the per-shard coverage and | |
| # re-emits the configured reporters (cobertura/lcov) under | |
| # build/coverage/ -- the exact path + artifact name the Coverage Gate | |
| # already consumes. A failed shard makes this exit non-zero, surfacing | |
| # the failure here too. | |
| run: npx vitest --merge-reports --coverage | |
| - name: Reconcile shard verdicts against blob results | |
| # Runs even when the merge step failed -- that is the case it exists | |
| # for. A shard (or the merge) can exit non-zero without naming a | |
| # failing test anywhere: a worker killed mid-run (the worker-OOM mode | |
| # documented in website/vite.config.ts) fails the job while every test | |
| # in the blob reads "pass". This step decodes the downloaded blobs, | |
| # prints every failing test and unhandled runner error by name, and | |
| # states the contradiction explicitly when a red verdict has no failing | |
| # test behind it. Diagnostic only: it always exits 0 and never changes | |
| # the job's verdict. | |
| if: ${{ !cancelled() }} | |
| working-directory: website | |
| env: | |
| FRONTEND_TEST_RESULT: ${{ needs.frontend-test.result }} | |
| MERGE_RESULT: ${{ steps.merge.outcome }} | |
| run: node "$GITHUB_WORKSPACE/.github/scripts/frontend-blob-reconcile.mjs" .vitest-reports | |
| - name: Upload merged coverage | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: coverage-frontend | |
| # vitest's reportsDirectory is website/build/coverage (see | |
| # vite.config.ts). Fail loudly if the cobertura report is missing so a | |
| # broken merge can never hand the Coverage Gate an empty artifact. | |
| path: website/build/coverage/ | |
| if-no-files-found: error | |
| bundle-size: | |
| name: Bundle Size Gate | |
| needs: [changes, await-fast-gate] | |
| # The analyze build reserves a 6 GiB Node heap; use the large memory tier. | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| timeout-minutes: 25 | |
| # A backend-only diff cannot change the frontend bundle, so this gate has | |
| # nothing to measure there. Meta diffs (.github/**, scripts/**) leave both | |
| # surface flags false and still run the gate -- this job's own config lives | |
| # in that bucket. | |
| if: needs.changes.outputs.only_backend != 'true' | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: Install dependencies | |
| working-directory: website | |
| run: npm ci --no-audit --no-fund | |
| - name: Build in analyze mode | |
| working-directory: website | |
| # `--mode analyze` activates the kirocrew-bundle-report plugin (see | |
| # vite.config.ts), which emits dist/bundle-report.json with per-chunk | |
| # sizes. A normal `npm run build` deliberately never writes one, so the | |
| # gate below measures a dedicated build instead of taxing the real one. | |
| # The memory flag mirrors package.json's `build` script. | |
| run: node --max-old-space-size=6144 ./node_modules/vite/bin/vite.js build --mode analyze | |
| - name: Per-chunk size gate | |
| working-directory: website | |
| # Explicit ceilings for known-large chunks, a 500 KB default for every | |
| # other chunk; fails with one line per breach. Budgets and rationale: | |
| # website/scripts/check-bundle-size.mjs (CHUNK_BUDGETS). | |
| run: node scripts/check-bundle-size.mjs | |
| - name: Chunk cycle gate | |
| working-directory: website | |
| # Reuses the analyze build above rather than paying for a second one, which | |
| # is why it lives in this job instead of its own. The job KEEPS its name: | |
| # `Bundle Size Gate` is a required check, and renaming it to match the | |
| # widened scope would silently stop satisfying branch protection. | |
| # | |
| # Different failure from the size gate, and one nothing else here can see. | |
| # Two chunks that statically import each other have no valid | |
| # initialization order, so a body runs against an uninitialized binding | |
| # and the page blanks before React mounts. Bytes stay fine, tsc stays | |
| # fine, and the unit suite never loads a bundle -- a config edit | |
| # (rolldown's `includeDependenciesRecursively: false`) produced two cycles | |
| # with every other gate green, which is what this closes. | |
| run: node scripts/check-chunk-cycles.mjs | |
| # The browser gate runs on the CodeBuild fleet. The one step that needed a | |
| # user namespace -- the real private-workflow MCP run -- moved to | |
| # `e2e-private-namespace` below, the only piece of this lane the fleet cannot | |
| # host (`unshare --map-root-user` is EPERM there). Everything left here needs | |
| # a gateway, a Node toolchain and a Chromium, all of which the fleet has. | |
| # | |
| # The suite's own gateway takes unsandboxed consent for its disposable config | |
| # (`test/test_playwright_e2e.py`, `sandbox_allow_unsandboxed_exec`): the agent | |
| # binary is a stdlib echo stub, and the container refuses `CLONE_NEWUSER` at | |
| # the runtime policy level rather than through a sysctl this job could clear. | |
| # A sandboxed spawn doing real work stays proven by `e2e-private-namespace` | |
| # and by `e2e-boot-matrix`, which asserts both sandbox tiers. | |
| # | |
| # Same non-root boundary as `e2e-boot-matrix`: the gateway refuses provider | |
| # execution under euid 0, so the tests run through ci-shell as `runner`. | |
| # Setup (installs, the SPA build, the browser download) runs before the | |
| # boundary under the runner identity and writes GITHUB_OUTPUT / GITHUB_ENV, | |
| # which the boundary then makes read-only; the test steps after it write only | |
| # the workspace and RUNNER_TEMP. `PLAYWRIGHT_BROWSERS_PATH` pins the browser | |
| # to RUNNER_TEMP (written to GITHUB_ENV by the version step, since job-level | |
| # env cannot read the runner context) so the cache restore (root, HOME=/root) | |
| # and the specs (runner, HOME=/home/runner) resolve the same directory. | |
| e2e: | |
| name: E2E (stub ACP backend, offline) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| timeout-minutes: 25 | |
| defaults: | |
| run: | |
| shell: /usr/local/bin/ci-shell {0} | |
| steps: | |
| # persist-credentials retained: e2e/Scan the rendered dashboard runs `git fetch origin` | |
| # against the same remote, through the shared diff-base resolver script. | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| cache: pip | |
| - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: 24 | |
| cache: npm | |
| cache-dependency-path: website/package-lock.json | |
| - name: Install backend (+ dev) | |
| # setuptools is explicit: Python 3.12 no longer bundles it, and an | |
| # editable install doesn't add it to the runtime venv -- but | |
| # `setup.py test_e2e` imports setuptools.Command, so it must be present. | |
| shell: bash | |
| run: | | |
| python -m pip install --upgrade pip setuptools | |
| pip install -e ".[voice,desktop]" --group dev | |
| - name: Install frontend deps | |
| shell: bash | |
| working-directory: website | |
| run: npm ci --no-audit --no-fund | |
| - name: Build frontend | |
| shell: bash | |
| working-directory: website | |
| run: npm run build | |
| - name: Stage frontend into package | |
| shell: bash | |
| # The dashboard Playwright specs render the bundled UI the harness | |
| # gateway serves from src/kiro_crew/static/dist. Mirror nightly.yml's | |
| # build-wheel staging so the specs hit a real dashboard, not a 404. | |
| run: | | |
| rm -rf src/kiro_crew/static/dist | |
| mkdir -p src/kiro_crew/static | |
| cp -R website/dist src/kiro_crew/static/dist | |
| # The browser install is the single largest source of red on this job, and | |
| # never because a test failed: across 112 consecutive E2E jobs the | |
| # conclusions were 76 success / 36 cancelled / 0 failure, and 22 of those | |
| # cancels trace to this step. `--with-deps` runs `apt-get update` first, and | |
| # when the runner's default mirror (azure.archive.ubuntu.com) answers `Ign:` | |
| # apt falls back to archive.ubuntu.com and stalls silently -- observed at | |
| # 23min, 15min and 12min, against a 25min job budget that then has nothing | |
| # left for the specs. | |
| # | |
| # `--with-deps` is dropped rather than retried because it buys nothing this | |
| # gate asserts on. Every shared library Chromium needs is already on the | |
| # ubuntu-latest image (the apt log reports libnss3, libx11-6, libxcb1, | |
| # libasound2 and libxkbcommon0 as `already the newest version`); the only | |
| # packages it newly installs are 9 CJK/Thai/Cyrillic font packages. The suite | |
| # has no pixel comparison (no toHaveScreenshot / toMatchSnapshot anywhere | |
| # under website/), playwright.config.ts pins `locale: 'en-US'` so every spec | |
| # renders the English catalog, and the i18n render gate below reads | |
| # textContent rather than measuring geometry. A future spec that asserts | |
| # glyph METRICS for a non-Latin script would need those fonts back -- add | |
| # them as their own bounded, non-fatal step, not by restoring `--with-deps`. | |
| # | |
| # The version key must be exact: a restore-key prefix could hand the job a | |
| # different Chromium revision than @playwright/test expects. | |
| - name: Resolve Playwright version | |
| id: playwright-version | |
| shell: bash | |
| working-directory: website | |
| run: | | |
| version="$(node -p \ | |
| "require('./package-lock.json').packages['node_modules/@playwright/test'].version")" | |
| echo "version=$version" >> "$GITHUB_OUTPUT" | |
| echo "PLAYWRIGHT_BROWSERS_PATH=$RUNNER_TEMP/ms-playwright" >> "$GITHUB_ENV" | |
| - name: Cache Playwright browsers | |
| uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | |
| with: | |
| path: ${{ runner.temp }}/ms-playwright | |
| key: ms-playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}-chromium | |
| # Runs unconditionally rather than behind `cache-hit != 'true'`: on a hit it | |
| # is a ~2s no-op, and that keeps a partially-written cache entry from turning | |
| # into a missing-browser failure inside the specs. | |
| - name: Install Playwright browser (Chromium) | |
| # A step-level cap. The download itself takes ~7s, so anything approaching | |
| # this is a stalled mirror or CDN -- fail here with budget left for the | |
| # specs instead of consuming the job's 25min and reporting nothing. | |
| timeout-minutes: 6 | |
| shell: bash | |
| working-directory: website | |
| run: npx playwright install chromium | |
| - name: MCP-App message rows render both delivery states (asserting harness) | |
| # The same self-asserting harness that produced the PR's screenshot | |
| # evidence: serves the built SPA over loopback with fixture-backed | |
| # /api/** responses and fails on any scene assertion (envelope leak, | |
| # missing attribution, missing text). Lives here to reuse the build | |
| # and the Chromium install above; needs no gateway, token or backend. | |
| timeout-minutes: 5 | |
| shell: bash | |
| working-directory: website | |
| run: node scripts/capture-mcp-app-message-rows.mjs "$RUNNER_TEMP/mcp-app-message-rows" | |
| - name: Resolve base for the parallel i18n render-time gate | |
| # Phase 5 of the i18n plan. Every other i18n gate reads SOURCE or CATALOG | |
| # JSON, so three defect classes are invisible to all of them: a string that | |
| # was never extracted (ordinary TypeScript to a scanner), a sentence | |
| # assembled from several keys (several correct lookups), and English in a | |
| # title/aria-label (never in the text flow). This renders the real SPA under | |
| # the en-XA pseudolocale and reads them off the DOM. | |
| # | |
| # It lives in this job to reuse the Chromium install on the step above -- | |
| # its own job would re-download the browser for a ~5min script. It needs no | |
| # gateway, no token and no backend: the script serves the build over | |
| # loopback and answers every /api/** call from fixtures, like the | |
| # capture-*.mjs harnesses. It builds its OWN bundle rather than reusing | |
| # `npm run build` above, because en-XA is DEV-only and a production build | |
| # tree-shakes the catalog out entirely. | |
| # | |
| # Two runs, one per tree: [vs-base] renders the BASE commit with the same | |
| # scanner and fails on any per-surface increase, reading no committed number. | |
| # It is the ONLY thing that fails; the debt record in | |
| # `src/i18n/render-baseline.json` is printed and never enforced, because a | |
| # stored total is written by whichever branch measured it last and two branches | |
| # that never touch the same file can combine into a number neither produced -- | |
| # `projects.latent 16 -> 24` from #1107 + #985, red on main with no diff anyone | |
| # could fix. A push to main therefore gets a base too (the commit it replaced), | |
| # which is what keeps that interaction enforced somewhere. It costs a second | |
| # bundle build on merges, the same ~5min every PR already pays. | |
| # The base must be fetched first because this checkout is shallow; the script | |
| # EXITS NON-ZERO if a configured ref cannot be resolved rather than skipping | |
| # itself green, for the reason the frontend-lint job documents. | |
| shell: bash | |
| working-directory: website | |
| env: | |
| I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }} | |
| run: | | |
| I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \ | |
| "$I18N_BASE_REF" 'the diff-scoped render gate')" || exit 1 | |
| printf 'I18N_BASE_REF=%s\n' "$I18N_BASE_REF" >> "$GITHUB_ENV" | |
| # Every step below runs as `runner` through ci-shell (the job default). | |
| # Nothing after this point writes a GitHub file command; the i18n base ref | |
| # above is the last one. | |
| - uses: ./.github/actions/run-as-runner | |
| # The SkillForm block-scalar parity guard (issue #7763). It runs the REAL | |
| # TypeScript mirror (backendReadsValue -> backendFoldsLiteral, transpiled | |
| # with the website esbuild) against the Python read path over a generated | |
| # corpus, so the editor's simulation of the reader cannot drift from the | |
| # reader. It lives HERE because it is the only pull-request lane with BOTH | |
| # runtimes present AND usable: the backend is pip-installed above (Python + | |
| # kiro_crew) and `npm ci` has populated website/node_modules (node + the | |
| # `yaml` package the mirror imports + esbuild). The sharded backend-test job | |
| # has Python but no website node deps, and frontend-test has node but no | |
| # Python, so in each of those the file skips with a named reason -- which is | |
| # why REQUIRE=1 is set only here, turning any skip into a failure so the | |
| # guard cannot report green having run nothing. Anchored pass count, the same | |
| # canary idiom as the boot matrices above: pytest exits 0 on a fully skipped | |
| # module, so the count is what proves the corpus actually ran. | |
| - name: SkillForm frontmatter parity (TS mirror vs Python read path) | |
| env: | |
| KIROCREW_SKILLFORM_PARITY_REQUIRE: "1" | |
| run: | | |
| set -o pipefail | |
| # Order-safe: the harness file lands in a separate PR. Until it exists | |
| # this step is a no-op, so merging this workflow change first cannot red | |
| # the lane on a tree without the harness. Once the file is present the | |
| # REQUIRE variable above turns any skip into a failure. | |
| if [ ! -f test/test_skillform_frontmatter_parity.py ]; then | |
| echo "::notice::SkillForm parity harness not present yet; step is a no-op until it lands." | |
| exit 0 | |
| fi | |
| echo "::remove-matcher owner=python::" | |
| python -m pytest -v -n0 --no-cov -p no:cacheprovider --timeout=300 \ | |
| test/test_skillform_frontmatter_parity.py \ | |
| | tee skillform-parity.log | |
| # 880 = 110 body shapes x (3 bare-| headers + 5 refused headers). Raise | |
| # this when the corpus grows; a collapse to a handful of cases (a broken | |
| # transpile that skips, a renamed symbol) fails here instead of passing | |
| # green. Anchored so "1880 passed" cannot satisfy "880 passed". | |
| grep -qE '(^|[^0-9])880 passed' skillform-parity.log || { | |
| echo "::error::the SkillForm frontmatter parity corpus did not report 880 passed (skipped, collapsed, or the corpus size changed without updating this canary)" | |
| exit 1 | |
| } | |
| - name: Run E2E and dedicated memory UI evidence in parallel | |
| # Smoke and UI read the staged production bundle; i18n builds a separate | |
| # dist-dev and a temporary base tree. All three share only read-only | |
| # dependency/browser installs, with separate homes, ports and outputs. | |
| # The helper awaits ALL outcomes even if one fails, preserves the UI's | |
| # 12-minute cap, and drains adopted gateway/browser children on cancel. | |
| # Overlap head+base i18n rendering rather than consume ~8 minutes of the | |
| # fixed 25-minute job budget before the ordinary E2E lane can start. | |
| # KIROCREW_E2E=1 is set by setup.py for smoke and only in the UI child | |
| # for evidence; REQUIRE turns missing prerequisites into hard failures. | |
| env: | |
| KIROCREW_E2E_REQUIRE: "1" | |
| VOICE_EVIDENCE_HEAD: ${{ github.event.pull_request.head.sha || github.sha }} | |
| KIROCREW_STRICT_ON_LOOP_PERSIST: "1" | |
| KIROCREW_MEMORY_UI_EVIDENCE_DIR: ${{ runner.temp }}/memory-embedding-evidence | |
| run: exec python scripts/ci_e2e_parallel.py | |
| - name: Upload read-aloud recovery evidence | |
| if: ${{ always() }} | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: voice-recovery-evidence | |
| path: website/test-results/**/voice-recovery/ | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| - name: Upload member memory UI evidence | |
| if: ${{ always() }} | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: member-memory-ui-evidence | |
| path: | | |
| website/test-results/**/member-memory-*.png | |
| website/test-results/**/legacy-member-memory-*.png | |
| website/test-results/**/member-memory-walkthrough.webm | |
| website/test-results/**/member-memory-evidence.json | |
| website/test-results/**/memory-v1-*.png | |
| website/test-results/**/memory-v2-*.png | |
| ${{ runner.temp }}/memory-embedding-evidence/**/memory-v2-embedding-*.png | |
| ${{ runner.temp }}/memory-embedding-evidence/**/memory-v2-embedding-evidence.json | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| # The specs' own failure record. Playwright writes one directory per | |
| # failed attempt under website/test-results/ -- the ARIA snapshot of the | |
| # page at the failing assertion (error-context.md), the on-first-retry | |
| # trace, and any screenshot -- and the html report next to it. Until this | |
| # step existed none of it left the runner: #8526 was triaged for hours from | |
| # a log line that named `error-context.md` and nothing else. Failure-only, | |
| # because the successful recovery screenshots are uploaded above and | |
| # the job already uploads coverage. `ignore` rather than `error`: a run | |
| # that failed before the specs started (a browser install that stalled) | |
| # has no results, and this step must not turn that into a second failure. | |
| - name: Upload Playwright failure artifacts | |
| if: failure() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: e2e-playwright-failures | |
| path: | | |
| website/test-results/ | |
| website/playwright-report/ | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| # The one E2E step the fleet cannot host. test_private_workflow_memory.py boots | |
| # a real gateway and runs a Crew Member's private workflow MCP inside the | |
| # member sandbox, which needs an unprivileged user namespace | |
| # (`unshare --mount --map-root-user`); the CodeBuild container answers EPERM. | |
| # Keeping it separate limits hosted runner use to its ~3.5-minute run. No SPA, | |
| # no Node, no browser: the test drives the gateway over HTTP, like the boot matrix. | |
| # | |
| # `--timeout=600` is a BACKSTOP, not the test's budget. The module caps all of a | |
| # test's waiting at `_TEST_BUDGET_SECONDS` (570) anchored at test start, so a | |
| # stalled workflow fails inside `_finished` -- which names the run and its | |
| # progress -- while this cap only catches a hang outside those waits. Two | |
| # gateway boots and nine workflow waits need more room than the ~3.5-minute | |
| # typical run, and a cap at that scale fires on runner slowness while saying | |
| # nothing about which workflow hung. Keep this above | |
| # `_TEST_BUDGET_SECONDS` and below `timeout-minutes`; | |
| # `test_e2e_private_namespace_budget.py` enforces both. | |
| e2e-private-namespace: | |
| name: E2E (private member namespace, hosted) | |
| needs: [await-fast-gate] | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 15 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - name: Enable private member namespace isolation | |
| run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 | |
| - name: Verify private member namespace isolation | |
| run: unshare --mount --map-root-user true | |
| - name: Require real private workflow MCP execution | |
| env: | |
| KIROCREW_E2E: "1" | |
| KIROCREW_E2E_REQUIRE: "1" | |
| KIROCREW_STRICT_ON_LOOP_PERSIST: "1" | |
| run: | | |
| echo "::remove-matcher owner=python::" | |
| python -m pytest -q -n0 --no-cov --timeout=600 test/e2e/test_private_workflow_memory.py | |
| # The integration layer: the real gateway booted IN the pytest process. | |
| # | |
| # `test/integration/` sits between the unit shards (bare `web.Application()` | |
| # shells, everything mocked) and the E2E jobs above (a `kirocrew gateway` | |
| # subprocess). Each test boots `GatewayOrchestrator.run()` on a scratch | |
| # `KIROCREW_HOME` with the packaged fake ACP backend, then talks HTTP to the | |
| # port it bound -- same interpreter, same event loop. That is what lets a test | |
| # here assert on a restart (second boot, same home), on an event-loop stall | |
| # seen from a second request, or on a policy file the boot itself wrote. | |
| # | |
| # Opt-in by `KIROCREW_INTEGRATION=1`; the unit shards collect and skip it. | |
| # Linux only: the fixture snapshots and restores the SIGINT/SIGTERM handlers | |
| # `run()` installs, which is a POSIX dance the Windows shards must not pay. | |
| # | |
| # The gate is ROUTE coverage, not line coverage: the conftest records which | |
| # registered dashboard routes the suite requested, and | |
| # `scripts/check_integration_route_coverage.py` reports the share. `--min` is | |
| # a ratchet -- raise it a little under what main measures, never above. | |
| integration: | |
| name: Integration (in-process gateway) | |
| needs: [changes, await-fast-gate] | |
| runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }} | |
| timeout-minutes: 20 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - uses: ./.github/actions/run-as-runner | |
| - name: Run the integration layer | |
| shell: /usr/local/bin/ci-shell {0} | |
| env: | |
| KIROCREW_INTEGRATION: "1" | |
| KIROCREW_INTEGRATION_HITS_DIR: ${{ runner.temp }}/integration-hits | |
| run: | | |
| echo "::remove-matcher owner=python::" | |
| python -m pytest -q -n 2 --dist loadgroup --max-worker-restart=2 \ | |
| --timeout=300 --no-cov test/integration | |
| - name: Route coverage | |
| shell: /usr/local/bin/ci-shell {0} | |
| env: | |
| HITS_DIR: ${{ runner.temp }}/integration-hits | |
| run: | | |
| python3 scripts/check_integration_route_coverage.py "$HITS_DIR" --missing --min 95 \ | |
| | tee "$HITS_DIR/route-coverage.txt" | |
| - name: Upload route-coverage report | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: integration-route-coverage | |
| path: ${{ runner.temp }}/integration-hits/ | |
| if-no-files-found: ignore | |
| retention-days: 7 | |
| # The only job that BOOTS A GATEWAY on macOS and Windows. | |
| # | |
| # Everything else on those two runners is unit tests. `e2e` above spawns a real | |
| # gateway, but it is ubuntu-latest only, because `KIROCREW_E2E` is set solely by | |
| # `setup.py test_e2e`. That was one of the two holes #8117 fell through: it | |
| # added a settings-file probe to `wrap_argv`'s Windows delegation branch, so on | |
| # a fresh Windows host -- where that file does not exist -- the Kiro ACP spawn | |
| # stopped delegating, took the no-backend fail-closed path, and the gateway | |
| # never became usable. The unit test that pinned that branch lives in | |
| # `test_sandbox_argv.py`, which `windows-collect-ignore.txt` excludes, and the | |
| # PR itself changed that test's mock to hardcode the one answer a fresh Windows | |
| # host cannot give. Neither hole is closable by another unit test; only a real | |
| # boot on the real platform is. | |
| # | |
| # Offline and credential-less by construction: the agent binary is the packaged | |
| # fake ACP backend, wired through `KIROCREW_KIRO_BIN`, so there is no model, no | |
| # token, no network and no cost. See docs/ci/e2e-gate.md. | |
| e2e-boot-matrix: | |
| name: E2E Gateway Boot (${{ matrix.os }}) | |
| needs: [changes, await-fast-gate] | |
| strategy: | |
| # Each OS is its own verdict. A Windows-only regression must not hide | |
| # behind a cancelled macOS leg, which is exactly the shape of the escape | |
| # this job exists for. | |
| fail-fast: false | |
| matrix: | |
| # macos-15 is on the push-to-main path only. It is the same queue story as | |
| # the moved pytest lane: the mac leg waited ~200 minutes for a runner on | |
| # every pull request, and it is the only leg that did. On main the wait | |
| # costs nobody a merge, and nightly's `pod-scenarios` boots a real | |
| # service-managed gateway on macos-15 as well, so the real-Darwin boot | |
| # remains covered twice -- just never in front of a pull request. | |
| os: ${{ fromJSON(github.event_name == 'pull_request' | |
| && '["ubuntu-latest","windows-latest"]' | |
| || '["ubuntu-latest","macos-15","windows-latest"]') }} | |
| # Keep matrix.os stable: names, readiness windows and artifact names use it. | |
| runs-on: ${{ matrix.os == 'ubuntu-latest' && (needs.changes.outputs.linux_runner_large || 'ubuntu-latest') || matrix.os == 'windows-latest' && (needs.changes.outputs.windows_runner || 'windows-latest') || matrix.os }} | |
| timeout-minutes: 20 | |
| defaults: | |
| run: | |
| shell: ${{ matrix.os == 'ubuntu-latest' && '/usr/local/bin/ci-shell {0}' || 'bash' }} | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - uses: ./.github/actions/setup-windows-tests | |
| if: runner.os == 'Windows' | |
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| enable-cache: true | |
| # Same no-uv.lock caveat as backend-test: key the cache on the files | |
| # that define the dependency set. | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| setup.cfg | |
| - name: Install dependencies | |
| shell: bash | |
| run: uv pip install --system -e ".[voice,desktop]" --group dev | |
| - uses: ./.github/actions/run-as-runner | |
| if: runner.os == 'Linux' | |
| - name: Verify Windows test prerequisites | |
| if: runner.os == 'Windows' && runner.environment == 'self-hosted' | |
| shell: bash | |
| run: python .github/actions/setup-windows-tests/probe.py | |
| - name: Boot a gateway end to end and assert the pass count | |
| # Linux uses the same non-root boundary as the backend shards. | |
| # Windows/macOS retain bash and the exact same test command. | |
| env: | |
| # Lifts the module skip. | |
| KIROCREW_E2E: "1" | |
| # Turns an unmet PRECONDITION (the packaged fake backend missing, the | |
| # testing package not importable) from a graceful skip into a hard | |
| # failure. Without it the job could report green having booted zero | |
| # gateways, because pytest counts a skip as a pass. Same mechanism as | |
| # KIROCREW_E2E_REQUIRE on the `e2e` job. | |
| KIROCREW_E2E_REQUIRE: "1" | |
| # Readiness window per OS, set here rather than in the test so a slow | |
| # runner is retunable without a code change. Windows gets the widest: | |
| # subprocess spawn and filesystem latency there are measurably slower, | |
| # the conditions #9172 addressed when a slow disk killed the gateway. | |
| KIROCREW_HARNESS_READY_TIMEOUT: ${{ matrix.os == 'windows-latest' && '180' || matrix.os == 'macos-15' && '90' || '60' }} | |
| run: | | |
| set -o pipefail | |
| # Same as backend-test (#7296): the `python` matcher annotates any | |
| # traceback in the log, including one printed inside a warning, and | |
| # never matches a pytest failure. | |
| echo "::remove-matcher owner=python::" | |
| # -n0: this module spawns a real gateway per test, and under xdist a | |
| # block would take the worker down with it, which on Windows aborts the | |
| # whole run. --timeout is above the widest readiness window (180s) plus | |
| # the per-turn reply ceiling, so a stuck turn fails by name instead of | |
| # killing the run. | |
| pytest -v -n0 --no-cov -p no:cacheprovider --timeout=420 \ | |
| test/e2e/test_gateway_boot_matrix.py \ | |
| | tee boot-matrix.log | |
| # The canary. `pytest` exits 0 on a fully skipped module, so the exit | |
| # code alone cannot tell "seven gateways booted" from "the module was | |
| # never collected". Grepping the exact count is what makes this job | |
| # honest; raise the number when you add a test here. | |
| # Anchored: see the pod-boot canary above. | |
| grep -qE '(^|[^0-9])7 passed' boot-matrix.log || { | |
| echo "::error::gateway boot matrix did not report 7 passed (skipped, or a test was added without updating this canary)" | |
| exit 1 | |
| } | |
| - name: Upload the boot-matrix log | |
| if: failure() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: e2e-boot-matrix-${{ matrix.os }} | |
| path: boot-matrix.log | |
| if-no-files-found: ignore | |
| retention-days: 7 |