Skip to content

test: pin the clock in three tests that raced the real one on Windows #74129

test: pin the clock in three tests that raced the real one on Windows

test: pin the clock in three tests that raced the real one on Windows #74129

Workflow file for this run

name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
# PRs cancel superseded revisions: only a PR's tip commit needs a verdict.
# main does not, so let an in-flight run finish instead of killing it.
# This does NOT produce a verdict per commit. The group still permits one
# pending run, and GitHub evicts that pending run when a newer one queues
# (a documented GitHub concurrency behavior). With a ~19min run and a
# ~1.4min median merge gap, main gets periodic verdicts instead of none.
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
# There is deliberately no BACKEND_DESELECTS here any more. Eleven files used to be
# deselected from EVERY backend pytest invocation because they drive a real
# `git`/`gh`/`pytest` through `sandboxed_spawn_argv`, and a GH runner denies
# `unshare(CLONE_NEWNS)` (EPERM) so `wrap_argv` fails closed. That kept 608 tests --
# including the suites guarding the ops autonomy gate -- off every pull request.
#
# They no longer need it: each one that requires a sandbox carries its own
# `skipif(not userns_available())`, so on a runner it skips with a named reason and on
# a capable host it runs for real. MEASURED against a faithful local simulation of the
# runner (`unshare(CLONE_NEWUSER)` ok, `CLONE_NEWNS` EPERM, no user session bus for the
# cgroup scope, no gh token): 608 passed, 85 skipped, 0 failed.
#
# A whole-file deselect is the wrong tool for a missing host capability, and this is
# why: it is invisible in the run output, it takes the file's OTHER tests with it, and
# nothing goes red when the reason expires. A capability guard says which capability,
# on which test, in the report. If a suite here starts needing one again, add the guard
# -- not a deselect.
jobs:
# ── Surface detection (issue #1556) ──────────────────────────────────
# Most tests only exercise their own surface, so a single-surface diff can
# skip the opposite suite's *provably* single-surface files. The selector
# (scripts/ci-surface-tests.py) is deny-by-default: every cross-surface
# parity guard -- and anything it cannot classify -- stays in the must-run
# set, so a heuristic miss costs CI time, never a skipped guard.
#
# only_backend / only_frontend are computed here ONCE. `backend` is a
# catch-all, so every changed file is exactly one of frontend / meta /
# backend: a diff that touches two buckets (or meta) leaves BOTH flags false
# and the full matrix runs. One exception: temp-screenshots/** evidence
# media counts toward NO bucket, so a screenshots-only diff also leaves both
# flags false and runs the full matrix (fails safe).
changes:
name: Detect changed surface
runs-on: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) && format('codebuild-kirocrew-gha-linux-{0}-{1}', github.run_id, github.run_attempt) || 'ubuntu-latest' }}
# 15, not 5: the job's own work is seconds, but its `fetch-depth: 0`
# checkout (paths-filter and the leaf resolver diff against the merge base
# with local git) is a full-history fetch of a large repository, and on a
# slow hosted runner that fetch alone has run past 5 minutes and had the
# job CANCELLED mid-checkout. Every test lane `needs` this job, so that one
# cancel skipped the whole matrix and failed Coverage Gate with nothing to
# read (#13111). The budget covers a slow fetch; a hung one still ends.
timeout-minutes: 15
# dorny/paths-filter runs with `token: ''` (below), so it detects changes
# with local git instead of the REST API and needs no pull-requests:read.
# Job-level permissions REPLACE the top-level grant, so contents:read is
# restated.
permissions:
contents: read
outputs:
only_backend: ${{ steps.decide.outputs.only_backend }}
only_frontend: ${{ steps.decide.outputs.only_frontend }}
linux_packaging: ${{ steps.filter.outputs.linux_packaging }}
leaf_tests: ${{ steps.leaf.outputs.leaf_tests }}
leaf_targets: ${{ steps.leaf.outputs.leaf_targets }}
leaf_gates: ${{ steps.leaf.outputs.leaf_gates }}
# Central runner labels: eligible repository/event/actor combinations
# use CodeBuild; all other runs use the corresponding hosted OS.
# Linux consumers retain a fallback when this job never reports:
# `${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}`.
# if this job fails or is cancelled the output is empty, and a job that
# runs anyway (`coverage-gate` fails closed with `if: always()`) must still
# get a runner rather than error out before its own checks execute.
linux_runner: ${{ steps.runner.outputs.linux_runner }}
linux_runner_large: ${{ steps.runner.outputs.linux_runner_large }}
windows_runner: ${{ steps.windows-runner.outputs.windows_runner }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# The leaf-test selector diffs against the PR base, so the base commit
# must be present. Paid once here rather than on every shard job.
fetch-depth: 0
persist-credentials: false
- uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
id: filter
with:
# An empty token switches the action to LOCAL git change detection on
# pull_request events (documented on the `token` input; the default is
# the shared installation token). The pinned v4.0.3 build then runs
# getChanges(pull_request.base.sha, HEAD): a plain two-dot
# `git diff` -- the checkout above already has the base commit
# (fetch-depth: 0), so no fetch and no API call happen. This matters
# because the REST list-files call spends the per-hour installation
# quota shared by EVERY workflow in this repo, and at dozens of
# ci.yml runs per hour it exhausted the quota and failed this job
# repo-wide (#10434). With an empty token the `base` input stays
# empty on purpose: the action falls back to the PR base SHA from
# the event payload. Both diff endpoints are frozen when the run is
# created (base.sha from the payload, HEAD at the event's merge
# commit), so a base-branch move after that changes nothing. The
# result is the NET change the merge introduces relative to base.sha:
# a PR-listed file whose identical change already landed in base
# drops out, which is correct for surface detection -- that surface
# is unchanged relative to base -- and any wider change set still
# pushes toward the full matrix. push events never consult `token`,
# so main builds are unchanged.
token: ''
# 'some-with-excludes': a file matches a filter when it matches at
# least one pattern AND none of the negated ones. That lets `backend`
# be a true CATCH-ALL ("every changed file that is not frontend, not
# meta, and not ignored evidence media") instead of an allowlist that
# would have to be complete to be safe. Every changed file therefore
# lands in exactly one bucket -- or none, for temp-screenshots/**
# evidence -- which is what keeps "only_X means only X changed" true
# for real files: an unrecognised path (docs/**, *.md, a new root
# file) counts as backend rather than silently riding along under a
# narrowed suite.
# NOTE: this quantifier requires >= v4.0.3 (added upstream in PR #322);
# v4.0.2 rejects the input and throws, so do not lower the pin.
predicate-quantifier: 'some-with-excludes'
filters: |
frontend:
- 'website/**'
meta:
- '.github/**'
- 'scripts/**'
backend:
- '**'
- '!website/**'
- '!.github/**'
- '!scripts/**'
# temp-screenshots/** is never-packaged, never-imported evidence
# media (gitignored; PR evidence travels as a GitHub attachment),
# so it must not drag a frontend-only PR into the
# full backend matrix (#8027). Excluding it keeps the catch-all
# safety property: any unrecognised REAL file still lands in
# backend, a screenshots+frontend diff stays only_frontend=true,
# and a screenshots-ONLY diff matches no bucket so both flags
# stay false and the FULL matrix runs (fails safe).
- '!temp-screenshots/**'
# Independent of the three buckets above: it answers a different
# question ("could this diff change what a Linux package installs?")
# and so may overlap them freely.
linux_packaging:
- 'website/electron/package.json'
- 'website/electron/auto-update.js'
- 'website/electron/bundle-location.js'
- 'packaging/build-desktop.sh'
- 'scripts/stamp-distribution.sh'
- '.github/workflows/build-desktop.yml'
- name: Decide surface
id: decide
env:
FRONTEND: ${{ steps.filter.outputs.frontend }}
BACKEND: ${{ steps.filter.outputs.backend }}
META: ${{ steps.filter.outputs.meta }}
# Escape hatch: the ci-full-run label forces the full matrix. NOTE:
# this workflow does not listen for the `labeled` event (that would
# re-run everything on every bot label), so the label takes effect on
# the NEXT push, not the instant it is applied.
FULL_RUN: ${{ contains(github.event.pull_request.labels.*.name, 'ci-full-run') }}
run: |
# A surface is "only" when it changed and NO other bucket did.
# `backend` is a catch-all, so an unrecognised path counts as backend
# and cannot ride along under a narrowed suite.
only_backend=false; only_frontend=false
if [ "$FULL_RUN" != "true" ] && [ "$META" != "true" ]; then
if [ "$BACKEND" = "true" ] && [ "$FRONTEND" != "true" ]; then only_backend=true; fi
if [ "$FRONTEND" = "true" ] && [ "$BACKEND" != "true" ]; then only_frontend=true; fi
fi
echo "only_backend=$only_backend" >> "$GITHUB_OUTPUT"
echo "only_frontend=$only_frontend" >> "$GITHUB_OUTPUT"
echo "backend=$BACKEND frontend=$FRONTEND meta=$META full_run=$FULL_RUN -> only_backend=$only_backend only_frontend=$only_frontend"
# A diff that touches ONLY leaf test modules cannot change a production code
# path, and cannot change another test file's outcome either -- the selector
# escalates on conftest, shared helpers, fixture trees, and any test module
# some other file imports. So the sharded suite is replaced by the changed
# files plus the gates that read the test/ tree as data, run several times.
#
# This exists because a one-line flaky-test fix used to wait ~30 minutes for
# the full matrix, which is what makes a flake block every open PR. Repeats
# matter more than breadth here: a flaky test run once in isolation passes.
#
# Every failure path below yields the FULL suite. `ci-full-run` still forces
# the whole matrix when a fix depends on suite-wide interleaving.
- name: Decide leaf-test scope
id: leaf
env:
FULL_RUN: ${{ contains(github.event.pull_request.labels.*.name, 'ci-full-run') }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -uo pipefail
leaf_tests=false
# BOTH must be pre-initialized: the heredoc below expands each one
# unconditionally, while they are assigned only in the `else` branch. A
# push build (no PR base) and a `ci-full-run` PR both take the `if`
# branch, so an uninitialized variable there dies on `set -u` and takes
# the whole `changes` job -- and every job that needs it -- with it,
# breaking the very full-matrix escape hatch this branch exists to serve.
targets=""
gates=""
if [ "$FULL_RUN" = "true" ] || [ -z "${BASE_SHA:-}" ]; then
echo "Full suite: ci-full-run label set, or no PR base (push build)."
else
rc=0
targets="$(SCOPED_TESTS_BASE_REF="$BASE_SHA" \
python3 scripts/leaf_test_scope.py --targets)" || rc=$?
gates="$(SCOPED_TESTS_BASE_REF="$BASE_SHA" \
python3 scripts/leaf_test_scope.py --gates)" || true
if [ "$rc" = "0" ] && [ -n "$targets" ]; then
leaf_tests=true
echo "Leaf-test diff: $(printf '%s\n' "$targets" | grep -c .) target file(s)."
printf '%s\n' "$targets" | sed 's/^/ - /'
else
echo "Full suite: not a leaf-test-only diff (selector rc=$rc)."
gates=""
fi
fi
echo "leaf_tests=$leaf_tests" >> "$GITHUB_OUTPUT"
{
echo "leaf_targets<<LEAF_EOF"
printf '%s\n' "$targets"
echo "LEAF_EOF"
echo "leaf_gates<<GATE_EOF"
printf '%s\n' "$gates"
echo "GATE_EOF"
} >> "$GITHUB_OUTPUT"
# Same-repository push/PR AND an actor admitted by the fleet webhook.
# Missing CODEBUILD_ACTOR_IDS fails toward hosted, never a queued fleet
# job the webhook cannot accept. The webhook remains an independent fence.
- name: Pick the Linux runner label
id: runner
env:
ELIGIBLE: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) }}
LABEL: codebuild-kirocrew-gha-linux-${{ github.run_id }}-${{ github.run_attempt }}
run: |
if [ "$ELIGIBLE" = "true" ]; then
echo "linux_runner=$LABEL" >> "$GITHUB_OUTPUT"
echo "linux_runner_large=$LABEL instance-size:large" >> "$GITHUB_OUTPUT"
else
echo "linux_runner=ubuntu-latest" >> "$GITHUB_OUTPUT"
echo "linux_runner_large=ubuntu-latest" >> "$GITHUB_OUTPUT"
fi
- name: Pick the Windows runner label
id: windows-runner
env:
ELIGIBLE: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) }}
LABEL: codebuild-kirocrew-gha-windows-${{ github.run_id }}-${{ github.run_attempt }}
run: |
if [ "$ELIGIBLE" = "true" ]; then
echo "windows_runner=$LABEL instance-size:large" >> "$GITHUB_OUTPUT"
else
echo "windows_runner=windows-latest" >> "$GITHUB_OUTPUT"
fi
# Build the Linux desktop packages and install them for real, but ONLY when the
# diff could change what a package installs. The check is expensive (a PBS
# interpreter, a full pip closure, then electron-builder three times), and it
# verifies things no cheaper gate can: dependency names that must exist in the
# target distro's repositories, the generated .desktop entry, and the maintainer
# scripts. Same trade the macOS-packaging path gate already makes.
#
# A packaging change that lands without this having run is the failure mode this
# exists to prevent: a package that builds green and refuses to install.
linux-packaging:
name: Linux Packaging (build + smoke-install)
needs: [changes, await-fast-gate]
if: ${{ needs.changes.outputs.linux_packaging == 'true' }}
runs-on: ubuntu-22.04
timeout-minutes: 45
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
# Builds all three Linux formats from one backend tree, re-stamping the
# beacon distribution between electron-builder invocations.
- name: Build the Linux desktop packages
env:
CSC_IDENTITY_AUTO_DISCOVERY: "false"
run: bash packaging/build-desktop.sh
- name: Install the packages in their target distros
run: ./scripts/smoke-linux-packages.sh website/electron/dist
# ── Fast gate barrier (see .github/workflows/fast-gate.yml) ──────────
# The eleven cheap gates live in their own workflow so the fork reviewers can
# key on them, which means ci.yml cannot reach them with `needs:` -- a needs
# edge cannot cross a workflow file. This job is that edge: it reads the Fast
# Gate run for this exact commit and fails closed, and every heavy job needs
# it. One extra ~1-minute job buys the whole matrix the right to not start.
#
# Fails closed on purpose, in all three ways it can go wrong: a Fast Gate run
# that never appears, one that never completes inside the budget, and one whose
# conclusion is a decided non-success all exit non-zero. `action_required` is
# not one of them: GitHub reports a fork run awaiting maintainer approval that
# way, so it is polled as pending. A barrier that passed when it could
# not read its subject would be worse than no barrier, because the matrix would
# run anyway and the log would claim it was cleared to.
await-fast-gate:
name: Await Fast Gate
runs-on: ${{ github.repository == 'kirodotdev/KiroCrew' && contains(fromJSON(vars.CODEBUILD_ACTOR_IDS || '[]'), github.actor_id) && (github.event_name == 'push' || (github.event_name == 'pull_request' && (github.event.action == 'opened' || github.event.action == 'synchronize') && github.event.pull_request.head.repo.full_name == github.repository)) && format('codebuild-kirocrew-gha-linux-{0}-{1}', github.run_id, github.run_attempt) || 'ubuntu-latest' }}
timeout-minutes: 15
permissions:
# actions:read is what lets this read another workflow's runs. Job-level
# permissions REPLACE the top-level grant, so contents:read is restated.
actions: read
contents: read
steps:
- name: Wait for the Fast Gate verdict on this commit
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
# The head sha, not github.sha: on a pull_request event github.sha is
# the ephemeral merge commit, and Fast Gate's own run is keyed to the
# same head the PR reports. Matching on `event` too keeps a push run on
# the same commit from answering for a pull_request run.
SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
EVENT: ${{ github.event_name }}
# A head SHA is NOT a unique key for a run. Two pull requests can carry
# the same head commit -- a shared branch, a cherry-pick, a fork whose
# branch was pushed to a second PR -- and each one gets its own Fast Gate
# run on that commit. Keyed on the SHA alone, this job could read the
# OTHER PR's verdict and release its own matrix on a gate that never ran
# against this PR's base. The head branch plus the head repository is
# what makes the triple unique, so both are matched below and then
# re-verified on the run that is selected.
BRANCH: ${{ github.event_name == 'pull_request' && github.head_ref || github.ref_name }}
HEAD_REPO: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name || github.repository }}
run: |
set -euo pipefail
# Both workflows start from the same webhook, so Fast Gate's run may not
# be visible for the first few seconds. APPEAR_BUDGET covers that race;
# after it, a missing run is a real misconfiguration and fails.
APPEAR_BUDGET=180
TOTAL_BUDGET=720
started="$(date +%s)"
seen=false
while :; do
elapsed=$(( $(date +%s) - started ))
runs="$(gh api --method GET \
"repos/$REPO/actions/workflows/fast-gate.yml/runs" \
-f "head_sha=$SHA" -f "event=$EVENT" -f "branch=$BRANCH" \
-f per_page=100 2>/dev/null || true)"
# `branch=` is passed to the API as a narrowing hint, but the match is
# re-asserted here: a filter the server silently ignores would hand
# back another PR's run, and selecting max_by(.id) out of an unfiltered
# list would pick the NEWEST such run -- the most likely one to be a
# different PR's. Filter first, collapse second.
latest="$(printf '%s' "$runs" \
| jq -c --arg branch "$BRANCH" --arg repo "$HEAD_REPO" \
'[(.workflow_runs // [])[]
| select(.head_branch == $branch)
| select(.head_repository.full_name == $repo)]
| if length == 0 then null else max_by(.id) end' \
2>/dev/null || echo null)"
if [ "$latest" = "null" ] || [ -z "$latest" ]; then
if [ "$elapsed" -ge "$APPEAR_BUDGET" ]; then
echo "::error::No Fast Gate run found for $SHA on $HEAD_REPO@$BRANCH" \
"($EVENT) after ${elapsed}s. The gates cannot be confirmed, so the" \
"matrix is not cleared to run."
exit 1
fi
sleep 10
continue
fi
status="$(printf '%s' "$latest" | jq -r '.status')"
conclusion="$(printf '%s' "$latest" | jq -r '.conclusion // ""')"
url="$(printf '%s' "$latest" | jq -r '.html_url')"
if [ "$seen" = false ]; then
echo "Fast Gate run: $url"
seen=true
fi
if [ "$status" = "completed" ]; then
case "$conclusion" in
success)
echo "Fast Gate passed. Releasing the heavy jobs."
exit 0
;;
action_required)
# A fork run still awaiting maintainer approval is reported as
# completed/action_required. That is pending, not decided, so keep
# polling under TOTAL_BUDGET -- otherwise which of the two pending
# runs a maintainer approves first decides whether the matrix runs.
status="awaiting maintainer approval"
;;
*)
echo "::error::Fast Gate concluded '$conclusion' ($url)." \
"Fix the gate it reports; the heavy jobs are skipped on purpose."
exit 1
;;
esac
fi
if [ "$elapsed" -ge "$TOTAL_BUDGET" ]; then
echo "::error::Fast Gate still '$status' after ${elapsed}s ($url)." \
"Failing closed rather than starting the matrix unverified."
exit 1
fi
sleep 10
done
backend-lint:
name: Backend Lint & Type Check
needs: [changes, await-fast-gate]
# black/flake8/isort/mypy read no file under website/, so a frontend-only
# diff cannot change this verdict. Gated on only_frontend (not on a
# positive "python changed" allowlist) so that anything the selector could
# not classify as frontend -- docs, a new root file, meta -- still runs it.
if: needs.changes.outputs.only_frontend != 'true'
# Large alone did not fix black's process-pool failure. Bound its workers
# below as well, on hosted and fleet alike; preserve every lint/type gate.
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
# The job's floor on today's hosted runners is ~14-15 min (black --check
# over the whole tree alone takes 7-8 min, flake8 ~2 min, mypy cold ~3 min),
# so 15 cancelled mypy mid-run on green PRs. 25 is the budget frontend-test
# and bundle-size already run on, and leaves headroom for runner variance.
timeout-minutes: 25
strategy:
matrix:
python-version: ["3.12"]
steps:
# Full history so the black gate can compute THIS change's own diff. A
# shallow clone truncates parent info and leaves no base ref, and the gate
# then falls back to judging the whole tree -- which reddens the PR for
# files the base branch merged after the baseline was taken. depth 2 was
# tried first and was not enough; the gate now prints which scope it used,
# so a regression here is visible in the log instead of silent.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 0
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# This repo has no uv.lock (pip/pyproject-managed), so point the
# cache key at the files that define the dependency set. Without
# this, setup-uv defaults to globbing **/uv.lock, finds nothing,
# and errors: "No file matched to [**/uv.lock]".
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
# uv resolves/installs the heavy deps far faster than pip and caches
# across runs. --system targets the setup-python interpreter (no venv);
# --group pulls [dependency-groups].dev from pyproject.
run: uv pip install --system -e ".[voice]" --group dev
# Python vulnerability visibility. The repository had none: the step this
# replaces ran an unpinned `pip install pip-audit`, and pip-audit is now
# pinned in the dev group installed immediately above, so the audit needs no
# install of its own and two runs a week apart audit with the same tool.
#
# continue-on-error: this REPORTS on day one. Widening dependency auditing
# and making it block in the same change is how a gate ends up switched off
# by whoever it first stopped, and pip-audit reaches the network, so a
# registry blip would otherwise fail unrelated PRs. What makes this
# different from the advisory step it replaces is the baseline: the script
# compares the count against python-audit-baseline.json and says so when it
# rises, which turns a standing wall of known advisories into a number that
# only moves when someone moves it. Dropping continue-on-error is the
# follow-up once a count is recorded and has held steady.
- name: Python dependency audit (pinned, report-only)
continue-on-error: true
run: python3 scripts/check_python_audit.py
# black IS enforced, behind a shrinking baseline. A bare `black --check`
# cannot be turned on here: 1,420 files under src/ and test/ are not
# black-clean, so the bulk pass it waits on is ~95,800 lines and has been
# "pending" long enough for the old estimate in this comment to drift by 3x.
# Leaving it off meanwhile was the worse failure: AGENTS.md tells every
# contributor to run `black src/kiro_crew test` before committing, and
# following that instruction reformats those 1,420 files on top of their
# change. The baseline makes the gate real for everything else, so new code
# is formatted without any PR being forced to carry unrelated churn.
- name: Check formatting (black, baselined)
# Every runner recycles black workers: one file per child, at most eight,
# capped at 2 GiB each on Linux. Hosted `ubuntu-latest` needs this too:
# the compiled wheel retains memory per worker and the VM has no cgroup
# cap, so a native pool takes the whole 16 GB runner down mid-step (exit
# 143). Recycling costs ~1.6x native wall time on four cores.
# Keep cgroup evidence: per-process limits do not bound the whole job.
run: python3 scripts/ci_black_diagnostics.py scripts/check_black_formatting.py
# Text-mode subprocess calls must pin their decode encoding: without one
# the child's output decodes with the Windows ANSI code page (#5249).
# Self-test first: it plants one probe per rule family, so a typo that
# silently disables a rule fails here instead of shipping green.
- name: Check subprocess text-mode encoding (gate, baselined)
run: |
python3 scripts/check_subprocess_encoding.py --test
python3 scripts/check_subprocess_encoding.py
# A committed ACP frame fixture carries what the WIRE carried. The class that
# gets through a credential scrub is an ENUMERATION -- a provider catalog, a
# command inventory -- because it reads as product data while describing what
# the recording host HAS. Today's leaks are carried in the baseline as
# pre-existing and it is shrink-only, per file AND per marker class, so a new
# fixture cannot inherit an exemption.
- name: Check ACP frame fixtures for host data (gate, baselined)
run: python3 scripts/check_acp_frame_host_data.py
# Application code must reach the agent backend through kiro_crew.agent_sdk,
# not by importing kiro_crew.acp (or kiro_crew.providers, which re-exports
# the same ACP shapes and would otherwise be a laundering channel). Today's
# edges are carried in the baseline as pre-existing and it is shrink-only. Design
# record: docs/request-for-change/rfc-crew-agent-sdk-boundary.md.
# Self-test first, same reason as the gate above.
- name: Check agent-SDK import boundary (gate, baselined)
run: |
python3 scripts/check_agent_sdk_boundary.py --test
python3 scripts/check_agent_sdk_boundary.py
# Blocking IO inside an `async def` stalls every session's turn, and a
# stall past dashboard.loop_stall_exit_after_secs (25s) makes the watchdog
# kill the gateway — the user sees a restart for no apparent reason and
# loses every in-flight turn (#1572). #3057's audit found the sites
# cluster where nothing was watching: ~70 on-loop store.db.execute() calls
# in dashboard/handlers/knowledge.py against zero in handlers/memory.py
# next door. Baselined for the same reason as the gates above — the
# known set is recorded as legacy so it can only shrink, while a NEW call
# (or one on a line this change adds) fails here.
- name: Check for blocking IO inside async def (gate, baselined)
run: |
# Self-test first: it plants one probe per rule family, so a rule that
# silently stops matching fails here instead of shipping green.
python3 scripts/check_sync_io_in_async.py --test
python3 scripts/check_sync_io_in_async.py
# A secret-bearing file must be locked down BEFORE any content reaches it:
# a post-publish lockdown leaves the payload at its final path under the
# inherited DACL, which on Windows is not covered by atomic_write's POSIX
# mode at all. #5307 converted the last seven such writers; this keeps a
# new one from reintroducing the shape. Known-unconverted sites live in
# KNOWN_UNCONVERTED in the script, keyed to an open issue, and that list is
# shrink-only -- a paid-off entry fails the gate until it is deleted.
- name: Check lockdown-before-publish (secret writers)
run: python3 scripts/check_lockdown_before_publish.py
# isort/flake8/mypy below are BLOCKING (config in pyproject.toml + setup.cfg).
#
# The two repo-root modules are named explicitly: `src/kiro_crew test` misses
# them, and between them they hold the host-protection floor for all three
# testpaths -- the isolation fixtures and the xdist worker budget -- so they
# are the last files that should go unlinted.
- name: Check imports (isort)
run: isort --check-only src/kiro_crew test conftest.py xdist_budget.py
- name: Lint (flake8)
run: flake8 src/kiro_crew test conftest.py xdist_budget.py
- name: Type check (mypy)
run: mypy src/kiro_crew/
backend-test:
name: Backend Tests
needs: [changes, await-fast-gate]
# All eight shards use large compute and the measured non-root boundary.
# Forks, non-admitted actors and absent resolver output stay hosted.
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
# Keep the existing wall-clock budget; individual tests still time out at
# 120 seconds. File sharding reduces repeated collection, not test coverage.
timeout-minutes: 60
strategy:
# Don't cancel sibling shards when one fails -- we want the full failure
# picture across the suite, not just the first shard to break.
fail-fast: false
matrix:
python-version: ["3.12"]
# Eight runners reduce per-shard work at the cost of more runner slots.
# Whole files are assigned before import; -n auto parallelizes within
# each shard. To change the count, update this list and SHARD_COUNT.
group: [1, 2, 3, 4, 5, 6, 7, 8]
env:
SHARD_COUNT: 8
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# This repo has no uv.lock (pip/pyproject-managed), so point the
# cache key at the files that define the dependency set. Without
# this, setup-uv defaults to globbing **/uv.lock, finds nothing,
# and errors: "No file matched to [**/uv.lock]".
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
# Setup/install stays under the runner identity. Select test scope also
# stays there because it writes the runner-owned GITHUB_OUTPUT file.
- uses: ./.github/actions/run-as-runner
- name: Select test scope
id: scope
env:
ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }}
LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }}
LEAF_TARGETS: ${{ needs.changes.outputs.leaf_targets }}
TARGETS_FILE: ${{ runner.temp }}/backend-targets.txt
# A frontend-only diff cannot affect a backend test that never touches
# the frontend surface, so we run only the files the selector could NOT
# prove single-surface (every parity guard, plus anything unclassified).
# Every failure path here falls back to the FULL suite on purpose --
# never trade a missed test for a faster run.
run: |
# Leaf-test verdict wins: it is the narrower and more specific reduction,
# and it is computed once in the `changes` job. `reduced=true` rides along
# so every existing coverage guard keyed on it keeps working unchanged --
# a subset's coverage is not comparable to the repo floor either way.
if [ "$LEAF_TESTS" = "true" ] && [ -n "${LEAF_TARGETS//[[:space:]]/}" ]; then
printf '%s\n' "$LEAF_TARGETS" | grep . > "$TARGETS_FILE" || true
if [ -s "$TARGETS_FILE" ]; then
echo "reduced=true" >> "$GITHUB_OUTPUT"
echo "leaf=true" >> "$GITHUB_OUTPUT"
echo "Leaf-test scope: $(wc -l < "$TARGETS_FILE") file(s), repeated."
exit 0
fi
echo "::warning::leaf targets arrived empty -- running the FULL backend suite."
fi
if [ "$ONLY_FRONTEND" != "true" ]; then
echo "reduced=false" >> "$GITHUB_OUTPUT"
echo "Full backend suite (diff is not frontend-only)."
exit 0
fi
rc=0
python3 scripts/ci-surface-tests.py --surface backend --summary > "$TARGETS_FILE" || rc=$?
if [ "$rc" != "0" ] || [ ! -s "$TARGETS_FILE" ]; then
echo "::warning::surface selector unusable (rc=$rc) -- running the FULL backend suite."
echo "reduced=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "reduced=true" >> "$GITHUB_OUTPUT"
echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface backend file(s)."
- name: Run tests (shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
shell: /usr/local/bin/ci-shell {0}
# scripts.ci_file_shards assigns each file to one shard before pytest
# imports it. Discovery and platform ignores remain pytest's own; files
# are never promoted to explicit arguments just to partition the suite.
# Hash assignment is stable, not duration-balanced. Every file's tests
# and parameters stay together; xdist still uses loadgroup within a job.
#
# Coverage runs on every shard: since the floor moved to >= 3.12 the
# matrix has a single Python, so there is no --no-cov lane left. Each
# shard emits a partial data file merged later by the coverage-combine
# job. Keep the package-name boundary: a whole source-directory selector
# also measures vendored and standalone scripts outside that boundary.
# The two explicit directories contain already-measured source whose
# synthetic/app-local module names otherwise make tracing import-order dependent.
env:
PYTHONPATH: ${{ github.workspace }}
REDUCED: ${{ steps.scope.outputs.reduced }}
LEAF: ${{ steps.scope.outputs.leaf }}
LEAF_REPEAT: "3"
LEAF_GATES: ${{ needs.changes.outputs.leaf_gates }}
TARGETS_FILE: ${{ runner.temp }}/backend-targets.txt
run: |
# Reuse payload-free progress on full, reduced and repeated leaf runs.
# A worker crash stays red; do not replace it and lose its failure report.
PROGRESS=(-p scripts.ci_pytest_progress --ci-progress-dir "$RUNNER_TEMP/pytest-progress" --max-worker-restart=0)
# Take the `python` problem matcher registered by actions/setup-python
# out of scope before pytest writes a line. Its pattern is a traceback
# frame followed by `raise SomeError('msg')`, matched anywhere in the
# log -- including inside pytest's warnings summary. MEASURED on the six
# jobs in #7296: it matched a warning traceback every time and a pytest
# failure not once, so every red was annotated `Event loop is closed`
# at asyncio/base_events.py's line 545 while the tests that actually
# failed were named in no annotation at all, and four unrelated PRs
# were misdiagnosed as an event-loop teardown flake. conftest.py's
# `pytest_terminal_summary` now emits the annotations from pytest's own
# report objects; this keeps a louder wrong answer off the check run.
# Not `add-problem-matchers: false`: that input does not exist on this
# action pin, and passing it only adds an "Unexpected input(s)" warning.
echo "::remove-matcher owner=python::"
# Reduced scope (frontend-only diff): run just the cross-surface
# files, still sharded, with coverage off -- a subset's coverage is
# not comparable to the repo floor, so Coverage Combine/Gate skip the
# backend lane for this run (see coverage-gate).
TARGETS=()
if [ "$REDUCED" = "true" ]; then
while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then TARGETS+=("$line"); fi; done < "$TARGETS_FILE"
fi
# Leaf-test scope: run the whole narrow set on every matrix job, NOT
# split across shards -- 14 files do not shard usefully, and the point
# is repeats. Each shard job is then an independent roll on its own
# runner, which is what a flake fix actually needs. Measured
# on this checkout: one pass over the narrow set is ~38s (910 tests)
# against ~29 minutes for the slowest full shard.
if [ "$LEAF" = "true" ]; then
# The corpus gates run ONCE: they are deterministic content assertions,
# so repeating them buys nothing and -- at ~3 minutes a pass over ~158
# files -- would dominate the run and undo the reduction. Only the
# CHANGED files repeat, because that is where a flake lives.
GATES=()
while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then GATES+=("$line"); fi; done <<< "$LEAF_GATES"
CHANGED=()
for target in ${TARGETS[@]+"${TARGETS[@]}"}; do
keep=1
for gate in ${GATES[@]+"${GATES[@]}"}; do
if [ "$target" = "$gate" ]; then keep=0; break; fi
done
if [ "$keep" = "1" ]; then CHANGED+=("$target"); fi
done
if [ "${#CHANGED[@]}" -eq 0 ]; then
echo "::error::leaf run resolved no changed files -- refusing to report green."
exit 1
fi
if [ "${#GATES[@]}" -gt 0 ]; then
echo "::group::leaf corpus gates (${#GATES[@]} file(s), once)"
pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -q -n auto --timeout=120 --no-cov -m "not ipv6_required" ${GATES[@]+"${GATES[@]}"}
echo "::endgroup::"
fi
for pass in $(seq 1 "$LEAF_REPEAT"); do
echo "::group::leaf changed files, pass $pass/$LEAF_REPEAT"
pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -q -n auto --timeout=120 --no-cov -m "not ipv6_required" ${CHANGED[@]+"${CHANGED[@]}"}
echo "::endgroup::"
done
exit 0
fi
if [ "$REDUCED" = "true" ]; then
python -m pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -p scripts.ci_file_shards -q -n auto --timeout=120 -m "not ipv6_required" \
--file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \
--no-cov \
${TARGETS[@]+"${TARGETS[@]}"}
exit 0
fi
python -m pytest ${PROGRESS[@]+"${PROGRESS[@]}"} -p scripts.ci_file_shards -q -n auto --timeout=120 -m "not ipv6_required" \
--file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }} \
--cov=kiro_crew --cov=sage_lib \
--cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging \
--cov=src/kiro_crew/apps/builtins/code_review_sage/tests
# The shard union plus backend-test-ipv6 retains the full test population.
# The eleven sandbox files -- the two sandbox suites, the two ops git/gh
# suites, and the seven auto_improvement suites --
# each carry `skipif(not userns_available())` on the tests that need a namespace
# sandbox, so on this runner (`unshare(CLONE_NEWNS)` -> EPERM) those skip with a
# named reason and the other ~523 run. See the note above the `jobs:` key.
- name: Upload Linux pytest progress
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: linux-pytest-progress-${{ matrix.group }}
path: ${{ runner.temp }}/pytest-progress/*.jsonl
if-no-files-found: ignore
retention-days: 7
- name: Stage shard coverage data
shell: /usr/local/bin/ci-shell {0}
if: steps.scope.outputs.reduced != 'true'
# Fold any xdist parallel data files into a single .coverage (no-op if
# pytest-cov already combined this shard), then name it per-shard so the
# combine job can merge all shards with `coverage combine`.
run: |
coverage combine 2>/dev/null || true
test -f .coverage || { echo "::error::shard ${{ matrix.group }} produced no coverage data"; exit 1; }
mv .coverage ".coverage.${{ matrix.group }}"
- name: Upload shard coverage
if: steps.scope.outputs.reduced != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-shard-${{ matrix.group }}
path: .coverage.${{ matrix.group }}
# upload-artifact v4 excludes dotfiles unless this is set.
include-hidden-files: true
if-no-files-found: error
# Windows regression line: the backend supports Windows natively via
# kiro_crew.platform_compat (see docs/guides/windows-install.md for the
# per-feature status), but until this job existed nothing in CI held
# that line -- a POSIX-only regression would land silently. Single
# Python (3.12: the preferred Windows runtime; numpy 1.x ships no 3.13
# Windows wheel), trace-free (--no-cov: the coverage-combine job merges
# ubuntu shards only).
#
# Whole-file shards, assigned before import using the same plugin as POSIX.
# Eight runners share the work; the per-job timeout remains unchanged.
# File hashing avoids full-suite item collection in every worker, but does not
# claim equal durations across files or platforms.
backend-test-windows:
name: Backend Tests (Windows)
needs: [changes, await-fast-gate]
# All eight shards share the actor-gated large label; absent output is hosted.
runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }}
# Retain the existing Windows budget and coverage-free run.
timeout-minutes: 60
strategy:
# Full failure picture across shards, same as backend-test.
fail-fast: false
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8]
env:
SHARD_COUNT: 8
# windows-latest subprocess spawn + filesystem latency is slow and
# highly variable; the md_notebook git-ops default (30s) intermittently
# times out on plain `git add` here (#1673). Give git invocations a
# wider budget on this job only.
MDNB_GIT_TIMEOUT_SEC: 120
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-windows-tests
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# Same no-uv.lock caveat as backend-test: key the cache on the
# files that define the dependency set.
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
- name: Verify Windows test prerequisites
if: runner.environment == 'self-hosted'
shell: bash
run: python .github/actions/setup-windows-tests/probe.py
# No surface selector here, unlike the POSIX backend-test job. Narrowing
# this job to the cross-surface subset on a frontend-only diff was
# measured as a PESSIMISATION: 4 shards x ~18 min = 70 job-min for the
# subset, against 62-67 job-min for the whole suite (runs 33412618648 vs
# 33406565382). `-n auto` already saturates each runner, so a smaller
# file list does not shorten a shard, and the cross-surface files are not
# the cheap ones. Running everything is both cheaper and strictly more
# coverage, so there is nothing to select.
- name: Run tests (Windows shard ${{ matrix.group }}/${{ env.SHARD_COUNT }})
# bash (Git Bash on windows-latest), not pwsh: the invocation shares
# the POSIX job's line-continuation and env-interpolation syntax.
#
# --max-worker-restart=0 overrides setup.cfg's =2 for THIS job only.
# pytest-timeout has no SIGALRM on Windows, so it falls back to the
# thread method, which cannot fail a single test -- it terminates the
# whole worker. Every per-test timeout here therefore lands as
# "node down: Not properly terminated", and xdist then enters its
# node-replacement path, where #2803 documented a broken invariant (a
# node in assigned_work but absent from registered_collections). That
# path either dies with INTERNALERROR or, as in #4227, never completes
# the session and burns the full 40-minute cap. Refusing the
# replacement keeps us out of it: the first dead worker ends the run
# promptly with the victim named. setup.cfg keeps =2 deliberately, to
# absorb a one-off memory-pressure crash on a developer machine; a CI
# runner has no such need.
#
# Measured on windows-latest-equivalent pins (pytest 9.0.3, xdist
# 3.5.0, pytest-timeout 2.2.0) with one test forced past the timeout:
# =2 hung indefinitely with no summary and no victim named, and a
# single slow test took down TWO workers because the crashed item is
# re-dispatched to a live one. =0 exited 1 in 12s and named the test.
# The trade-off is deliberate: refusing the replacement abandons the
# work still pending on that shard, so a killed shard reports fewer
# tests than it collected. That is the right trade for CI -- the job
# is red either way, and a named red in 12 minutes beats an
# uninterpretable one at the 40-minute cap.
shell: bash
env:
PYTHONPATH: ${{ github.workspace }}
run: |
# Same as backend-test (#7296): the `python` matcher annotates any
# traceback in the log, including one printed inside a warning, and
# never matches a pytest failure.
echo "::remove-matcher owner=python::"
python -m pytest -p scripts.ci_pytest_progress -p scripts.ci_file_shards \
--ci-progress-dir "$RUNNER_TEMP/pytest-progress" \
-q -n auto --timeout=180 --no-cov --max-worker-restart=0 -m "not ipv6_required" \
--file-shards "$SHARD_COUNT" --file-shard ${{ matrix.group }}
# Ordinary shards plus backend-test-ipv6 retain every Windows test.
# Windows has no namespace sandbox, so `userns_available()` is False and every
# sandbox-dependent test skips. Residual POSIX assumptions in
# test_dogfood_learnings / test_pr_watchers stay tracked as node ids in
# test/windows-expected-failures.txt, which is the mechanism a whole-file deselect
# was standing in for.
- name: Upload Windows pytest progress
# Job timeout can skip even always() steps; periodic CI_PROGRESS lines
# retain partial evidence in the Actions log without relying on this upload.
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: windows-pytest-progress-${{ matrix.group }}
path: ${{ runner.temp }}/pytest-progress/*.jsonl
if-no-files-found: ignore
retention-days: 7
# Native IPv6 loopback is required here, not enabled on the shared fleet.
# Keep both OS contracts, including Windows' real IPv6 listener discovery.
backend-test-ipv6:
name: Backend Tests (IPv6, ${{ matrix.os }})
needs: [changes, await-fast-gate]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e . --group dev
- name: Run required IPv6 tests
env:
ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }}
LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }}
run: |
echo "::remove-matcher owner=python::"
COVERAGE=(--no-cov)
if [ "$RUNNER_OS" = "Linux" ] && [ "$ONLY_FRONTEND" != "true" ] && [ "$LEAF_TESTS" != "true" ]; then
COVERAGE=(--cov=kiro_crew --cov=sage_lib
--cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging
--cov=src/kiro_crew/apps/builtins/code_review_sage/tests)
fi
# The bounded IPv6 population always runs, even in reduced scope.
# Leaf runs repeat it three times; ordinary corpus gates stay once.
REPEAT=1
if [ "$LEAF_TESTS" = "true" ]; then REPEAT=3; fi
for pass in $(seq 1 "$REPEAT"); do
python -m pytest -q -n0 --timeout=120 -m ipv6_required \
${COVERAGE[@]+"${COVERAGE[@]}"} --junitxml="$RUNNER_TEMP/ipv6.xml" \
test/test_meetings_providers.py test/test_platform_compat.py
python - <<'PY'
import os
from pathlib import Path
from defusedxml import ElementTree
report = Path(os.environ["RUNNER_TEMP"]) / "ipv6.xml"
cases = ElementTree.parse(report).findall(".//testcase")
assert len(cases) == 5, f"Expected five native IPv6 cases, got {len(cases)}"
skipped = [case.attrib["name"] for case in cases if case.find("skipped") is not None]
expected_skips = [] if os.environ["RUNNER_OS"] == "Windows" else [
"test_windows_finds_real_ipv6_loopback_listener"
]
assert skipped == expected_skips, f"Unexpected IPv6 skips: {skipped}"
assert not any(case.find("failure") is not None or case.find("error") is not None for case in cases)
PY
done
- name: Stage IPv6 coverage data
if: runner.os == 'Linux' && needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true'
run: |
test -f .coverage || { echo "::error::IPv6 lane produced no coverage data"; exit 1; }
mv .coverage .coverage.ipv6
- name: Upload IPv6 coverage
if: runner.os == 'Linux' && needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-ipv6
path: .coverage.ipv6
include-hidden-files: true
if-no-files-found: error
# Some container kernels omit a dead flock acquirer's PID from their records.
# The orphan test there proves the honest unknown-owner refusal only after
# independently observing the live owner and the later kernel evidence. This
# hosted VM runs the same two tests with KIROCREW_LOCK_OWNER_STRICT=1, which
# refuses that branch: the dead parent must be named exactly, on the runner's
# ordinary filesystem, with no job container and no designated lock root. The
# ordinary shards keep running both tests in their own environment.
backend-test-kernel-lock-owner:
name: Backend Tests (Kernel Lock Owner, hosted)
needs: [changes, await-fast-gate]
runs-on: ubuntu-latest
timeout-minutes: 15
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e . --group dev
- name: Run strict kernel lock owner tests
env:
KIROCREW_LOCK_OWNER_STRICT: "1"
ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }}
LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }}
run: |
echo "::remove-matcher owner=python::"
test -z "${KIROCREW_LOCK_TEST_ROOT:-}"
test "$(stat -f -c %T /proc)" = proc
COVERAGE=(--no-cov)
if [ "$ONLY_FRONTEND" != "true" ] && [ "$LEAF_TESTS" != "true" ]; then
COVERAGE=(--cov=kiro_crew --cov=sage_lib
--cov=src/kiro_crew/apps/builtins/aws_control/crew/packaging
--cov=src/kiro_crew/apps/builtins/code_review_sage/tests)
fi
# Both strict cases always run, even in reduced scope. Leaf runs
# repeat them three times; ordinary corpus gates stay once.
REPEAT=1
if [ "$LEAF_TESTS" = "true" ]; then REPEAT=3; fi
for pass in $(seq 1 "$REPEAT"); do
python -m pytest -q -n0 --timeout=120 \
${COVERAGE[@]+"${COVERAGE[@]}"} --junitxml="$RUNNER_TEMP/kernel-lock-owner.xml" \
"test/test_gateway_lock.py::TestLockHolder::test_live_holder_is_named_and_alive" \
"test/test_gateway_lock_diagnosis.py::test_flock_is_held_by_a_fork_orphan"
python - <<'PY'
import os
from pathlib import Path
from defusedxml import ElementTree
report = Path(os.environ["RUNNER_TEMP"]) / "kernel-lock-owner.xml"
cases = ElementTree.parse(report).findall(".//testcase")
names = sorted(f"{case.attrib['classname']}::{case.attrib['name']}" for case in cases)
assert names == [
"test.test_gateway_lock.TestLockHolder::test_live_holder_is_named_and_alive",
"test.test_gateway_lock_diagnosis::test_flock_is_held_by_a_fork_orphan",
], f"Unexpected strict kernel lock owner cases: {names}"
for case in cases:
for outcome in ("skipped", "failure", "error"):
assert case.find(outcome) is None, f"{case.attrib['name']} {outcome}"
PY
done
- name: Stage kernel lock owner coverage data
if: needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true'
run: |
test -f .coverage || { echo "::error::kernel lock owner lane produced no coverage data"; exit 1; }
mv .coverage .coverage.kernel-lock-owner
- name: Upload kernel lock owner coverage
if: needs.changes.outputs.only_frontend != 'true' && needs.changes.outputs.leaf_tests != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-kernel-lock-owner
path: .coverage.kernel-lock-owner
include-hidden-files: true
if-no-files-found: error
# `kirocrew pod` on Windows, booted for real under Task Scheduler.
#
# A SEPARATE JOB for the same reason the fail-closed canary below is one: a
# step inside backend-test-windows would run in every shard and pay for
# one real gateway boot per shard to make one assertion. It is also the only Windows
# job that needs a built `.venv` INSIDE the checkout, because a pod boots the
# worktree's own binary rather than the control plane's -- the shards install
# with `uv pip install --system` and create no venv at all, which is why
# test_pod_windows_boot.py is listed in test/windows-collect-ignore.txt and
# named explicitly here instead.
#
# What only this job can establish: that the runner user can create a
# scheduled task at all, that `/SC ONCE /ST 00:00` does not auto-fire, that the
# generated .cmd wrapper's quoting is something cmd.exe accepts, that the
# gateway answers over TCP (Windows has no AF_UNIX, so the pod's private
# dashboard socket does not exist there), and that `pod down` really reaps both
# the task and the process tree. Every one of those is faked in the sharded
# test_pod_windows.py, which is what makes this job worth its minutes.
#
# LABEL OPT-IN: a pull request carrying the `ci:pod-scenarios` label ALSO runs
# the full 55-test pod scenario suite (test/e2e/scenarios/) here, against a
# real Vite-built SPA, with the same invocation and count pin as nightly.yml's
# pod-scenarios leg. That is how a current revision gets hosted Windows
# evidence for the suite without dispatching the release-publishing nightly and
# without charging every PR for a Node toolchain, an SPA build, a wheel build
# and a second pod bring-up. Every step the label adds carries the same
# `if: env.POD_SCENARIOS == 'true'`; unlabelled PRs and pushes to main run the
# original boot-only shape, byte for byte. The workflow does not listen for
# `labeled`, so the label must be on the PR BEFORE the push that should run it,
# and a re-run of an earlier run keeps that run's original event payload.
# test/test_pod_scenario_matrix.py pins both shapes.
pod-boot-windows:
name: Pod Boot Canary (Windows)
needs: [await-fast-gate]
# Hosted, and it cannot move. The Windows fleet is a CONTAINER
# (`ContainerAdministrator`), where `schtasks /Create` exits 1 on its own
# `/ST is earlier than current time` warning -- so the runner user cannot
# create a scheduled task, which is the single fact this canary exists to
# establish. Measured there: all 3 tests fail at the probe with
# `WindowsTaskError ... schtasks /Create rc=1`, before any pod boots.
runs-on: windows-latest
# A cold boot pays for the venv build, interpreter start, config seeding and
# the health wait. The test's own bound is 180s; this is the job's. The
# label opt-in measured 76s (boot canary) + 205s (55 scenarios) plus the SPA
# build on a hosted runner, well inside the same ceiling.
timeout-minutes: 30
env:
# `true` only for a pull_request event whose labels include the opt-in;
# anything else (push to main, an unlabelled PR) is the string `false`.
POD_SCENARIOS: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci:pod-scenarios') }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# Same no-uv.lock caveat as backend-test: key the cache on the files
# that define the dependency set.
cache-dependency-glob: |
pyproject.toml
setup.cfg
# Label opt-in only: the scenario suite's pod serves the built SPA, so the
# frontend toolchain is a prerequisite there. Same pin and cache shape as
# the `e2e` job. Skipped on the default boot-only run.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
if: env.POD_SCENARIOS == 'true'
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
# The CONTROL plane: the interpreter that runs pytest and drives the pod
# verbs. Deliberately separate from the payload below, which is the real
# split the pod design rests on.
- name: Install dependencies (control plane)
run: uv pip install --system -e ".[voice,desktop]" --group dev
# Label opt-in only: the wheel scenario runs `python -m build --wheel` with
# THIS interpreter (sys.executable) and, under KIROCREW_E2E_REQUIRE=1, fails
# rather than skips without it. `build` is not in the dev group on purpose.
- name: Install the wheel scenario's build tool (label opt-in)
if: env.POD_SCENARIOS == 'true'
run: uv pip install --system build==1.3.0
# The PAYLOAD: the worktree's own `.venv\Scripts\kirocrew.exe`, which is
# what the scheduled task actually boots. Without this the test FAILS
# naming the missing venv rather than skipping, so a broken step here
# cannot leave the canary silently green.
- name: Build the checkout's own venv (the binary the pod boots)
run: |
uv venv .venv
uv pip install --python .venv/Scripts/python.exe -e "."
# `pod up` refuses a checkout with no served SPA bundle, which is correct
# product behaviour and not what this canary is testing (Task Scheduler
# supervision is). The runner is disposable, so the one-file stand-in is
# placed by the WORKFLOW rather than by the test: a test that wrote into
# the checkout could be killed by its own timeout before cleaning up.
- name: Place a one-file SPA stand-in for `pod up`
if: env.POD_SCENARIOS != 'true'
shell: bash
run: |
mkdir -p src/kiro_crew/static/dist
[ -f src/kiro_crew/static/dist/index.html ] || \
printf '<!doctype html><title>pod boot canary</title>' > src/kiro_crew/static/dist/index.html
# Label opt-in only: the scenario suite drives the served dashboard and
# packages it into the wheel it installs, so the bundle is the real Vite
# build, staged exactly as the `e2e` job and nightly.yml stage it. It
# replaces the stand-in above, so the boot canary serves the same bundle.
- name: Build and stage the SPA bundle (label opt-in)
if: env.POD_SCENARIOS == 'true'
shell: bash
run: |
set -euo pipefail
cd website
npm ci --no-audit --no-fund
npm run build
cd ..
rm -rf src/kiro_crew/static/dist
mkdir -p src/kiro_crew/static
cp -R website/dist src/kiro_crew/static/dist
test -f src/kiro_crew/static/dist/index.html
- name: Assert a real pod boots and is reclaimed (not skipped)
shell: bash
env:
# The module's own opt-in: without it the canary skips itself, and the
# grep below turns that skip into a failure.
KIROCREW_E2E_POD_WINDOWS: "1"
run: |
set -o pipefail
# Same as backend-test (#7296): the `python` matcher annotates any
# traceback in the log, including one printed inside a warning, and
# never matches a pytest failure.
echo "::remove-matcher owner=python::"
# The whole opt-in module: the creation probe, the create/run/delete
# round trip, and the full pod boot. These are the only tests in the
# tree that drive the real Task Scheduler, which is why they live in
# one module that is skipped without KIROCREW_E2E_POD_WINDOWS.
pytest -v -n0 --no-cov --timeout=600 \
test/test_pod_windows_boot.py \
| tee pod-boot.log
# The canary. `pytest` exits 0 on a fully skipped module, so the exit
# code alone cannot tell "a pod booted and was reclaimed" from "the
# test never ran". Grepping the exact count is what makes this honest;
# raise the number when you add a test to this module.
# ANCHORED on purpose: `grep -q "3 passed"` is a substring match, so it
# also HITs "13 passed" and "23 passed" — measured. A pin that accepts a
# larger number pins nothing when a test is added, which is the direction
# a count pin exists to catch. `^3 passed` cannot be used: pytest's
# summary line starts with `=` padding.
grep -qE '(^|[^0-9])3 passed' pod-boot.log || {
echo "::error::the Windows pod boot canary did not report 3 passed (skipped, not collected, or a test was added or renamed without updating this step)"
exit 1
}
# Label opt-in only: the full scenario suite, on the prerequisites the
# canary just used, with the SAME invocation and anchored count as
# nightly.yml's pod-scenarios leg so a green here is evidence for that
# matrix entry at THIS revision. `-o addopts=` drops `-n auto` (one pod for
# the whole session, and a killed xdist worker on Windows aborts the run),
# `--timeout=600` is the per-test cap (the wheel scenario carries its own
# wider mark), and `--basetemp` pins the plane under runner.temp so a failed
# run's pod logs land where the upload step below can find them.
# KIROCREW_E2E_REQUIRE=1 turns every precondition skip into a failure; the
# anchored `55 passed` grep is the second lock, for the same reason as the
# canary's: pytest exits 0 on a collected-but-skipped suite.
- name: Run the full pod scenario suite (label opt-in, 55 tests, not skipped)
if: env.POD_SCENARIOS == 'true'
shell: bash
env:
KIROCREW_E2E_SCENARIOS: "1"
KIROCREW_E2E_REQUIRE: "1"
run: |
set -uo pipefail
echo "::remove-matcher owner=python::"
pytest -v -p no:cacheprovider -o addopts= -n0 --timeout=600 \
--basetemp "${RUNNER_TEMP}/pod-scenarios-tmp" \
test/e2e/scenarios/ 2>&1 | tee pod-scenarios.log
status="${PIPESTATUS[0]}"
echo "pytest exit: ${status}"
# Six user-flow scenarios and 49 fixture-isolation checks must execute.
# Anchored so 155 passed cannot satisfy the 55-test contract; raise the
# number here AND in nightly.yml when you add a scenario.
if ! grep -qE '(^|[^0-9])55 passed' pod-scenarios.log; then
echo "::error::the pod scenario suite did not report 55 passed on windows-latest; a skipped or collapsed suite is not a pass"
exit 1
fi
exit "${status}"
# On the default run only pod-boot.log exists; the scenario globs match
# nothing and `if-no-files-found: ignore` keeps that from being a second
# failure. On a labelled run the pod's own log files are the only place a
# boot refusal is legible, and the plane root is the --basetemp above.
- name: Upload the pod boot log
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: pod-boot-windows
path: |
pod-boot.log
pod-scenarios.log
${{ runner.temp }}/pod-scenarios-tmp/kce2e-*/a/**
${{ runner.temp }}/pod-scenarios-tmp/kce2e-*/h/**/*.log
if-no-files-found: ignore
retention-days: 7
# The Windows fail-closed sandbox opt-in contract, asserted to have RUN.
#
# PR #8117 made `wrap_argv`'s Windows Kiro delegation depend on
# `kiro_internal_sandbox_enabled()`, which reads a settings file a fresh Windows
# install does not have. Delegation stopped, the no-backend policy fail-closed,
# and the ACP backend never started. The suite that pinned that exact branch,
# test_sandbox_argv.py, is line 2 of test/windows-collect-ignore.txt -- so on
# windows-latest it was never collected, and in the Linux shards it reached the
# branch only by monkeypatching sys.platform to "win32" AND patching the
# settings probe to return True, hardcoding the one answer a fresh Windows host
# cannot give.
#
# A SEPARATE JOB rather than a step in backend-test-windows: that job is a
# 4-shard matrix, so a canary step there would run four times and pay the
# gateway boot four times over for one assertion. It also overrides
# --max-worker-restart for xdist reasons that do not apply to an -n0 run.
#
# Run by NODE ID with `-v -n0` and grep the pass count, the same shape as the
# macOS peer-identity canary below and for the same reason: `pytest -q` does not
# name passing tests and a skip exits 0, so a canary that quietly stopped
# running (a changed skipif, a collect-ignore entry, a renamed node) would leave
# this job green while the contract it proves went unverified. That is precisely
# the blindness that let test_sandbox_argv.py sit unrun inside conftest's
# Windows collect_ignore.
#
# The file is deliberately NOT in test/windows-collect-ignore.txt, so the
# backend-test-windows shards collect it too. This job is the assertion that it
# RAN, not the only place it runs.
backend-test-windows-fail-closed:
name: Windows Fail-Closed Sandbox Contract
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }}
# Keep the real gateway/ACP contract and its existing timeout unchanged.
timeout-minutes: 25
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-windows-tests
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# Same no-uv.lock caveat as backend-test: key the cache on the files
# that define the dependency set.
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
- name: Verify Windows test prerequisites
if: runner.environment == 'self-hosted'
shell: bash
run: python .github/actions/setup-windows-tests/probe.py
# A fresh windows-latest image has no ~/.kiro/settings/amazon-internal.json,
# which IS the precondition under test. Assert it from the workflow as well
# as from the fixture: if a future runner image or an earlier step ever
# created it, the suite would SKIP with a clear reason and the grep below
# would fail -- but this step names the cause on the run page instead of
# leaving it to be read out of a pytest skip line.
- name: Assert the host carries no kiro internal-sandbox settings file
shell: pwsh
run: |
$settings = Join-Path $env:USERPROFILE ".kiro\settings\amazon-internal.json"
if (Test-Path -LiteralPath $settings) {
Write-Host "::error::$settings exists on this runner; the fresh-install precondition this job pins cannot be observed"
exit 1
}
Write-Host "no $settings on this host, as a fresh Windows install would be"
- name: Assert the Windows fail-closed contract runs (not skipped)
shell: bash
run: |
set -o pipefail
# Same as backend-test (#7296): the `python` matcher annotates any
# traceback in the log, including one printed inside a warning, and
# never matches a pytest failure.
echo "::remove-matcher owner=python::"
pytest -v -n0 --no-cov --timeout=900 \
test/test_windows_fail_closed_optin.py \
| tee canary.log
# Anchored: an unanchored "7 passed" also matches "17 passed" / "27
# passed". See the pod-boot canary above for the measurement.
grep -qE '(^|[^0-9])7 passed' canary.log || {
echo "::error::the Windows fail-closed sandbox contract did not fully PASS (skipped, not collected, or a test was added/removed without updating this count)"
exit 1
}
# The macOS pytest lane is NOT here any more. It lives in
# .github/workflows/platform-tests.yml, which nightly.yml calls, and it gates the
# nightly PUBLISH jobs there.
#
# It was moved for one MEASURED reason, and the reason is queueing, not runtime.
# On three consecutive green PR runs (34866269260, 34864945056, 34863753125) the
# macOS jobs WAITED 176, 190 and 213 minutes for a macos-15 runner and then ran
# for 26-33. Everything else in this workflow finished at 78-98 minutes; the runs
# themselves took 248-268. So ~64 percent of a pull request's CI wall clock was
# macOS queue time, and "PR Readiness" -- this repository's only required check --
# is triggered by `workflow_run` on THIS workflow's completion, so that queue sat
# directly on the merge button. Over the same window `Backend Tests (macOS)`
# failed 0 times and was CANCELLED 129 times, mostly still queued when a newer
# push superseded the run: the lane was charging every pull request ~2.8 hours and
# usually returning no verdict at all.
#
# Linux does not have this problem because it does not use GitHub's hosted pool:
# `changes` routes it to this repository's CodeBuild runners. There is no
# self-hosted macOS equivalent, so the queue is not something this workflow can
# tune -- only something it can stop paying per pull request.
#
# NOTHING on the pull_request path may instantiate a macOS runner, and that is
# asserted, not hoped: a `continue-on-error` macOS job would still hold this
# workflow's completion and therefore still hold readiness, so an advisory lane
# cannot live in this file. The opt-in real-Darwin run is its own workflow
# (macos-on-demand.yml, which calls platform-tests.yml), outside readiness'
# lane list.
# test_macos_platform_tests_gate.py pins all of it.
# The job where Linux mount-namespace tests EXECUTE rather than skip. They need
# unprivileged user namespaces (unshare NEWNS), which ubuntu-24.04 restricts via
# AppArmor -- the same restriction codex-review.yml clears for its bwrap sandbox.
# Without this job the ~/.kiro/crew sensitive-path keystone has no CI coverage at all,
# even though branch protection treats breaking it as a blocking offence.
#
# The shards COLLECT all eleven of these files (nothing is deselected any more), but the
# `skipif(not userns_available())` tests inside SIX of them skip there -- so this job is
# the only place those 85 assertions execute, the ~/.kiro/crew sensitive-path keystone
# among them. The other five carry no such guard and run fully in the shards too; they
# are listed here because running them against a REAL sandbox is a different test than
# running them against a stubbed one, and this is the only lane that can.
#
# The list is hand-maintained, so `test_coverage_omit_contract.py` asserts that every
# file carrying the guard appears on the argv below. Without that, a twelfth guarded
# file would skip in every shard, join no lane, and be silently green -- which is
# exactly how these nine came to execute nowhere.
#
# The nine ops/auto_improvement suites were rejected from this job once, as "~6 minutes
# against a real git" and "a poor fit for a 20-minute job". That cost was the launcher's
# pre-exec hardlink scan arming on every spawn (1.5s per sandboxed spawn), since fixed.
# MEASURED afterwards under this job's own environment -- real sandbox, no gh token, no
# user session bus -- all eleven: 694 passed, 1 skipped, 38s.
#
backend-test-sandbox:
name: Backend Tests (namespace sandbox)
needs: [await-fast-gate]
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 2
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
- name: Enable unprivileged user namespaces
run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
- name: Verify the namespace sandbox is actually available
# Fail LOUDLY if the runner image ever stops allowing this, rather than
# letting the suite silently skip and the gate go green having asserted
# nothing -- the same failure mode the deselects created.
run: unshare --mount --map-root-user true
- name: Verify Kiro Crew's own probe agrees with the kernel
# The probe is stricter than the bare `unshare` above: it also performs the
# launcher's propagation mount inside the new namespace, because a host
# that grants the namespaces and refuses that mount cannot build the
# sandbox. A probe that wrongly said "no sandbox" here would make every
# guarded test below skip and this job go green having asserted nothing,
# so the probe's verdict is asserted on its own, with its reason.
run: |
python3 -c "from kiro_crew.sandbox import userns_available, unavailable_reason; assert userns_available(), unavailable_reason()"
- name: Run sandbox-dependent tests
# `--no-cov` deliberately. This is the only lane that reaches these lines, so
# measuring it is what would let setup.cfg drop the omit for the six guarded test
# files -- but that needs this job's data uploaded as an artifact and
# `coverage-combine` taught to consume it, which couples the coverage gate to a
# job it does not depend on today. Separate change; the omit carries the pointer.
run: |
# Same as backend-test (#7296): the `python` matcher annotates any
# traceback in the log, including one printed inside a warning, and
# never matches a pytest failure.
echo "::remove-matcher owner=python::"
pytest -q --timeout=180 --no-cov \
test/test_script_hooks.py \
test/test_cron_script.py \
src/kiro_crew/apps/builtins/ops_mission_control/tests/test_ledger_sync_git.py \
src/kiro_crew/apps/builtins/ops_mission_control/tests/test_schedule_file.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_dogfood_learnings.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_lint_findings.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_github_profile.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_profile_capture.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_runner.py \
src/kiro_crew/apps/builtins/auto_improvement/tests/test_perf_track_propose.py
- name: Require real V1 vector acceleration regressions
timeout-minutes: 5
# Keep optional accelerator dependencies out of the base shards and
# type-check environment. Reuse installed test dependencies read-only.
run: |
uv venv --system-site-packages "$RUNNER_TEMP/member-faiss-venv"
uv pip install --python "$RUNNER_TEMP/member-faiss-venv/bin/python" "faiss-cpu==1.12.0"
"$RUNNER_TEMP/member-faiss-venv/bin/python" -m pytest -v -n0 --no-cov --timeout=60 \
test/test_memory_record_metadata.py::test_bulk_episode_edit_invalidates_live_and_saved_faiss_vectors \
test/test_memory_record_metadata.py::test_another_process_vector_edit_cannot_reuse_old_faiss_scores_after_restart \
test/test_memory_v1_acceleration.py::test_v1_forgotten_row_keeps_resident_numpy_scoring \
test/test_memory_v1_acceleration.py::test_v1_faiss_overfetches_past_forgotten_top_hits \
--junitxml="$RUNNER_TEMP/member-faiss-results.xml"
python - <<'PY'
import os
import xml.etree.ElementTree as ET
from pathlib import Path
report = Path(os.environ["RUNNER_TEMP"]) / "member-faiss-results.xml"
cases = list(ET.parse(report).iter("testcase"))
expected = {
name + "[" + version + "]"
for name in (
"test_bulk_episode_edit_invalidates_live_and_saved_faiss_vectors",
"test_another_process_vector_edit_cannot_reuse_old_faiss_scores_after_restart",
)
for version in ("v1", "v2")
}
expected.update(
{
"test_v1_forgotten_row_keeps_resident_numpy_scoring",
"test_v1_faiss_overfetches_past_forgotten_top_hits",
}
)
assert len(cases) == 6 and {case.attrib["name"] for case in cases} == expected
assert all(case.find(tag) is None for case in cases for tag in ("skipped", "failure", "error"))
PY
- name: Upload FAISS regression evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: member-faiss-regressions
path: ${{ runner.temp }}/member-faiss-results.xml
if-no-files-found: warn
- name: Select optional member recall evidence
id: member-benchmark
if: github.event_name == 'pull_request'
run: |
CHANGED="$(git diff --name-only HEAD^1 HEAD)"
if printf '%s\n' "$CHANGED" | grep -Eq '^(src/kiro_crew/(.*memory.*|embeddings\.py|context\.py|eval/bench/.*)|scripts/ci-member-memory-benchmark\.py|\.github/workflows/ci\.yml)$'; then
echo "selected=true" >> "$GITHUB_OUTPUT"
fi
- name: Locate the pinned member benchmark model
id: member-benchmark-model
if: steps.member-benchmark.outputs.selected == 'true'
continue-on-error: true
env:
KIROCREW_HOME: ${{ runner.temp }}/member-memory-benchmark-home
run: |
python - <<'PY' >> "$GITHUB_OUTPUT"
from kiro_crew import embeddings
print(f"digest={embeddings._GGUF_SHA256}")
print(f"path={embeddings.default_model_path().parent}")
PY
- name: Cache the pinned member benchmark model
if: steps.member-benchmark-model.outputs.digest != ''
continue-on-error: true
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.member-benchmark-model.outputs.path }}
key: gguf-${{ steps.member-benchmark-model.outputs.digest }}
- name: Measure member recall with the real pinned Qwen model
if: steps.member-benchmark.outputs.selected == 'true'
continue-on-error: true
timeout-minutes: 11
# Model availability is optional; retrieval scores are observations.
# Completed structural measurements are enforced by the next step.
env:
KIROCREW_HOME: ${{ runner.temp }}/member-memory-benchmark-home
KIROCREW_SKIP_MODEL_DOWNLOAD: ""
MEMBER_BENCHMARK_OUTPUT: ${{ runner.temp }}/member-memory-benchmark-evidence
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
mkdir -p "$MEMBER_BENCHMARK_OUTPUT"
set -o pipefail
if timeout --kill-after=15s 600s python scripts/ci-member-memory-benchmark.py \
2>&1 | tee "$MEMBER_BENCHMARK_OUTPUT/run.log"; then
echo "Member recall measurement completed; inspect the artifact's scores, bounds and provenance." >> "$GITHUB_STEP_SUMMARY"
else
echo "::warning::Member recall measurement did not finish cleanly; inspect the benchmark artifact and structural validation."
echo "Read provenance for availability versus completed structural failures; no cached report is substituted." >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
- name: Enforce completed member recall structural checks
if: always() && steps.member-benchmark.outputs.selected == 'true'
env:
MEMBER_BENCHMARK_OUTPUT: ${{ runner.temp }}/member-memory-benchmark-evidence
run: python scripts/ci-member-memory-benchmark.py --validate-structures
- name: Upload current member recall evidence
if: always() && steps.member-benchmark.outputs.selected == 'true'
continue-on-error: true
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: member-recall-qwen-${{ github.sha }}
path: ${{ runner.temp }}/member-memory-benchmark-evidence/
retention-days: 30
if-no-files-found: warn
backend-test-crew-container:
name: Backend Tests (crew container)
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/requirements.txt
# Two installs, deliberately not merged into one resolution.
#
# `-e .` and the dev group are what every other backend lane installs, and this
# suite needs them for a reason that is not obvious: no container test imports
# `kiro_crew`, but the repo-root conftest's autouse isolation fixtures do
# (`importlib.import_module("kiro_crew.apps")`), so a venv without the package
# errors at test SETUP rather than at collection. Measured: 233 errors, every
# one a ModuleNotFoundError on a transitive app dependency.
#
# The image's own runtime pins come from `container/requirements.txt` and are
# installed ONLY here. That file's header forbids them becoming dependencies of
# the application ("the app's backend hooks are imported into every owner's
# gateway process, and a web framework there buys nothing"), which is exactly
# why this cannot be folded into the `backend-test` shards: their environment IS
# the application's.
#
# Not `container/requirements-dev.txt`, which is the file the review named. It
# carries `-r requirements.txt`, pytest and pytest-asyncio and nothing else,
# while this rootdir's addopts pass `-n auto --dist loadgroup
# --max-worker-restart=2 --timeout=120`; with no xdist and no pytest-timeout
# installed that is an argument error, not a test run. Its `pytest==8.3.4` also
# contradicts the dev group's `9.0.3` that the rootdir conftest is written
# against, so the two sets cannot share one resolution. The image build still
# uses that file; CI installs the runtime half plus this repo's own toolchain.
- name: Install dependencies
run: |
uv pip install --system -e "." --group dev
uv pip install --system -r \
src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/requirements.txt
# CREW_CONTAINER_TESTS_REQUIRED is the point of this job, not a detail of it.
# The suite's conftest answers a missing image dependency with
# `collect_ignore_glob`, which takes the WHOLE suite out of collection while the
# run still reports success. Under this variable every reason it would decline to
# collect becomes a hard error, and the collection is additionally checked against
# the tree: every module that defines tests must yield at least one, and the total
# must clear a floor. So a dependency rename reds THIS lane instead of quietly
# restoring a lane that reports success while collecting nothing.
#
# `-n0`: the suite forks real process trees and binds real ports, so xdist
# workers would race each other for both. It runs in about ten seconds serially,
# which is less than the workers cost to start.
#
# `--no-cov` for the same reason as the sibling sandbox lane: this data is not
# part of the coverage gate's combine, and half-reported coverage is worse than
# none.
- name: Run the crew container suite
env:
CREW_CONTAINER_TESTS_REQUIRED: "1"
run: |
# As in backend-test (#7296): the `python` matcher annotates any traceback
# in the log, including one printed inside a warning, and never matches a
# pytest failure.
echo "::remove-matcher owner=python::"
pytest -q -n0 --timeout=180 --no-cov \
src/kiro_crew/apps/builtins/aws_control/crew/runtime/container_tests/
# ── Real-adapter contract lane (the measurements no other runner makes) ──
# The codex and opencode projections rest on facts about the ADAPTER, not about
# Crew: which `mcpServers` element shape a `session/new` accepts, whether the
# named child is really launched, what environment reaches it, and which
# transports are refused outright. Those cannot be asserted from Crew's side, so
# they are measured against a real adapter -- and every guarded test skipped on
# every runner, because no runner had one installed. A guard that never runs
# where drift arrives is the one place drift is invisible: a codex-acp release
# that reshaped the element would break every codex session's MCP surface with
# nothing red to say so.
#
# Per pull request rather than nightly. The facts change when the adapter revs,
# but what makes them load-bearing changes when Crew's own projection changes,
# and this lane is what tells the author of that change. The whole job is ~5
# minutes with the install cached.
real-adapter-contract:
name: Real Adapter Contract Tests
needs: [changes, await-fast-gate]
# Skipped on a frontend-only diff, like every other backend lane: nothing the
# frontend surface can change moves an adapter contract, and the install is a
# registry download this job should not make to prove that. Deliberately NOT a
# narrower path filter -- what makes these facts load-bearing is any change to
# Crew's projection, and the selector cannot prove which backend file that is.
if: needs.changes.outputs.only_frontend != 'true'
# On the fleet, under the same non-root boundary as the backend shards: the
# adapters are spawned through Crew's own session code, which refuses
# provider execution as root. Setup runs as the runner identity; the
# contract tests and the junit read-back run through ci-shell.
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 30
env:
# THE point of this job. Every guarded test skips where its adapter is absent
# -- correct on a developer's box, and what kept these contracts unmeasured in
# CI. Here the adapters are installed deliberately, so an absent one is a
# FAILURE: without this a broken npm install produces a green job that
# asserted nothing, which is the state the lane exists to end.
#
# This is the repository's EXISTING switch for "this job declared its
# preconditions must hold" -- the E2E scenario suites, the Playwright gate and
# the gateway-boot matrix all read the same name -- rather than a second
# spelling of it. Each job sets it for its own module set; this job's set is
# the `real_adapter` marker below.
KIROCREW_E2E_REQUIRE: "1"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
# No extras: these tests drive a subprocess adapter and a stub MCP server,
# and the rootdir conftest needs the package importable. Neither reaches the
# voice or desktop surfaces.
- name: Install dependencies
run: uv pip install --system -e "." --group dev
# Node 24, the version every other lane in this file installs. codex-acp is
# a Node stdio server and the codex guard also asserts `node` is on PATH.
# Node 24, the version every other lane in this file installs. codex-acp is
# a Node stdio server and the codex guard also asserts `node` is on PATH.
# The npm cache is keyed on the adapters' OWN lockfile -- the repo pattern,
# with a different lockfile -- so a warm run pays extraction only.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: test/real_adapters/package-lock.json
# From the COMMITTED lockfile, never an ad hoc `npm install pkg@X`. The
# manifest in test/real_adapters/ pins one exact version per adapter, and
# `npm ci` installs the closure that lockfile records -- top-level AND
# transitive -- so a cache miss cannot change what this lane measures. A pin
# alone does not do that: `npm install pkg@X` re-resolves the package's own
# dependency ranges on every run (zizmor adhoc-packages).
#
# The same manifest is what test/real_adapter_gate.py reads for the measured
# versions, and what Dependabot refreshes weekly. A bump therefore arrives as
# its own PR, and THAT PR's run of this lane is where adapter drift shows up.
# Nothing here holds a credential: the tests fabricate an API key inside a
# throwaway CODEX_HOME, and session/new makes no model call.
- name: Install the pinned adapters from the lockfile
working-directory: test/real_adapters
run: npm ci --no-audit --no-fund
# ASSERTS the pin held, rather than trusting the install: an adapter whose
# binary cannot start, or a manifest edited without its lockfile, would
# otherwise be measured as if it were the pinned release. The expected
# versions come from the manifest through the gate module -- the one reader
# the tests use -- so this step and the assertions cannot name different
# releases.
- name: Assert the installed adapters are the pinned ones
run: |
python3 - <<'PY'
import json
import subprocess
import sys
sys.path.insert(0, "test")
import real_adapter_gate as gate
modules = gate.MANIFEST_DIR / "node_modules"
manifest = modules / gate.CODEX_ACP_PACKAGE / "package.json"
codex = json.loads(manifest.read_text(encoding="utf-8"))["version"]
assert codex == gate.MEASURED_CODEX_ACP_VERSION, (
f"codex-acp {codex} installed, {gate.MEASURED_CODEX_ACP_VERSION} pinned"
)
# The binary's own answer, not its manifest: this is also the proof the
# installed executable starts on this runner.
opencode = subprocess.run(
[str(modules / ".bin" / "opencode"), "--version"],
check=True, capture_output=True, encoding="utf-8", timeout=120,
).stdout.strip()
assert opencode == gate.MEASURED_OPENCODE_VERSION, (
f"opencode {opencode!r} installed, {gate.MEASURED_OPENCODE_VERSION} pinned"
)
print(f"codex-acp {codex}, opencode {opencode}: both the pinned release")
PY
- uses: ./.github/actions/run-as-runner
- name: Run the real-adapter contract tests
shell: /usr/local/bin/ci-shell {0}
# Both resolution ladders read an explicit override first, so the lane
# names the entry it installed instead of measuring whatever else a runner
# image happens to ship.
env:
CODEX_ACP_BIN: ${{ github.workspace }}/test/real_adapters/node_modules/@agentclientprotocol/codex-acp/dist/index.js
OPENCODE_BIN: ${{ github.workspace }}/test/real_adapters/node_modules/.bin/opencode
run: |
# As in backend-test: the `python` matcher annotates any traceback in the
# log, including one printed inside a warning, and never matches a pytest
# failure.
echo "::remove-matcher owner=python::"
pytest -v -n0 --no-cov --timeout=1200 -m real_adapter \
--junitxml="$RUNNER_TEMP/real-adapter-results.xml" \
test/test_codex_session_mcp.py \
test/test_opencode_session_mcp.py
# The second half of the no-silent-skip rule, and the half that does not trust
# the first. KIROCREW_E2E_REQUIRE turns an absent adapter into a failure; this
# asserts on the REPORT that the selected cases ran. A variable that never
# arrived, a marker selection that matched nothing, or a skip added inside a
# test body all reach this step as a skipped or missing case rather than as a
# green lane.
#
# FLOOR, not an inventory: a count can only be raised, so it cannot go stale
# the way a copied list of names does, and its one job is to refuse a run that
# measured FEWER than the contracts known to exist. Which tests those are is
# the marker's business, and the parity test names the files.
- name: Require every contract test to have executed
if: always()
env:
MEASURED_CONTRACT_FLOOR: "4"
run: |
python3 - <<'PY'
import os
import xml.etree.ElementTree as ET
from pathlib import Path
report = Path(os.environ["RUNNER_TEMP"]) / "real-adapter-results.xml"
cases = list(ET.parse(report).iter("testcase"))
floor = int(os.environ["MEASURED_CONTRACT_FLOOR"])
names = sorted(case.attrib["name"] for case in cases)
assert len(cases) >= floor, f"selected {len(cases)} of at least {floor}: {names}"
for case in cases:
for tag in ("skipped", "failure", "error"):
found = case.find(tag)
assert found is None, (
f"{case.attrib['name']} reported {tag}: "
f"{found.attrib.get('message', '')[:400]}"
)
print(f"{len(cases)} real-adapter contract tests executed, none skipped: {names}")
PY
coverage-combine:
name: Coverage Combine
needs: [backend-test, backend-test-ipv6, backend-test-kernel-lock-owner, changes]
# A frontend-only diff runs the backend suite as a coverage-free subset, so
# there are no shard artifacts to combine. Coverage Gate asserts this job is
# exactly `skipped` in that case, and `success` otherwise.
#
# A leaf-test diff is the SAME situation for a different reason, and it needs
# naming separately: it is a BACKEND diff, so `only_frontend` is false, and
# without this clause the job would run and fail on artifacts the leaf lane
# deliberately never uploaded.
if: >-
needs.changes.outputs.only_frontend != 'true'
&& needs.changes.outputs.leaf_tests != 'true'
# CodeBuild-hosted runner (pilot); artifact-only job, default size. Rollback: `ubuntu-latest`.
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install coverage
run: uv pip install --system coverage==7.10.6
- name: Download shard coverage data
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: coverage-shard-*
merge-multiple: true
- name: Download required IPv6 coverage data
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: coverage-ipv6
- name: Download required kernel lock owner coverage data
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: coverage-kernel-lock-owner
- name: Combine into unified coverage.xml
# The downloaded .coverage.<group> files are parallel data files;
# `coverage combine` merges them. -o pins the output path (setup.cfg's
# [coverage:xml] otherwise redirects it under build/coverage/).
run: |
test -f .coverage.ipv6 || { echo "::error::required IPv6 coverage data missing"; exit 1; }
test -f .coverage.kernel-lock-owner || { echo "::error::required kernel lock owner coverage data missing"; exit 1; }
coverage combine
coverage xml -o coverage.xml
coverage report
- name: Upload combined coverage
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-backend
path: coverage.xml
# ── Coverage Gate ──────────────────────────────────────────────────────
# Enforces minimum line-coverage thresholds. Rather than re-running the
# entire backend + frontend suites (which duplicated ~30 min of work the
# test jobs already did on a single serial runner), this job CONSUMES the
# coverage artifacts those jobs already produced in this same run:
# * backend-test (3.12 shards) -> coverage-combine -> coverage.xml
# (artifact: coverage-backend)
# * frontend-test -> build/coverage/cobertura-coverage.xml
# (artifact: coverage-frontend)
# Result: download + parse + threshold-check only -- runs in seconds.
#
# Fail-closed: if an expected artifact/report is missing the job errors
# (download-artifact errors on a missing artifact; the report presence is
# asserted below), so a broken coverage pipeline can never silently pass.
# Lives here (not code-review.yml) so it can use `needs:` for a
# deterministic same-run handoff instead of a cross-workflow artifact race.
# The check name "Coverage Gate" is unchanged, so branch protection that
# requires it is still satisfied.
coverage-gate:
name: Coverage Gate
# CodeBuild-hosted runner (pilot); artifact-only job, default size. Rollback: `ubuntu-latest`.
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 10
needs: [coverage-combine, frontend-test, frontend-coverage-merge, backend-test, backend-test-ipv6, backend-test-kernel-lock-owner, changes]
# Single source of truth for the floors -- referenced by the check steps,
# their names, and the summary table so the three can never drift apart.
#
# RATCHET POLICY: keep each floor a little under what `main` actually
# measures, so the gate blocks a real coverage regression without going red
# on ordinary noise. Raise a floor only after main has held the higher
# number for a while, and never above the LOWEST recent main measurement.
#
# Backend measured 90.57% on main (2026-08-18) -> floor 90%
# Frontend measured 91.32-91.47% on main (2026-08-18) -> floor 90%
#
# Both were sampled across three consecutive successful main runs, because
# the policy above is written against the LOWEST recent measurement and one
# sample cannot establish that.
#
# The headroom is deliberately THIN, and asymmetrically so: frontend keeps
# ~1.5pp, backend only ~0.6pp. At 0.6pp a single merge that adds a few
# hundred uncovered statements reds the gate, so backend coverage now has to
# be maintained with each change rather than recovered in later waves. That
# is the intended trade: the two lanes are held at the round number they
# actually reached. If backend does start bouncing off 90, the fix is to
# lower THAT floor to 89 and say why -- not to weaken the gate's logic.
#
# Why not 85%, which is where this change started: the lanes crossed 90%
# while it sat unmerged, and a floor 5-6pp under main protects almost
# nothing. The point of the gate is to catch a regression, so the number has
# to track what main actually holds.
#
# How backend got here, so the next person can judge the headroom:
# 78.69% starting point
# 80.15% #2090 stopped measuring test files CI never executes (the
# BACKEND_DESELECTS set) -- an accounting fix, not new tests.
# Since retired: CI runs those files, so they are measured again.
# 81.50% #2103 added unit coverage for 7 pure-logic-ish modules
# 81.94% #2115 added 11 dashboard/Slack/app-platform modules
# 86.93% #2947 and 88.84% #3005, ranked by uncovered-statement MASS
# rather than by percentage -- a 550-statement module at 17% is
# worth more than a small one at 60%
# 89.57% #3118, which converted only 58% of the statements it targeted
# where #3005 converted more than it targeted: what remains sits
# behind I/O, subprocess and network boundaries
# 90.57% work merged since
#
# And frontend, which moved later and converts differently:
# 60.46% 2026-08-07, before any dedicated coverage work
# 83.38% the zero-coverage sweep (files with no test file at all)
# 86.14% #2948 and #3014
# 88.29% #3132, where three surfaces starting at or near 0% produced 739
# of its 1,470 statements while ChatPage.tsx -- the single largest
# target by uncovered count, 481 -- produced nothing measurable
# 91.47% work merged since
# Two lessons in those two lines, both of which cost a wave to learn: rank
# candidates by how UNCOVERED a file is and not merely by its uncovered
# count, because a file near 0% converts far better than a large page
# already at 80%; and do not size a frontend wave off a backend one.
env:
BACKEND_MIN: 90
FRONTEND_MIN: 90
# The per-file floor, applied to BOTH lanes with a per-lane baseline of
# the files that were already below it. A project-wide average lets a
# well-covered large file pay for a bare small one, so it can be satisfied
# without touching the files that carry the risk; a per-file floor removes
# that subsidy. The rationale, the measured cost of this number, and the
# three baseline rules live in scripts/check_per_file_coverage.py's
# docstring -- deliberately not restated here, where it would go stale.
PER_FILE_MIN: 80
# Always run so the required check emits a real verdict. Without
# `if: always()`, a failing OR skipped dependency would SKIP this job, and
# GitHub treats a skipped required check as SATISFIED -- silently defeating
# the fail-closed guarantee (including the `if-no-files-found: error` upload
# guard this workflow relies on). The first step below converts any
# non-success upstream result into an explicit failure.
if: ${{ always() }}
steps:
- name: Require upstream coverage jobs to have succeeded
env:
CC: ${{ needs.coverage-combine.result }}
LEAF_TESTS: ${{ needs.changes.outputs.leaf_tests }}
FE: ${{ needs.frontend-test.result }}
FCM: ${{ needs.frontend-coverage-merge.result }}
BE: ${{ needs.backend-test.result }}
IPV6: ${{ needs.backend-test-ipv6.result }}
KERNEL_LOCK_OWNER: ${{ needs.backend-test-kernel-lock-owner.result }}
ONLY_BACKEND: ${{ needs.changes.outputs.only_backend }}
ONLY_FRONTEND: ${{ needs.changes.outputs.only_frontend }}
run: |
echo "coverage-combine=$CC frontend-test=$FE frontend-coverage-merge=$FCM backend-test=$BE only_backend=$ONLY_BACKEND only_frontend=$ONLY_FRONTEND"
# Empty flags mean the surface-detection job never reported (it failed
# or was cancelled) -- its verdict can't be trusted, so fail closed.
if [ -z "$ONLY_BACKEND" ] || [ -z "$ONLY_FRONTEND" ]; then
echo "::error::surface detection (changes) never reported -- failing closed."
exit 1
fi
# Both test jobs always RUN (only their scope narrows), so both must
# always have succeeded -- a subset run that fails is still a failure.
# backend-test is checked explicitly because on a frontend-only diff
# coverage-combine is skipped by design and would otherwise be the only
# signal, masking a failed reduced backend suite.
if [ "$FE" != "success" ]; then
echo "::error::frontend-test=$FE -- failing closed."
exit 1
fi
# Frontend coverage lane: frontend-coverage-merge unions the shard
# blobs into the coverage-frontend artifact. Required to succeed,
# EXCEPT on a backend-only diff where the frontend suite ran as a
# coverage-free subset and the merge job is skipped by design.
# Anything else (failure, cancellation, dependency-skip, or a run that
# happened when it shouldn't have) fails closed.
if [ "$ONLY_BACKEND" = "true" ]; then
if [ "$FCM" != "skipped" ]; then
echo "::error::frontend-coverage-merge=$FCM but this run was backend-only (expected exactly 'skipped') -- inconsistent."
exit 1
fi
elif [ "$FCM" != "success" ]; then
echo "::error::frontend-coverage-merge=$FCM -- failing closed."
exit 1
fi
if [ "$IPV6" != "success" ]; then
echo "::error::backend-test-ipv6=$IPV6 -- failing closed."
exit 1
fi
# The hosted strict lane is required in every scope, like the IPv6
# lane: it always runs, so anything but success (failure, cancellation,
# skip, or an empty result) fails closed.
if [ "$KERNEL_LOCK_OWNER" != "success" ]; then
echo "::error::backend-test-kernel-lock-owner=$KERNEL_LOCK_OWNER -- failing closed."
exit 1
fi
if [ "$BE" != "success" ]; then
echo "::error::backend-test=$BE -- failing closed."
exit 1
fi
# Backend coverage lane: required to succeed, EXCEPT on a frontend-only
# diff where the backend suite ran as a coverage-free subset and
# coverage-combine is therefore skipped by design. Anything else
# (failure, cancellation, dependency-skip, or a run that happened when
# it shouldn't have) fails closed.
# A leaf-test run is coverage-free by design (the backend lane runs a
# subset, whose coverage is not comparable to the repo floor), so
# coverage-combine is skipped for it exactly as it is for a
# frontend-only run. Checked BEFORE the frontend branch because the two
# flags are independent and a leaf diff sets only this one.
if [ "$LEAF_TESTS" = "true" ]; then
if [ "$CC" != "skipped" ]; then
echo "::error::coverage-combine=$CC but this run was a leaf-test run (expected exactly 'skipped') -- inconsistent."
exit 1
fi
echo "Leaf-test run: coverage-combine skipped by design; backend coverage not asserted."
elif [ "$ONLY_FRONTEND" = "true" ]; then
if [ "$CC" != "skipped" ]; then
echo "::error::coverage-combine=$CC but this run was frontend-only (expected exactly 'skipped') -- inconsistent."
exit 1
fi
elif [ "$CC" != "success" ]; then
echo "::error::coverage-combine=$CC -- failing closed."
exit 1
fi
# Checked out for the per-file gate: it needs scripts/check_per_file_coverage.py
# and the two baselines from the tree. It sits BEFORE the artifact
# downloads because actions/checkout cleans the workspace by default and
# would otherwise delete them.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# The gate parses the Cobertura reports through defusedxml rather than the
# stdlib parser, which resolves external entities -- and on a fork PR the
# report is produced by contributor-controlled code. Installed here because
# this job otherwise needs no Python dependencies.
#
# setup-python is a PREREQUISITE, not decoration: the runner image's /usr
# interpreter is externally managed (PEP 668), so `uv pip install --system`
# against it exits 2 with "externally managed". coverage-combine provisions
# the same pair for the same reason.
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Install the per-file gate's XML parser
run: uv pip install --system 'defusedxml>=0.7,<1'
- name: Download backend coverage
if: needs.coverage-combine.result == 'success'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: coverage-backend
path: backend-cov
- name: Download frontend coverage
if: needs.changes.outputs.only_backend != 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: coverage-frontend
path: frontend-cov
- name: Check backend coverage (min ${{ env.BACKEND_MIN }}%)
if: needs.coverage-combine.result == 'success'
run: |
F=backend-cov/coverage.xml
test -f "$F" || { echo "::error::coverage.xml missing from coverage-backend artifact"; exit 1; }
# Compare the raw line-rate against the threshold; round only for
# display (a round-then-compare would let 89.95%..89.99% pass 90%).
python3 - "$F" "$BACKEND_MIN" Backend <<'PY'
import sys, xml.etree.ElementTree as ET
path, floor, label = sys.argv[1], float(sys.argv[2]), sys.argv[3]
rate = float(ET.parse(path).getroot().attrib["line-rate"]) * 100
print(f"{label} coverage: {rate:.2f}%")
if rate < floor:
print(f"::error::{label} coverage {rate:.2f}% below {floor:.0f}% minimum")
sys.exit(1)
PY
- name: Check frontend coverage (min ${{ env.FRONTEND_MIN }}%)
if: needs.changes.outputs.only_backend != 'true'
run: |
F=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1)
test -n "$F" || { echo "::error::cobertura report missing from coverage-frontend artifact"; exit 1; }
python3 - "$F" "$FRONTEND_MIN" Frontend <<'PY'
import sys, xml.etree.ElementTree as ET
path, floor, label = sys.argv[1], float(sys.argv[2]), sys.argv[3]
rate = float(ET.parse(path).getroot().attrib["line-rate"]) * 100
print(f"{label} coverage: {rate:.2f}%")
if rate < floor:
print(f"::error::{label} coverage {rate:.2f}% below {floor:.0f}% minimum")
sys.exit(1)
PY
- name: Check per-file coverage, backend (floor ${{ env.PER_FILE_MIN }}%)
# Gated on the same condition as the project floor: on a frontend-only
# diff the backend suite ran as a coverage-free subset, and a subset's
# per-file rates are not comparable to a baseline recorded on the full
# suite -- enforcing them there would fail files whose tests never ran.
if: needs.coverage-combine.result == 'success'
run: |
python3 scripts/check_per_file_coverage.py backend-cov/coverage.xml \
--label Backend --floor "$PER_FILE_MIN" \
--baseline .github/coverage-baselines/backend.txt
- name: Check per-file coverage, frontend (floor ${{ env.PER_FILE_MIN }}%)
if: needs.changes.outputs.only_backend != 'true'
run: |
F=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1)
test -n "$F" || { echo "::error::cobertura report missing from coverage-frontend artifact"; exit 1; }
python3 scripts/check_per_file_coverage.py "$F" \
--label Frontend --floor "$PER_FILE_MIN" \
--baseline .github/coverage-baselines/frontend.txt
- name: Coverage summary
if: always()
run: |
echo "## Coverage" >> "$GITHUB_STEP_SUMMARY"
echo "| Component | Coverage | Min |" >> "$GITHUB_STEP_SUMMARY"
echo "|-----------|----------|-----|" >> "$GITHUB_STEP_SUMMARY"
BF=backend-cov/coverage.xml
if [ -f "$BF" ]; then
B=$(python3 -c "import xml.etree.ElementTree as ET; print(f\"{float(ET.parse('$BF').getroot().attrib['line-rate'])*100:.2f}\")")
echo "| Backend | ${B}% | ${BACKEND_MIN}% |" >> "$GITHUB_STEP_SUMMARY"
fi
FF=$(find frontend-cov -name 'cobertura*.xml' 2>/dev/null | head -1)
if [ -n "$FF" ]; then
FR=$(python3 -c "import xml.etree.ElementTree as ET; print(f\"{float(ET.parse('$FF').getroot().attrib['line-rate'])*100:.2f}\")")
echo "| Frontend | ${FR}% | ${FRONTEND_MIN}% |" >> "$GITHUB_STEP_SUMMARY"
fi
frontend-lint:
name: Frontend Lint & Type Check
needs: [changes, await-fast-gate]
# eslint/tsc/i18n over website/** only; a backend-only diff leaves every
# input byte-identical. only_backend is the catch-all side of the same
# decision, so an unclassifiable path still runs this.
if: needs.changes.outputs.only_backend != 'true'
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 20
steps:
# persist-credentials retained: frontend-lint/Check for new hardcoded UI strings runs `git fetch origin`
# against the same remote, through the shared diff-base resolver script.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install dependencies
working-directory: website
run: npm ci --no-audit --no-fund
- name: Type check
working-directory: website
# Spelled out rather than `npm run typecheck` so this gate cannot be
# changed from package.json. The project is named explicitly: the root
# tsconfig is `files: []` + project references, so a bare `tsc --noEmit`
# there compiles an empty program. Not `tsc -b`: build mode judges
# "up to date" from the project's own inputs only, so with the incremental
# cache in tsconfig.app.json it would green-light a changed dependency
# `.d.ts` without re-checking a single file. `-p` re-hashes the whole program.
run: npx tsc -p tsconfig.app.json
- name: Lint
working-directory: website
# Blocking, and the tree is at ZERO: the ceiling is a hard zero rather than
# a stored baseline, so any warning a change introduces fails here. A hard
# zero is the only ceiling that needs no bookkeeping — there is no count to
# re-measure and no ceiling that can drift above the truth.
#
# NEVER raise this number. A warning admitted into slack never surfaces
# again: it is indistinguishable from the pre-existing ones and nothing
# downstream reports it. Fix the warning, or — if the code is genuinely
# right — suppress that one line with
# `// eslint-disable-next-line <rule> -- <why the code is correct>`, which
# is reviewable in the diff where a raised ceiling is not.
run: npx eslint src/ --max-warnings 0
- name: Copy/paste detection
working-directory: website
run: npx jscpd .
- name: Check i18n extraction, key references and plurals
working-directory: website
env:
# Two gates need a commit to diff against: `check-source-strings.mjs` compares
# the catalogs against it to find NEWLY ADDED keys and CHANGED values, and
# `check-i18n-strings.mjs` uses it to find literals on lines this branch wrote.
# On a PR this is `base.sha` — the commit the MERGE REF was computed against —
# NOT `origin/<base.ref>`. The tip is a moving target measured at step time while
# the checked-out tree is a snapshot from job start, so anything that lands on
# `main` in between appears only on the base side and is charged to every PR
# running in that window. Observed: a PR whose run started 36s before #1223
# merged was billed 58 render findings for `WelcomeView` translations its head
# simply did not have yet. `base.sha` makes the two sides consistent by
# construction. On a push to main it is the commit the push replaced, so a merge
# that only breaks in combination with another merge is still charged to a diff. Note `frontend-lint`'s
# checkout is shallow, so it has to be fetched — see the step below.
# `check-i18n-strings.mjs` falls back from the merge base to the base tip for
# exactly that reason, and EXITS NON-ZERO if a configured ref cannot be
# resolved rather than skipping itself green.
I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
# `npm run i18n:check` is a RUNNER (`website/scripts/i18n-check.mjs`), not an `&&`
# chain: it executes all six scripts and reports the eleven checks they contain in
# one table, because `&&` short-circuits and a PR only ever learned about its first
# failing gate. `--check` on the two codemods is deliberately not `--dry-run`:
# dry-run reports and exits 0, which would make this step decorative. An
# unrecognised flag makes those scripts refuse to run rather than fall through to
# their destructive conversion path.
#
# ONLY THREE KINDS OF CHECK CAN FAIL THIS STEP.
#
# DIFF-SCOPED, zero tolerance — a finding on a line you wrote is yours and there
# is no number to raise:
# [added-lines] a user-visible literal on a line THIS BRANCH WROTE.
# [vs-base] a file you touched holds more than it did at the base.
# [source-strings] new ENGLISH keys must be well-shaped copy.
# [changed-values] catalog QA over values this branch added or changed.
#
# WHOLE-REPO HARD ZEROS — no ceiling to inherit, so a new one is always a diff:
# [key-refs] a `t()` naming a key that does not exist. i18next returns a
# missing key as its own fallback, so the symptom is a raw
# dotted key rendered to a user. Necessarily whole-repo: a
# reference and its catalog entry are edited independently, so
# a catalog-only rename dangles references in files the PR
# never opens. Needs no base ref, so it cannot skip itself.
# [plurals] an i18nT-adjacent `{n} {t('item')}s` concatenation.
# [pseudolocale] `en-XA.json` stale against its generator.
#
# WHOLE-REPO CEILING — fails only when the class GROWS:
# [plurals-hardcoded] the same plural-glue defect with NO i18nT in it, e.g.
# a fully hardcoded aria-label gluing `?'s':''` onto a
# noun. The frozen sites are pinned at HARDCODED_CEILING
# in i18n-plural-codemod.mjs and ratchet DOWN as they are
# converted; a failure lists every site with file:line so
# the added one is findable.
#
# EVERYTHING ELSE IS REPORT ONLY — [untranslated], [allcaps], [extractable] and
# [dynamic-keys]. Each is a stored whole-repo total, written by whichever branch
# measured it last, so another branch can push it past its number without touching
# your files — and then the failure names no diff anyone can fix. Measured twice in
# one day: `main` sat at 1120 against its own ALL-CAPS ceiling of 1118 within
# minutes of it being set, and every open PR inherited the red. [dynamic-keys] was
# the worst, failing when a file IMPROVED. The numbers are still printed; they do
# not gate. See docs/ci/i18n-gates.md for the residual gap this leaves on [dynamic-keys]
# specifically, and the qa `CEILINGS` / `deadKeys` / `glossary` ratchets that are
# still exact in both directions.
#
# `check-i18n-strings.mjs` drives a SECOND eslint invocation with its own config.
# `no-literal-string` at `mode: 'all'` reports thousands of findings, and folding
# those into the budget above would make an i18n regression indistinguishable
# from a new `no-explicit-any`.
run: |
# `actions/checkout` fetches depth 1, so the base branch is not present and
# `git show origin/main:…` would fail. Fetch just that one ref.
#
# `set -e` on the chain is the point: `check-source-strings.mjs` SKIPS ITSELF
# and exits 0 when the base ref is unreadable, so a failed fetch used to sail
# past this step and silently disable the new-key gate for the whole PR --
# observed in practice, where the script printed "OK: skipped -- cannot read
# website/src/i18n/locales/en.json at origin/main (shallow clone?)" and the
# step still went green. A gate that cannot run must fail, not pass: an
# unavailable base ref is a CI problem to fix, not a reason to stop checking.
# Absolute path: this step's working-directory is `website`.
I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \
"$I18N_BASE_REF" 'the new-key i18n gate')" || exit 1
export I18N_BASE_REF
npm run i18n:check
- name: Check the color utilities this change touches name real tokens
# A Tailwind utility naming a token the theme never declared is not an
# error anywhere: the class is simply not emitted, the element renders
# with no color, and the build, tsc and eslint all stay silent. Diff-
# scoped for the same reason as the focus-cue gate above: the tree
# carried a backlog of these, so a whole-tree gate would charge it to
# whoever pushed next. Literals a change touched are complete for
# regression, and the whole-tree count is printed as a non-failing
# report either way.
working-directory: website
env:
PHANTOM_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
run: |
# Absolute path: this step's working-directory is `website`.
PHANTOM_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \
"$PHANTOM_BASE_REF" 'the phantom-classes gate')" || exit 1
export PHANTOM_BASE_REF
# Self-test first: it plants one probe per rule family, so a change
# that breaks the lexer or promotes a non-class string to a class
# fails here instead of shipping a gate that passes everything.
npm run lint:phantom-classes -- --test
npm run lint:phantom-classes
- name: Check no file restyles more ui/ primitives than the baseline allows
# `shadcn/no-restyle` stays off in eslint.config.js: the tree carries a
# few hundred call sites that restyle ui/ primitives and the eslint gate
# is a hard zero. Off means the backlog can grow silently, so this step
# runs the rule through its own config and holds each file at the count
# recorded in website/scripts/restyle-baseline.json — a rise or a new
# file fails; a fall fails too until `--update-baseline` records it, so
# progress is locked in rather than left to a log line. That flag never
# raises a number. Whole-tree, no base ref: the baseline is committed,
# so the gate is complete for regression without a diff. Not eslint's
# own bulk suppressions: the Node API editors use ignores that file, and
# the CLI loads it for every config, so the i18n eslint run would fail
# on "unused" entries (both measured; see the script header).
working-directory: website
run: |
# Self-test first: it plants one restyled primitive and asserts the
# rule fires under this config, so a plugin or parser change that
# silences the rule fails here instead of passing every count as 0.
npm run lint:restyle-ratchet -- --test
npm run lint:restyle-ratchet
# Prove `npm ci` still works on the OLDEST Node the frontend claims to support.
# Every other job here runs the .nvmrc toolchain (Node 24 / npm 11), and npm 11
# and npm 10 (Node 22) build different ideal trees for an OPTIONAL PEER
# dependency: npm 10 wants a nested entry for it, npm 11 does not install it at
# all. A lockfile regenerated under npm 11 therefore drops what npm 10 demands,
# and `npm ci` dies with EUSAGE ("Missing: <pkg> from lock file") for everyone
# on the declared floor while all of CI stays green. That happened twice to the
# same dependency, the second time as a side effect of a change about something
# else entirely.
#
# The specific divergence is gone -- an `overrides` entry now makes both majors
# resolve that dependency identically, so neither can drop anything. This job
# exists because that is a property of today's dependency tree, not a guarantee:
# the next package with an unsatisfiable optional peer reintroduces the whole
# class, and without a job on the floor nothing would notice.
lockfile-engines-floor:
name: Lockfile Installs On Declared Node Floor
needs: [changes, await-fast-gate]
# Reads website/package.json + website/package-lock.json and nothing else.
if: needs.changes.outputs.only_backend != 'true'
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Pinned to the floor, NOT to `engines.node` via node-version-file: that
# input resolves a range like ">=22" to the NEWEST matching release, which
# would quietly re-test Node 24 and make this job vacuous. The literal is
# held to engines.node statically instead, by
# test_the_engines_floor_job_pins_the_declared_floor -- one derivation of
# the floor, in the file that already governs every other node pin.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22
cache: npm
cache-dependency-path: website/package-lock.json
- name: npm ci on the engines floor
working-directory: website
# A real install, not `--dry-run`: dry-run still fires the postinstall
# patch script but leaves no node_modules for it to patch, so it fails
# for a reason that has nothing to do with the lockfile. Installing for
# real also makes the gate mean what its name says -- `npm ci` works on
# the declared floor -- rather than only that resolution succeeded.
#
# The failure hint matters because this job is the ONLY one on the floor:
# whoever trips it is on Node 24 and cannot reproduce the red locally, so
# naming the reproduction turns an opaque failure into a self-servicing
# one.
run: |
npm ci --no-audit --no-fund || {
echo "::error::npm ci failed on Node $(node -v), the engines.node floor." \
"Every other job runs the .nvmrc toolchain (Node 24 / npm 11), so this can" \
"fail while they pass: the two npm majors resolve optional peer dependencies" \
"differently. Reproduce without installing Node 22:" \
"cd website && npx -y npm@10 ci. If it is a lockfile disagreement, prefer" \
"removing the divergence (an overrides entry both majors resolve the same way)" \
"over hand-adding whichever entry one major happens to want."
exit 1
}
cfn-lint:
name: CloudFormation Lint
needs: [changes, await-fast-gate]
# The templates it lints live under src/kiro_crew/deploy/, which the
# paths-filter counts as backend, so a frontend-only diff cannot touch one.
if: needs.changes.outputs.only_frontend != 'true'
# PILOT (first wave): runs on a CodeBuild-hosted runner (project
# `kirocrew-gha-linux`, Ubuntu 22.04) instead of ubuntu-latest. The label is
# chosen once in `changes` (forks and fork PRs get ubuntu-latest, because the
# runner never serves them). Rollback: `ubuntu-latest`. Policy, image caveat
# and exit condition: docs/ci/ci-and-reviews.md, "CodeBuild-hosted runner (pilot)".
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install cfn-lint
# Pinned: an unpinned lint tool silently changes the gate's behaviour on
# every upstream release. Bump deliberately.
run: pip install "cfn-lint==1.22.3"
- name: Lint deploy templates
run: cfn-lint src/kiro_crew/deploy/skills/artifact-deploy/templates/*.yaml
# The Electron shell (website/electron) has its own 460+ test node:test suite,
# and until now NOTHING ran it on a PR — its only runner was ota-test.yml,
# which is nightly/on-demand and called by no other workflow. So main-process
# regressions (permission handling, packaging/entitlements contracts, the
# updater IPC surface) could land green and surface a day later, or in a
# shipped DMG. That is not hypothetical: the missing macOS microphone
# entitlement reached users this way. This job is deliberately tiny and fast —
# pure unit tests, no Electron binary download, no packaging.
# NOT diff-gated, deliberately. The obvious gate (only_backend != 'true')
# is wrong: test/packaging.test.js pins the macOS entitlement contract
# against BOTH signing lanes, and one of them is
# packaging/signing/Entitlements.entitlements -- outside website/, so the
# paths-filter counts it as backend. A packaging-only diff would therefore
# skip the job whose absence already shipped a broken bundle once (the
# missing microphone entitlement above). Same for the
# packaging/installer-assets/* references it pins.
electron-test:
name: Electron Shell Tests
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/electron/package-lock.json
- name: Install dependencies
working-directory: website/electron
# Belt-and-braces: this pin currently downloads no Electron binary
# anyway (electron 43's package ships no install script), so the var is
# inert today — it is set so a future version that reintroduces the
# ~100MB postinstall download cannot silently slow this job down. The
# tests require() only main-process modules and the stdlib test runner,
# so the binary is never needed.
env:
ELECTRON_SKIP_BINARY_DOWNLOAD: "1"
run: npm ci --no-audit --no-fund
- name: Unit tests
working-directory: website/electron
run: npm test
# The Linux job above fakes realpath and Win32_Process. The Windows port-owner
# identity (windows-port.js + gateway-stop.js) rests on two OS premises no
# fake can prove -- realpathSync.native follows an NTFS junction, and a
# process launched through a junction is matched however Win32_Process spells
# its ExecutablePath -- so those test files also run once on a real Windows
# host. No npm ci: the two modules under test require only the Node stdlib.
electron-test-windows:
name: Electron Shell Tests (Windows port identity)
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.windows_runner || 'windows-latest' }}
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
- name: Windows port-identity tests
working-directory: website/electron
run: node --test test/windows-port.test.js test/windows-port-junction.test.js test/windows-port-real-junction.test.js
frontend-test:
name: Frontend Tests
needs: [changes, await-fast-gate]
# CodeBuild-hosted runner (pilot, second wave). `instance-size:large` for the
# vitest --coverage shards' memory; routing and rollback as for cfn-lint. See
# docs/ci/ci-and-reviews.md, "CodeBuild-hosted runner (pilot)".
# Rollback: `ubuntu-latest`.
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
# Per-shard budget. The whole suite was ~23 min on one runner; split four
# ways each shard is well under this, and a hung shard fails fast.
timeout-minutes: 25
strategy:
# Don't cancel sibling shards when one fails -- we want the full failure
# picture across the suite, not just the first shard to break.
fail-fast: false
matrix:
# vitest splits by TEST FILE (not test case) across shards. 4 shards
# cuts the ~950-file suite roughly fourfold and confines a fork-worker
# crash to one shard instead of reddening the whole job. To change the
# count, edit this list AND SHARD_COUNT together -- mind the ~20
# concurrent-job ceiling for public repos.
shard: [1, 2, 3, 4]
env:
SHARD_COUNT: 4
steps:
# persist-credentials retained: frontend-test/Run the diff-scoped vitest gates runs `git fetch origin`
# against the same remote, through the shared diff-base resolver script.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install dependencies
working-directory: website
run: npm ci --no-audit --no-fund
- name: Select test scope
id: scope
env:
ONLY_BACKEND: ${{ needs.changes.outputs.only_backend }}
TARGETS_FILE: ${{ runner.temp }}/frontend-targets.txt
# Mirror of the backend selector: a backend-only diff cannot affect a
# spec that never touches the backend surface. Cross-surface specs (and
# anything unclassified) still run. Any failure falls back to the FULL
# suite rather than silently narrowing it.
run: |
if [ "$ONLY_BACKEND" != "true" ]; then
echo "reduced=false" >> "$GITHUB_OUTPUT"
echo "Full frontend suite (diff is not backend-only)."
exit 0
fi
rc=0
python3 scripts/ci-surface-tests.py --surface frontend --summary > "$TARGETS_FILE.all" || rc=$?
if [ "$rc" != "0" ] || [ ! -s "$TARGETS_FILE.all" ]; then
echo "::warning::surface selector unusable (rc=$rc) -- running the FULL frontend suite."
echo "reduced=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# vitest runs with cwd=website and only covers website/src; the
# Electron shell specs belong to electron-test, which stays always-on.
sed -n 's#^website/##p' "$TARGETS_FILE.all" | grep -v '^electron/' > "$TARGETS_FILE" || true
if [ ! -s "$TARGETS_FILE" ]; then
echo "::warning::no in-scope cross-surface specs resolved -- running the FULL frontend suite."
echo "reduced=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "reduced=true" >> "$GITHUB_OUTPUT"
echo "Reduced scope: $(wc -l < "$TARGETS_FILE") cross-surface spec(s)."
- name: Unit tests (shard ${{ matrix.shard }}/${{ env.SHARD_COUNT }})
working-directory: website
env:
# Two vitest gates are DIFF-SCOPED rather than baseline-scoped —
# `localeFormatting.test.ts`'s `[added-lines]` / `[vs-base]`. They read the
# base commit out of git instead of storing a count, and they skip themselves
# when this is empty. Without it they would skip on every PR, which is the
# "gate nobody watched go red" failure mode their own docs warn about.
# Same expression as `frontend-lint`: the base branch on a PR, the commit the
# push replaced on a push to main.
I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
REDUCED: ${{ steps.scope.outputs.reduced }}
TARGETS_FILE: ${{ runner.temp }}/frontend-targets.txt
# `actions/checkout` fetches depth 1, so the base is not present and
# `git diff`/`git show` against it would fail. Fetch just that one commit, and
# fail the step if it cannot be fetched — the gates throw on an unresolvable
# configured ref rather than skipping green, and this keeps the two consistent.
run: |
I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \
"$I18N_BASE_REF" 'the diff-scoped i18n test gates')" || exit 1
export I18N_BASE_REF
if [ "$REDUCED" = "true" ]; then
# Subset run (backend-only diff): coverage of a subset is not
# comparable to the repo floor, so it is not collected and the
# Coverage Gate skips the frontend lane for this run. Still sharded
# and blob-reported so the merge job produces one unified result.
TARGETS=()
while IFS= read -r line; do line=${line%$'\r'}; if [ -n "$line" ]; then TARGETS+=("$line"); fi; done < "$TARGETS_FILE"
# Still sharded so the matrix step is uniform; a target set smaller
# than SHARD_COUNT leaves some shards with no files, so
# --passWithNoTests keeps an empty shard green (a legitimately empty
# slice, not a lost suite).
# blob alone prints no verdict (it only writes the artifact), so a
# failing shard's log would end at "exit code 1" with no FAIL line
# or summary; the second reporter keeps a human-readable verdict in
# this job's own log.
npx vitest run --reporter=blob --reporter=default --passWithNoTests \
--shard=${{ matrix.shard }}/"$SHARD_COUNT" ${TARGETS[@]+"${TARGETS[@]}"}
else
# --reporter=blob stores results + per-shard coverage under
# website/.vitest-reports/; the frontend-coverage-merge job stitches
# the four shards back into one report and the merged cobertura the
# Coverage Gate reads. Coverage is emitted per shard so the merge
# can union it -- a single shard covers only the files it ran.
# blob alone prints no verdict (it only writes the artifact), so a
# failing shard's log would end at "exit code 1" with no FAIL line
# or summary; the second reporter keeps a human-readable verdict in
# this job's own log.
npx vitest run --coverage --reporter=blob --reporter=default \
--shard=${{ matrix.shard }}/"$SHARD_COUNT"
fi
- name: Upload shard blob report
# Always upload (even a failed shard) so the merge job can render the
# full picture; skipped only if the run itself was cancelled. The blob
# carries this shard's results AND its coverage data; the merge job
# unions them. .vitest-reports is a dot dir -> include-hidden-files.
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: frontend-blob-${{ matrix.shard }}
path: website/.vitest-reports/*
include-hidden-files: true
if-no-files-found: error
retention-days: 1
frontend-coverage-merge:
name: Frontend Coverage Merge
needs: [frontend-test, changes]
# Runs whenever the sharded run collected coverage -- any diff that is not
# backend-only. (A backend-only diff runs the frontend suite as a
# coverage-free subset, so there is nothing to merge and the Coverage Gate
# skips the frontend lane.) `!cancelled()` lets it still stitch a report
# when a shard failed, so the failure surfaces in one place.
#
# The `!= 'skipped'` clause is what keeps that `!cancelled()` honest now that
# the shards sit behind await-fast-gate: a red fast gate skips frontend-test,
# and without this the merge would still run, find no coverage artifact, and
# go red for a reason that has nothing to do with the gate the developer has
# to fix. A skipped shard set has nothing to stitch; a FAILED one still does.
if: >-
${{ !cancelled()
&& needs.frontend-test.result != 'skipped'
&& needs.changes.outputs.only_backend != 'true' }}
# CodeBuild-hosted runner (pilot), default size: a merge is one process
# re-reading four blobs, not a test run. Rollback: `ubuntu-latest`.
runs-on: ${{ needs.changes.outputs.linux_runner || 'ubuntu-latest' }}
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install dependencies
working-directory: website
run: npm ci --no-audit --no-fund
- name: Download shard blob reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: website/.vitest-reports
pattern: frontend-blob-*
merge-multiple: true
- name: Normalize blob paths to this runner's workspace
# A blob stores every path absolute, and vitest's coverage map is keyed
# by those paths. Hosted runners all check out under the same root, so
# the shards and this job agree by accident; the CodeBuild-hosted runner
# gives every build its own root (/codebuild/output/src<random>/...), so
# four shards arrive with four roots and the merge unions nothing: each
# file is reported four times with one shard's numbers and the step
# exits non-zero with no failing test (pilot runs 1 and 2, 2026-09-12).
# This rewrites each blob's root to this checkout's; a no-op when they
# already match, so hosted runs are unaffected.
working-directory: website
run: node "$GITHUB_WORKSPACE/.github/scripts/frontend-blob-normalize-paths.mjs" .vitest-reports
- name: Merge reports and coverage
id: merge
working-directory: website
# --merge-reports stitches the per-shard blobs back into one result set
# WITHOUT re-running tests; --coverage unions the per-shard coverage and
# re-emits the configured reporters (cobertura/lcov) under
# build/coverage/ -- the exact path + artifact name the Coverage Gate
# already consumes. A failed shard makes this exit non-zero, surfacing
# the failure here too.
run: npx vitest --merge-reports --coverage
- name: Reconcile shard verdicts against blob results
# Runs even when the merge step failed -- that is the case it exists
# for. A shard (or the merge) can exit non-zero without naming a
# failing test anywhere: a worker killed mid-run (the worker-OOM mode
# documented in website/vite.config.ts) fails the job while every test
# in the blob reads "pass". This step decodes the downloaded blobs,
# prints every failing test and unhandled runner error by name, and
# states the contradiction explicitly when a red verdict has no failing
# test behind it. Diagnostic only: it always exits 0 and never changes
# the job's verdict.
if: ${{ !cancelled() }}
working-directory: website
env:
FRONTEND_TEST_RESULT: ${{ needs.frontend-test.result }}
MERGE_RESULT: ${{ steps.merge.outcome }}
run: node "$GITHUB_WORKSPACE/.github/scripts/frontend-blob-reconcile.mjs" .vitest-reports
- name: Upload merged coverage
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-frontend
# vitest's reportsDirectory is website/build/coverage (see
# vite.config.ts). Fail loudly if the cobertura report is missing so a
# broken merge can never hand the Coverage Gate an empty artifact.
path: website/build/coverage/
if-no-files-found: error
bundle-size:
name: Bundle Size Gate
needs: [changes, await-fast-gate]
# The analyze build reserves a 6 GiB Node heap; use the large memory tier.
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
timeout-minutes: 25
# A backend-only diff cannot change the frontend bundle, so this gate has
# nothing to measure there. Meta diffs (.github/**, scripts/**) leave both
# surface flags false and still run the gate -- this job's own config lives
# in that bucket.
if: needs.changes.outputs.only_backend != 'true'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install dependencies
working-directory: website
run: npm ci --no-audit --no-fund
- name: Build in analyze mode
working-directory: website
# `--mode analyze` activates the kirocrew-bundle-report plugin (see
# vite.config.ts), which emits dist/bundle-report.json with per-chunk
# sizes. A normal `npm run build` deliberately never writes one, so the
# gate below measures a dedicated build instead of taxing the real one.
# The memory flag mirrors package.json's `build` script.
run: node --max-old-space-size=6144 ./node_modules/vite/bin/vite.js build --mode analyze
- name: Per-chunk size gate
working-directory: website
# Explicit ceilings for known-large chunks, a 500 KB default for every
# other chunk; fails with one line per breach. Budgets and rationale:
# website/scripts/check-bundle-size.mjs (CHUNK_BUDGETS).
run: node scripts/check-bundle-size.mjs
- name: Chunk cycle gate
working-directory: website
# Reuses the analyze build above rather than paying for a second one, which
# is why it lives in this job instead of its own. The job KEEPS its name:
# `Bundle Size Gate` is a required check, and renaming it to match the
# widened scope would silently stop satisfying branch protection.
#
# Different failure from the size gate, and one nothing else here can see.
# Two chunks that statically import each other have no valid
# initialization order, so a body runs against an uninitialized binding
# and the page blanks before React mounts. Bytes stay fine, tsc stays
# fine, and the unit suite never loads a bundle -- a config edit
# (rolldown's `includeDependenciesRecursively: false`) produced two cycles
# with every other gate green, which is what this closes.
run: node scripts/check-chunk-cycles.mjs
# The browser gate runs on the CodeBuild fleet. The one step that needed a
# user namespace -- the real private-workflow MCP run -- moved to
# `e2e-private-namespace` below, the only piece of this lane the fleet cannot
# host (`unshare --map-root-user` is EPERM there). Everything left here needs
# a gateway, a Node toolchain and a Chromium, all of which the fleet has.
#
# The suite's own gateway takes unsandboxed consent for its disposable config
# (`test/test_playwright_e2e.py`, `sandbox_allow_unsandboxed_exec`): the agent
# binary is a stdlib echo stub, and the container refuses `CLONE_NEWUSER` at
# the runtime policy level rather than through a sysctl this job could clear.
# A sandboxed spawn doing real work stays proven by `e2e-private-namespace`
# and by `e2e-boot-matrix`, which asserts both sandbox tiers.
#
# Same non-root boundary as `e2e-boot-matrix`: the gateway refuses provider
# execution under euid 0, so the tests run through ci-shell as `runner`.
# Setup (installs, the SPA build, the browser download) runs before the
# boundary under the runner identity and writes GITHUB_OUTPUT / GITHUB_ENV,
# which the boundary then makes read-only; the test steps after it write only
# the workspace and RUNNER_TEMP. `PLAYWRIGHT_BROWSERS_PATH` pins the browser
# to RUNNER_TEMP (written to GITHUB_ENV by the version step, since job-level
# env cannot read the runner context) so the cache restore (root, HOME=/root)
# and the specs (runner, HOME=/home/runner) resolve the same directory.
e2e:
name: E2E (stub ACP backend, offline)
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
timeout-minutes: 25
defaults:
run:
shell: /usr/local/bin/ci-shell {0}
steps:
# persist-credentials retained: e2e/Scan the rendered dashboard runs `git fetch origin`
# against the same remote, through the shared diff-base resolver script.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: pip
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install backend (+ dev)
# setuptools is explicit: Python 3.12 no longer bundles it, and an
# editable install doesn't add it to the runtime venv -- but
# `setup.py test_e2e` imports setuptools.Command, so it must be present.
shell: bash
run: |
python -m pip install --upgrade pip setuptools
pip install -e ".[voice,desktop]" --group dev
- name: Install frontend deps
shell: bash
working-directory: website
run: npm ci --no-audit --no-fund
- name: Build frontend
shell: bash
working-directory: website
run: npm run build
- name: Stage frontend into package
shell: bash
# The dashboard Playwright specs render the bundled UI the harness
# gateway serves from src/kiro_crew/static/dist. Mirror nightly.yml's
# build-wheel staging so the specs hit a real dashboard, not a 404.
run: |
rm -rf src/kiro_crew/static/dist
mkdir -p src/kiro_crew/static
cp -R website/dist src/kiro_crew/static/dist
# The browser install is the single largest source of red on this job, and
# never because a test failed: across 112 consecutive E2E jobs the
# conclusions were 76 success / 36 cancelled / 0 failure, and 22 of those
# cancels trace to this step. `--with-deps` runs `apt-get update` first, and
# when the runner's default mirror (azure.archive.ubuntu.com) answers `Ign:`
# apt falls back to archive.ubuntu.com and stalls silently -- observed at
# 23min, 15min and 12min, against a 25min job budget that then has nothing
# left for the specs.
#
# `--with-deps` is dropped rather than retried because it buys nothing this
# gate asserts on. Every shared library Chromium needs is already on the
# ubuntu-latest image (the apt log reports libnss3, libx11-6, libxcb1,
# libasound2 and libxkbcommon0 as `already the newest version`); the only
# packages it newly installs are 9 CJK/Thai/Cyrillic font packages. The suite
# has no pixel comparison (no toHaveScreenshot / toMatchSnapshot anywhere
# under website/), playwright.config.ts pins `locale: 'en-US'` so every spec
# renders the English catalog, and the i18n render gate below reads
# textContent rather than measuring geometry. A future spec that asserts
# glyph METRICS for a non-Latin script would need those fonts back -- add
# them as their own bounded, non-fatal step, not by restoring `--with-deps`.
#
# The version key must be exact: a restore-key prefix could hand the job a
# different Chromium revision than @playwright/test expects.
- name: Resolve Playwright version
id: playwright-version
shell: bash
working-directory: website
run: |
version="$(node -p \
"require('./package-lock.json').packages['node_modules/@playwright/test'].version")"
echo "version=$version" >> "$GITHUB_OUTPUT"
echo "PLAYWRIGHT_BROWSERS_PATH=$RUNNER_TEMP/ms-playwright" >> "$GITHUB_ENV"
- name: Cache Playwright browsers
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/ms-playwright
key: ms-playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}-chromium
# Runs unconditionally rather than behind `cache-hit != 'true'`: on a hit it
# is a ~2s no-op, and that keeps a partially-written cache entry from turning
# into a missing-browser failure inside the specs.
- name: Install Playwright browser (Chromium)
# A step-level cap. The download itself takes ~7s, so anything approaching
# this is a stalled mirror or CDN -- fail here with budget left for the
# specs instead of consuming the job's 25min and reporting nothing.
timeout-minutes: 6
shell: bash
working-directory: website
run: npx playwright install chromium
- name: MCP-App message rows render both delivery states (asserting harness)
# The same self-asserting harness that produced the PR's screenshot
# evidence: serves the built SPA over loopback with fixture-backed
# /api/** responses and fails on any scene assertion (envelope leak,
# missing attribution, missing text). Lives here to reuse the build
# and the Chromium install above; needs no gateway, token or backend.
timeout-minutes: 5
shell: bash
working-directory: website
run: node scripts/capture-mcp-app-message-rows.mjs "$RUNNER_TEMP/mcp-app-message-rows"
- name: Resolve base for the parallel i18n render-time gate
# Phase 5 of the i18n plan. Every other i18n gate reads SOURCE or CATALOG
# JSON, so three defect classes are invisible to all of them: a string that
# was never extracted (ordinary TypeScript to a scanner), a sentence
# assembled from several keys (several correct lookups), and English in a
# title/aria-label (never in the text flow). This renders the real SPA under
# the en-XA pseudolocale and reads them off the DOM.
#
# It lives in this job to reuse the Chromium install on the step above --
# its own job would re-download the browser for a ~5min script. It needs no
# gateway, no token and no backend: the script serves the build over
# loopback and answers every /api/** call from fixtures, like the
# capture-*.mjs harnesses. It builds its OWN bundle rather than reusing
# `npm run build` above, because en-XA is DEV-only and a production build
# tree-shakes the catalog out entirely.
#
# Two runs, one per tree: [vs-base] renders the BASE commit with the same
# scanner and fails on any per-surface increase, reading no committed number.
# It is the ONLY thing that fails; the debt record in
# `src/i18n/render-baseline.json` is printed and never enforced, because a
# stored total is written by whichever branch measured it last and two branches
# that never touch the same file can combine into a number neither produced --
# `projects.latent 16 -> 24` from #1107 + #985, red on main with no diff anyone
# could fix. A push to main therefore gets a base too (the commit it replaced),
# which is what keeps that interaction enforced somewhere. It costs a second
# bundle build on merges, the same ~5min every PR already pays.
# The base must be fetched first because this checkout is shallow; the script
# EXITS NON-ZERO if a configured ref cannot be resolved rather than skipping
# itself green, for the reason the frontend-lint job documents.
shell: bash
working-directory: website
env:
I18N_BASE_REF: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
run: |
I18N_BASE_REF="$(bash "$GITHUB_WORKSPACE/.github/scripts/resolve-i18n-base.sh" \
"$I18N_BASE_REF" 'the diff-scoped render gate')" || exit 1
printf 'I18N_BASE_REF=%s\n' "$I18N_BASE_REF" >> "$GITHUB_ENV"
# Every step below runs as `runner` through ci-shell (the job default).
# Nothing after this point writes a GitHub file command; the i18n base ref
# above is the last one.
- uses: ./.github/actions/run-as-runner
# The SkillForm block-scalar parity guard (issue #7763). It runs the REAL
# TypeScript mirror (backendReadsValue -> backendFoldsLiteral, transpiled
# with the website esbuild) against the Python read path over a generated
# corpus, so the editor's simulation of the reader cannot drift from the
# reader. It lives HERE because it is the only pull-request lane with BOTH
# runtimes present AND usable: the backend is pip-installed above (Python +
# kiro_crew) and `npm ci` has populated website/node_modules (node + the
# `yaml` package the mirror imports + esbuild). The sharded backend-test job
# has Python but no website node deps, and frontend-test has node but no
# Python, so in each of those the file skips with a named reason -- which is
# why REQUIRE=1 is set only here, turning any skip into a failure so the
# guard cannot report green having run nothing. Anchored pass count, the same
# canary idiom as the boot matrices above: pytest exits 0 on a fully skipped
# module, so the count is what proves the corpus actually ran.
- name: SkillForm frontmatter parity (TS mirror vs Python read path)
env:
KIROCREW_SKILLFORM_PARITY_REQUIRE: "1"
run: |
set -o pipefail
# Order-safe: the harness file lands in a separate PR. Until it exists
# this step is a no-op, so merging this workflow change first cannot red
# the lane on a tree without the harness. Once the file is present the
# REQUIRE variable above turns any skip into a failure.
if [ ! -f test/test_skillform_frontmatter_parity.py ]; then
echo "::notice::SkillForm parity harness not present yet; step is a no-op until it lands."
exit 0
fi
echo "::remove-matcher owner=python::"
python -m pytest -v -n0 --no-cov -p no:cacheprovider --timeout=300 \
test/test_skillform_frontmatter_parity.py \
| tee skillform-parity.log
# 880 = 110 body shapes x (3 bare-| headers + 5 refused headers). Raise
# this when the corpus grows; a collapse to a handful of cases (a broken
# transpile that skips, a renamed symbol) fails here instead of passing
# green. Anchored so "1880 passed" cannot satisfy "880 passed".
grep -qE '(^|[^0-9])880 passed' skillform-parity.log || {
echo "::error::the SkillForm frontmatter parity corpus did not report 880 passed (skipped, collapsed, or the corpus size changed without updating this canary)"
exit 1
}
- name: Run E2E and dedicated memory UI evidence in parallel
# Smoke and UI read the staged production bundle; i18n builds a separate
# dist-dev and a temporary base tree. All three share only read-only
# dependency/browser installs, with separate homes, ports and outputs.
# The helper awaits ALL outcomes even if one fails, preserves the UI's
# 12-minute cap, and drains adopted gateway/browser children on cancel.
# Overlap head+base i18n rendering rather than consume ~8 minutes of the
# fixed 25-minute job budget before the ordinary E2E lane can start.
# KIROCREW_E2E=1 is set by setup.py for smoke and only in the UI child
# for evidence; REQUIRE turns missing prerequisites into hard failures.
env:
KIROCREW_E2E_REQUIRE: "1"
VOICE_EVIDENCE_HEAD: ${{ github.event.pull_request.head.sha || github.sha }}
KIROCREW_STRICT_ON_LOOP_PERSIST: "1"
KIROCREW_MEMORY_UI_EVIDENCE_DIR: ${{ runner.temp }}/memory-embedding-evidence
run: exec python scripts/ci_e2e_parallel.py
- name: Upload read-aloud recovery evidence
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: voice-recovery-evidence
path: website/test-results/**/voice-recovery/
if-no-files-found: ignore
retention-days: 7
- name: Upload member memory UI evidence
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: member-memory-ui-evidence
path: |
website/test-results/**/member-memory-*.png
website/test-results/**/legacy-member-memory-*.png
website/test-results/**/member-memory-walkthrough.webm
website/test-results/**/member-memory-evidence.json
website/test-results/**/memory-v1-*.png
website/test-results/**/memory-v2-*.png
${{ runner.temp }}/memory-embedding-evidence/**/memory-v2-embedding-*.png
${{ runner.temp }}/memory-embedding-evidence/**/memory-v2-embedding-evidence.json
if-no-files-found: ignore
retention-days: 7
# The specs' own failure record. Playwright writes one directory per
# failed attempt under website/test-results/ -- the ARIA snapshot of the
# page at the failing assertion (error-context.md), the on-first-retry
# trace, and any screenshot -- and the html report next to it. Until this
# step existed none of it left the runner: #8526 was triaged for hours from
# a log line that named `error-context.md` and nothing else. Failure-only,
# because the successful recovery screenshots are uploaded above and
# the job already uploads coverage. `ignore` rather than `error`: a run
# that failed before the specs started (a browser install that stalled)
# has no results, and this step must not turn that into a second failure.
- name: Upload Playwright failure artifacts
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-playwright-failures
path: |
website/test-results/
website/playwright-report/
if-no-files-found: ignore
retention-days: 7
# The one E2E step the fleet cannot host. test_private_workflow_memory.py boots
# a real gateway and runs a Crew Member's private workflow MCP inside the
# member sandbox, which needs an unprivileged user namespace
# (`unshare --mount --map-root-user`); the CodeBuild container answers EPERM.
# Keeping it separate limits hosted runner use to its ~3.5-minute run. No SPA,
# no Node, no browser: the test drives the gateway over HTTP, like the boot matrix.
#
# `--timeout=600` is a BACKSTOP, not the test's budget. The module caps all of a
# test's waiting at `_TEST_BUDGET_SECONDS` (570) anchored at test start, so a
# stalled workflow fails inside `_finished` -- which names the run and its
# progress -- while this cap only catches a hang outside those waits. Two
# gateway boots and nine workflow waits need more room than the ~3.5-minute
# typical run, and a cap at that scale fires on runner slowness while saying
# nothing about which workflow hung. Keep this above
# `_TEST_BUDGET_SECONDS` and below `timeout-minutes`;
# `test_e2e_private_namespace_budget.py` enforces both.
e2e-private-namespace:
name: E2E (private member namespace, hosted)
needs: [await-fast-gate]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
- name: Enable private member namespace isolation
run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
- name: Verify private member namespace isolation
run: unshare --mount --map-root-user true
- name: Require real private workflow MCP execution
env:
KIROCREW_E2E: "1"
KIROCREW_E2E_REQUIRE: "1"
KIROCREW_STRICT_ON_LOOP_PERSIST: "1"
run: |
echo "::remove-matcher owner=python::"
python -m pytest -q -n0 --no-cov --timeout=600 test/e2e/test_private_workflow_memory.py
# The integration layer: the real gateway booted IN the pytest process.
#
# `test/integration/` sits between the unit shards (bare `web.Application()`
# shells, everything mocked) and the E2E jobs above (a `kirocrew gateway`
# subprocess). Each test boots `GatewayOrchestrator.run()` on a scratch
# `KIROCREW_HOME` with the packaged fake ACP backend, then talks HTTP to the
# port it bound -- same interpreter, same event loop. That is what lets a test
# here assert on a restart (second boot, same home), on an event-loop stall
# seen from a second request, or on a policy file the boot itself wrote.
#
# Opt-in by `KIROCREW_INTEGRATION=1`; the unit shards collect and skip it.
# Linux only: the fixture snapshots and restores the SIGINT/SIGTERM handlers
# `run()` installs, which is a POSIX dance the Windows shards must not pay.
#
# The gate is ROUTE coverage, not line coverage: the conftest records which
# registered dashboard routes the suite requested, and
# `scripts/check_integration_route_coverage.py` reports the share. `--min` is
# a ratchet -- raise it a little under what main measures, never above.
integration:
name: Integration (in-process gateway)
needs: [changes, await-fast-gate]
runs-on: ${{ needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }}
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
run: uv pip install --system -e ".[voice,desktop]" --group dev
- uses: ./.github/actions/run-as-runner
- name: Run the integration layer
shell: /usr/local/bin/ci-shell {0}
env:
KIROCREW_INTEGRATION: "1"
KIROCREW_INTEGRATION_HITS_DIR: ${{ runner.temp }}/integration-hits
run: |
echo "::remove-matcher owner=python::"
python -m pytest -q -n 2 --dist loadgroup --max-worker-restart=2 \
--timeout=300 --no-cov test/integration
- name: Route coverage
shell: /usr/local/bin/ci-shell {0}
env:
HITS_DIR: ${{ runner.temp }}/integration-hits
run: |
python3 scripts/check_integration_route_coverage.py "$HITS_DIR" --missing --min 95 \
| tee "$HITS_DIR/route-coverage.txt"
- name: Upload route-coverage report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: integration-route-coverage
path: ${{ runner.temp }}/integration-hits/
if-no-files-found: ignore
retention-days: 7
# The only job that BOOTS A GATEWAY on macOS and Windows.
#
# Everything else on those two runners is unit tests. `e2e` above spawns a real
# gateway, but it is ubuntu-latest only, because `KIROCREW_E2E` is set solely by
# `setup.py test_e2e`. That was one of the two holes #8117 fell through: it
# added a settings-file probe to `wrap_argv`'s Windows delegation branch, so on
# a fresh Windows host -- where that file does not exist -- the Kiro ACP spawn
# stopped delegating, took the no-backend fail-closed path, and the gateway
# never became usable. The unit test that pinned that branch lives in
# `test_sandbox_argv.py`, which `windows-collect-ignore.txt` excludes, and the
# PR itself changed that test's mock to hardcode the one answer a fresh Windows
# host cannot give. Neither hole is closable by another unit test; only a real
# boot on the real platform is.
#
# Offline and credential-less by construction: the agent binary is the packaged
# fake ACP backend, wired through `KIROCREW_KIRO_BIN`, so there is no model, no
# token, no network and no cost. See docs/ci/e2e-gate.md.
e2e-boot-matrix:
name: E2E Gateway Boot (${{ matrix.os }})
needs: [changes, await-fast-gate]
strategy:
# Each OS is its own verdict. A Windows-only regression must not hide
# behind a cancelled macOS leg, which is exactly the shape of the escape
# this job exists for.
fail-fast: false
matrix:
# macos-15 is on the push-to-main path only. It is the same queue story as
# the moved pytest lane: the mac leg waited ~200 minutes for a runner on
# every pull request, and it is the only leg that did. On main the wait
# costs nobody a merge, and nightly's `pod-scenarios` boots a real
# service-managed gateway on macos-15 as well, so the real-Darwin boot
# remains covered twice -- just never in front of a pull request.
os: ${{ fromJSON(github.event_name == 'pull_request'
&& '["ubuntu-latest","windows-latest"]'
|| '["ubuntu-latest","macos-15","windows-latest"]') }}
# Keep matrix.os stable: names, readiness windows and artifact names use it.
runs-on: ${{ matrix.os == 'ubuntu-latest' && (needs.changes.outputs.linux_runner_large || 'ubuntu-latest') || matrix.os == 'windows-latest' && (needs.changes.outputs.windows_runner || 'windows-latest') || matrix.os }}
timeout-minutes: 20
defaults:
run:
shell: ${{ matrix.os == 'ubuntu-latest' && '/usr/local/bin/ci-shell {0}' || 'bash' }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-windows-tests
if: runner.os == 'Windows'
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
enable-cache: true
# Same no-uv.lock caveat as backend-test: key the cache on the files
# that define the dependency set.
cache-dependency-glob: |
pyproject.toml
setup.cfg
- name: Install dependencies
shell: bash
run: uv pip install --system -e ".[voice,desktop]" --group dev
- uses: ./.github/actions/run-as-runner
if: runner.os == 'Linux'
- name: Verify Windows test prerequisites
if: runner.os == 'Windows' && runner.environment == 'self-hosted'
shell: bash
run: python .github/actions/setup-windows-tests/probe.py
- name: Boot a gateway end to end and assert the pass count
# Linux uses the same non-root boundary as the backend shards.
# Windows/macOS retain bash and the exact same test command.
env:
# Lifts the module skip.
KIROCREW_E2E: "1"
# Turns an unmet PRECONDITION (the packaged fake backend missing, the
# testing package not importable) from a graceful skip into a hard
# failure. Without it the job could report green having booted zero
# gateways, because pytest counts a skip as a pass. Same mechanism as
# KIROCREW_E2E_REQUIRE on the `e2e` job.
KIROCREW_E2E_REQUIRE: "1"
# Readiness window per OS, set here rather than in the test so a slow
# runner is retunable without a code change. Windows gets the widest:
# subprocess spawn and filesystem latency there are measurably slower,
# the conditions #9172 addressed when a slow disk killed the gateway.
KIROCREW_HARNESS_READY_TIMEOUT: ${{ matrix.os == 'windows-latest' && '180' || matrix.os == 'macos-15' && '90' || '60' }}
run: |
set -o pipefail
# Same as backend-test (#7296): the `python` matcher annotates any
# traceback in the log, including one printed inside a warning, and
# never matches a pytest failure.
echo "::remove-matcher owner=python::"
# -n0: this module spawns a real gateway per test, and under xdist a
# block would take the worker down with it, which on Windows aborts the
# whole run. --timeout is above the widest readiness window (180s) plus
# the per-turn reply ceiling, so a stuck turn fails by name instead of
# killing the run.
pytest -v -n0 --no-cov -p no:cacheprovider --timeout=420 \
test/e2e/test_gateway_boot_matrix.py \
| tee boot-matrix.log
# The canary. `pytest` exits 0 on a fully skipped module, so the exit
# code alone cannot tell "seven gateways booted" from "the module was
# never collected". Grepping the exact count is what makes this job
# honest; raise the number when you add a test here.
# Anchored: see the pod-boot canary above.
grep -qE '(^|[^0-9])7 passed' boot-matrix.log || {
echo "::error::gateway boot matrix did not report 7 passed (skipped, or a test was added without updating this canary)"
exit 1
}
- name: Upload the boot-matrix log
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-boot-matrix-${{ matrix.os }}
path: boot-matrix.log
if-no-files-found: ignore
retention-days: 7