Skip to content

ci: record runner pressure during the e2e suite - #36628

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/22314-e2e-host-pressure
Nov 17, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/22314-e2e-host-pressure

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When the offline E2E job fails, it uploads a 5-second record of runner pressure, steal and load, so a gateway killed by its own loop-stall watchdog (exit -14) can be pinned on either the runner or the gateway.

Part of #22314

E2E (stub ACP backend, offline) sometimes fails because the gateway that test/test_playwright_e2e.py drives is killed by its own loop-stall watchdog (SIGALRM, exit -14), during the member-memory "V2 previews and atomically edits 65 email memories" spec. It has been seen ten times on unrelated PRs, all on fleet runners. The stack dumps show an idle event loop, idle pool workers, and a few threads in filesystem calls. For about 23 s before the exit nothing wrote a log line, not even the watchdog's own daemon thread. So the process was either not being scheduled, or one thread held the GIL. The failure output today cannot tell those two cases apart: the job records nothing about the host.

Why it matters

The flake reds unrelated PRs and has not reproduced locally in four tries. Without host-state data from the next sighting, each one is triaged from a stack dump that looks the same whether the runner froze or the gateway did.

Not a goal

  • Fixing the stall, or changing any product code, test harness (src/kiro_crew/testing/, conftest.py), spec, retry, timeout, ceiling or skip.
  • Adding the sampler to any job other than e2e, or putting its log into the existing e2e-playwright-failures artifact.

What changed (motivation → approach → change)

Symptom: a -14 exit with no record of the host's state → cause: the job samples no pressure, steal or cgroup counters → change, all in the e2e job of .github/workflows/ci.yml:

  • Start the host pressure sampler, right before Run E2E and dedicated memory UI evidence in parallel and after run-as-runner, so it runs through the job's default ci-shell and sets no shell:. It starts a background subshell that, every 5 s, writes a UTC timestamp, /proc/pressure/{cpu,io,memory} (whole files, total= included), /proc/loadavg, the cgroup v2 cpu.stat and memory.events for the cgroup in /proc/self/cgroup, and the cpu line of /proc/stat (steal). The loop runs with set +e, and each failed read writes read failed: <path> to the log instead of ending the loop. stdin is /dev/null and all output goes to $RUNNER_TEMP/e2e-host-pressure.log, so the step returns at once. The PID is saved to $RUNNER_TEMP/e2e-host-pressure.pid. A short comment gives the reason and the caveat: /proc/pressure and /proc/stat are host-wide inside the container, and a gap in the log's own timestamps means the sampler was frozen too.
  • Stop the host pressure sampler, right after the E2E step, runs whenever the sampler started: if: ${{ always() && steps.host-pressure-sampler.outcome == 'success' }}, the start step's id. A job that fails before run-as-runner has no ci-shell yet, so an ungated run step there would fail too. It kills only the saved PID (no pattern match) and passes when the process is already gone.
  • Upload host pressure samples, if: failure(), at the end of the job. It uploads ${{ runner.temp }}/e2e-host-pressure.log as its own e2e-host-pressure artifact (if-no-files-found: ignore, retention-days: 7), with the same pinned actions/upload-artifact line the job already uses.
  • docs/ci/e2e-gate.md: one paragraph after the e2e-playwright-failures paragraph that names the new artifact, what it records, and the caveat. That paragraph no longer calls the Playwright upload the final step.

Backwards compatibility

Compatible: three steps are added to one CI job. The sampler only reads files and writes inside $RUNNER_TEMP, every step tolerates failure, and the existing artifacts and their paths are unchanged.

Removes nothing: this diff only adds steps and a doc paragraph.

Tests

No test added. The decision on #22314 limits this change to the workflow and its doc. The existing pins on this job pass with the new steps: test/test_ci_nonroot_boundary.py (no shell: after run-as-runner, no file commands after it) and test/test_e2e_ci_scheduling.py, along with the ci.yml-reading tests local-gate.py selects.

Proof: not provable -- docs or config only

Manual verification

Host: Ubuntu 24.04.5 LTS (x86_64, Linux 7.0), non-root user.

The flake was not reproduced. The -14 kill was induced: a harness gateway (spawn_feature_gateway("minimal"), isolated home) was frozen with SIGSTOP for 35 s, past its 25 s watchdog exit budget, and then sent SIGCONT:

SIGSTOP sent at 12:25:42Z; freezing 35 s (> 25 s exit budget)
SIGCONT sent
returncode=-14 after 35.0s
gateway pid … exit=-14
'pressure': absent   'steal': absent   'loadavg': absent   'cpu.stat': absent

The stack dump has the same shape as the CI sightings (idle pool workers, a thread in pathlib.stat), and nothing in it or in the base ci.yml records host state.

Smoke test of the change: the sampler and stop run: texts were taken from the modified ci.yml by parsing the YAML, then run under bash -eo pipefail with RUNNER_TEMP set to a temp dir. CPU, disk and memory load ran for about 60 s:

step order: [('Start the host pressure sampler', 16), ('Run E2E and dedicated memory UI evidence in parallel', 17), ('Stop the host pressure sampler', 18), ('Upload host pressure samples', 22)] strictly increasing: True
run-as-runner index: 13
sampler sets shell: False | stop sets shell: False | sampler id: host-pressure-sampler | stop if: ${{ always() && steps.host-pressure-sampler.outcome == 'success' }} | upload if: failure()
start step exit=0, returned in .002614540 s
piped start returned in .003151729 s     (fds: 0 -> /dev/null, 1/2 -> e2e-host-pressure.log)
stop step exit=0
saved PID … gone after 1s
stop again (already gone): host pressure sampler already gone / exit=0
stop with no pid file: exit=0
samples in log: 13   (12:27:32.231Z … 12:28:32.610Z, 5 s apart)
cpu: some avg10 max=11.83
io: some avg10 max=12.33
--- /proc/stat
cpu  5141632 373 431487 20861078 153003 0 1320 133 0 0

To exercise missing cgroup files, the same text was run with /sys/fs/cgroup hidden behind an empty tmpfs in a user and mount namespace. Every sample logged the failed reads, and the loop kept sampling until the stop step ended it:

=== 2026-11-13T12:28:45.438Z
cat: /sys/fs/cgroup/…/cpu.stat: No such file or directory
read failed: /sys/fs/cgroup/…/cpu.stat
read failed: /sys/fs/cgroup/…/memory.events
=== 2026-11-13T12:28:50.447Z
… (4 samples, then stop-exit=0, PID gone)

The parse line above (step order, shells, id, if:) is read from the current head, and the stop step's gate names the start step, which comes earlier in the same job. The three steps' run: and with: texts are byte-identical to the ones the timings and samples above came from. The upload step's path is ${{ runner.temp }}/e2e-host-pressure.log, the file the sampler writes to $RUNNER_TEMP. actionlint 1.7.7 with shellcheck reports the same three findings on base and on a head with these run: texts, all at other jobs' lines; none are new.

Related Issues

Part of #22314

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c9e213886f4811e351d881df35f380dffed9b0ac and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c9e2138

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c9e213886f4811e351d881df35f380dffed9b0ac false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Monitor pass on head 85b248265292: every red on this head is either main's or an unpublished verdict, so no code changed and nothing was pushed.

Backend Tests (3.12, 7) and Coverage Gate are main's red, not this PR's. The shard reported peak memory in use: 15059 MiB of 15738 MiB and two nodes fell over in it: test_the_helper_modules_import_without_pytest timed out its import subprocess (#36407, fix open in #36575) and worker 'gw3' crashed on an unrelated node (#36592). Both are recorded with this run's URL, OS and failing line. The same shard was red on main's own push runs in the same window — 43458639790 (the runner lost communication with the server) and 43449942603 (test_ssrf_guard_fail_closed node timeouts, #36261, fix open in #36653 and #36655) — and this diff adds three steps to the e2e job plus a doc paragraph, touching nothing those nodes import. Coverage Gate fails closed on the backend shard, so it clears with it. Per the flake rule, no rerun was spent on a test with a fix already in flight: this PR waits for #36575 and #36653 to land, then rebases onto them.

The four review lanes published no verdict for this head because the shared API budget was exhausted when they ran, not because of the diff: Publish readiness signal stopped at pr-readiness: core rate limit -- 0/15000 remaining, Design and First Principles logged "did not produce a verdict ... (errored / overloaded / incomplete — NOT blocking)", Opus 5.5 Review failed closed on outcome=skipped, and UX logged "skipped ... (no UI-relevant changes — non-blocking)" and then failed in its override step. The budget has recovered, so those lanes and the readiness signal are being re-run on this same head; GPT 6.1 already reviewed it with no blocking findings.

Base: the PR merges cleanly (MERGEABLE) and main does not require branches to be up to date, so the base stays frozen and no re-sync comment was posted.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Correction to the note above: the four lane re-runs have not started. The shared Actions capacity is at its cap right now (36 branches with runs queued or in progress, against a ceiling of 25), so nothing new was started on this head and no run was re-run. The lanes that published no verdict for 85b248265292 stay unanswered until either capacity frees, or #36575 and #36653 land and this branch rebases onto them — a rebase re-runs every lane on the new head anyway, which is the cheaper of the two.

Nothing else changed on this pass: no code, no push, no body edit, no override.

The offline E2E job intermittently loses its gateway to the gateway's
own loop-stall watchdog (exit -14), and the failure output cannot tell
a runner that stopped scheduling the process from a stall inside it:
the job records nothing about the host.

Start a background sampler before the E2E step that logs, every 5 s, a
UTC timestamp, /proc/pressure/{cpu,io,memory}, /proc/loadavg, the
cgroup's cpu.stat and memory.events, and the cpu line of /proc/stat.
Failed reads are logged, never fatal. Stop it by its saved PID after
the step, and on failure upload the log as its own e2e-host-pressure
artifact. Document the artifact and its caveat in docs/ci/e2e-gate.md.

Part of #22314

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/ai-review-human-override.yml, .github/workflows/build.yml, .github/workflows/ci.yml, .github/workflows/claude-review.yml, .github/workflows/codex-review.yml, .github/workflows/design-review.yml, .github/workflows/fast-gate.yml, .github/workflows/first-principles-review.yml, .github/workflows/fork-design-review.yml, .github/workflows/fork-first-principles-review.yml, .github/workflows/fork-gpt-review.yml, .github/workflows/fork-opus-review.yml, .github/workflows/fork-ux-review.yml, .github/workflows/internal-content-scan-gate.yml, .github/workflows/macos-on-demand.yml, .github/workflows/nightly.yml, .github/workflows/ota-test.yml, .github/workflows/platform-tests.yml, .github/workflows/pr-hygiene.yml, .github/workflows/pr-readiness-sweep.yml, .github/workflows/pr-readiness.yml, .github/workflows/pr-size-override.yml, .github/workflows/publish-cli.yml, .github/workflows/publish-docker.yml, .github/workflows/publish-linux.yml, .github/workflows/publish-windows.yml, .github/workflows/release.yml, .github/workflows/sign-and-notarize.yml, .github/workflows/ux-review.yml, docs/ci/ci-and-reviews.md, docs/ci/e2e-gate.md, docs/ci/harness-parity-gate.md, docs/ci/i18n-gates.md, src/kiro_crew/apps/builtins/issue_radar/tests/test_ci_watch.py, test/test_ci_fleet_routing_expression_parity.py, test/test_ci_linux_packaging_filter.py and 12 test/test_workflows_*.py files since this head went green (c22557c -> a832580); CI re-running.

Two of those 1100 commits are what the old head's reds were waiting on: #36575 (the shard-7 import timeout that red Backend Tests (3.12, 7), and with it Coverage Gate) and #36461 (the same-repo AI review lanes now wait out a spent rate limit, which is why Design, First Principles, Opus 5.5, UX and the publish-readiness signal published no verdict for the old head).

One change to the PR itself, from a pre-push review: Stop the host pressure sampler now runs only when the start step succeeded (if: ${{ always() && steps.host-pressure-sampler.outcome == 'success' }}, with id: host-pressure-sampler on the start step). A job that fails before run-as-runner has no ci-shell yet, so the old always() gate would have added a second red step there. Otherwise the rebased e2e job of .github/workflows/ci.yml equals main's plus the three added steps, every other job in that file equals main's, and the docs/ci/e2e-gate.md hunk is the old head's.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/22314-e2e-host-pressure branch from 85b2482 to c9e2138 Compare November 17, 2026 18:15
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c9e213886f4811e351d881df35f380dffed9b0ac touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c9e213886f4811e351d881df35f380dffed9b0ac — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this adds read-only diagnostics to one CI job, every step tolerates failure, nothing is taken away, and it can be reverted cleanly.

Notes

  • The smoke test ran the sampler under plain bash, not through /usr/local/bin/ci-shell. If ci-shell kills the background subshell when the step ends, the artifact comes out empty. On the first real run, check that the log has more than one sample.

[DESIGN-REVIEWED] c9e2138

Description read

[DESCRIPTION-READ] 9370fd3eabc5d156a9cbaf0093bfec4d0234c5a0169db50f72647405f6ca8a42

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of c9e213886f4811e351d881df35f380dffed9b0ac — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The PR adds three CI-only diagnostic steps and one doc paragraph. No defaults change, and nothing else in the repo samples host state from outside the gateway.

What this change ships

Inventory (4 items) — 4 justified

Intent: when the offline E2E job fails with the watchdog's -14 exit, maintainers can tell whether the runner stopped scheduling the gateway or the gateway stalled itself. ADDITION (diagnostics only).

  1. The E2E job now samples PSI, loadavg, steal and cgroup counters every 5 s in the background while the suite runs — justified
  2. That sampler is stopped after the suite, but only if it started — justified
  3. A failed job uploads a new e2e-host-pressure artifact (7-day retention) — justified
  4. docs/ci/e2e-gate.md names the new artifact and no longer calls the Playwright upload "final" — justified

Existing-mechanism check: src/kiro_crew/diag/recorder.py already records loadavg, but it runs inside the gateway process, so it freezes along with the process it would need to explain. It also reads no PSI, steal or cgroup counters, so the sampler is not a second copy of it. A grep of .github/ and scripts/ for /proc/pressure|/proc/stat|cpu.stat|memory.events|vmstat|sampler found 0 matches outside this diff.

[FIRST-PRINCIPLES-REVIEWED] c9e2138

Description read

[DESCRIPTION-READ] 10d84015e5df0fe5e980ddb65c0879335724c8b09bace10c3d8703b9dcd8cddb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c9e213886f4811e351d881df35f380dffed9b0ac — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c9e2138

Verdict parsed from the review's SHA-scoped output markers for commit c9e213886f4811e351d881df35f380dffed9b0ac.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c9e213886f4811e351d881df35f380dffed9b0ac false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: reviewer-lanes-untrusted.

Adds a background host-pressure sampler to the e2e job (start step after run-as-runner, gated stop step after the E2E step, failure-only upload) plus one doc paragraph. ci-shell just execs runuser→bash (or bash), so the backgrounded subshell, which has redirected stdio, survives the step and doesn't hold the runner's pipes. Start and stop both run as runner, and $RUNNER_TEMP is 1777, so the kill and file writes work. Stop is gated on the start step's outcome and can't fail. Upload uses if-no-files-found: ignore. Existing nonroot-boundary pins still hold (no shell: after run-as-runner, no file commands), and the e2e leak inventory is scoped to the instance's slice, so the sampler isn't seen as a leak. No regressions or trust-boundary changes; the Dependency License Gate failure is an unrelated infrastructure issue.

Tier: 2, held for a person: reviewer-lanes-untrusted.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit c9e213886f · 54 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head c9e2138, which adds a background host pressure sampler, a gated stop step and a failure-only artifact upload to the offline E2E job. The sampler runs after the unprivileged boundary, holds none of the step's output pipes and writes only inside the runner temp dir. The stop step runs only when the sampler started and kills just the saved PID. I parsed the start and stop scripts out of the workflow at this head and ran them under the job's strict bash flags: they started at once, logged samples at the expected interval and stopped cleanly. The existing workflow tests that read ci.yml also pass, so I found no bug that blocks this change.

@KiroCrewCaptain
KiroCrewCaptain merged commit bddefa3 into main Nov 17, 2026
96 of 98 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/22314-e2e-host-pressure branch November 17, 2026 19:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants