Repository navigation
ci: record runner pressure during the e2e suite - #36628
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
[operator: bolichen97] Monitor pass on head
The four review lanes published no verdict for this head because the shared API budget was exhausted when they ran, not because of the diff: Base: the PR merges cleanly ( |
|
[operator: bolichen97] Correction to the note above: the four lane re-runs have not started. The shared Actions capacity is at its cap right now (36 branches with runs queued or in progress, against a ceiling of 25), so nothing new was started on this head and no run was re-run. The lanes that published no verdict for Nothing else changed on this pass: no code, no push, no body edit, no override. |
The offline E2E job intermittently loses its gateway to the gateway's
own loop-stall watchdog (exit -14), and the failure output cannot tell
a runner that stopped scheduling the process from a stall inside it:
the job records nothing about the host.
Start a background sampler before the E2E step that logs, every 5 s, a
UTC timestamp, /proc/pressure/{cpu,io,memory}, /proc/loadavg, the
cgroup's cpu.stat and memory.events, and the cpu line of /proc/stat.
Failed reads are logged, never fatal. Stop it by its saved PID after
the step, and on failure upload the log as its own e2e-host-pressure
artifact. Document the artifact and its caveat in docs/ci/e2e-gate.md.
Part of #22314
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rebased: main moved in .github/workflows/ai-review-human-override.yml, .github/workflows/build.yml, .github/workflows/ci.yml, .github/workflows/claude-review.yml, .github/workflows/codex-review.yml, .github/workflows/design-review.yml, .github/workflows/fast-gate.yml, .github/workflows/first-principles-review.yml, .github/workflows/fork-design-review.yml, .github/workflows/fork-first-principles-review.yml, .github/workflows/fork-gpt-review.yml, .github/workflows/fork-opus-review.yml, .github/workflows/fork-ux-review.yml, .github/workflows/internal-content-scan-gate.yml, .github/workflows/macos-on-demand.yml, .github/workflows/nightly.yml, .github/workflows/ota-test.yml, .github/workflows/platform-tests.yml, .github/workflows/pr-hygiene.yml, .github/workflows/pr-readiness-sweep.yml, .github/workflows/pr-readiness.yml, .github/workflows/pr-size-override.yml, .github/workflows/publish-cli.yml, .github/workflows/publish-docker.yml, .github/workflows/publish-linux.yml, .github/workflows/publish-windows.yml, .github/workflows/release.yml, .github/workflows/sign-and-notarize.yml, .github/workflows/ux-review.yml, docs/ci/ci-and-reviews.md, docs/ci/e2e-gate.md, docs/ci/harness-parity-gate.md, docs/ci/i18n-gates.md, src/kiro_crew/apps/builtins/issue_radar/tests/test_ci_watch.py, test/test_ci_fleet_routing_expression_parity.py, test/test_ci_linux_packaging_filter.py and 12 test/test_workflows_*.py files since this head went green (c22557c -> a832580); CI re-running. Two of those 1100 commits are what the old head's reds were waiting on: #36575 (the shard-7 import timeout that red One change to the PR itself, from a pre-push review: [operator: bolichen97] |
85b2482 to
c9e2138
Compare
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this adds read-only diagnostics to one CI job, every step tolerates failure, nothing is taken away, and it can be reverted cleanly. Notes
[DESIGN-REVIEWED] c9e2138 Description read[DESCRIPTION-READ] 9370fd3eabc5d156a9cbaf0093bfec4d0234c5a0169db50f72647405f6ca8a42 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The PR adds three CI-only diagnostic steps and one doc paragraph. No defaults change, and nothing else in the repo samples host state from outside the gateway. What this change shipsInventory (4 items) — 4 justifiedIntent: when the offline E2E job fails with the watchdog's -14 exit, maintainers can tell whether the runner stopped scheduling the gateway or the gateway stalled itself. ADDITION (diagnostics only).
Existing-mechanism check: [FIRST-PRINCIPLES-REVIEWED] c9e2138 Description read[DESCRIPTION-READ] 10d84015e5df0fe5e980ddb65c0879335724c8b09bace10c3d8703b9dcd8cddb That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: reviewer-lanes-untrusted. Adds a background host-pressure sampler to the e2e job (start step after run-as-runner, gated stop step after the E2E step, failure-only upload) plus one doc paragraph. ci-shell just execs runuser→bash (or bash), so the backgrounded subshell, which has redirected stdio, survives the step and doesn't hold the runner's pipes. Start and stop both run as Tier: 2, held for a person: reviewer-lanes-untrusted. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head c9e2138, which adds a background host pressure sampler, a gated stop step and a failure-only artifact upload to the offline E2E job. The sampler runs after the unprivileged boundary, holds none of the step's output pipes and writes only inside the runner temp dir. The stop step runs only when the sampler started and kills just the saved PID. I parsed the start and stop scripts out of the workflow at this head and ran them under the job's strict bash flags: they started at once, logged samples at the expected interval and stopped cleanly. The existing workflow tests that read ci.yml also pass, so I found no bug that blocks this change.
Problem / Motivation
Goal: When the offline E2E job fails, it uploads a 5-second record of runner pressure, steal and load, so a gateway killed by its own loop-stall watchdog (exit -14) can be pinned on either the runner or the gateway.
Part of #22314
E2E (stub ACP backend, offline)sometimes fails because the gateway thattest/test_playwright_e2e.pydrives is killed by its own loop-stall watchdog (SIGALRM, exit -14), during the member-memory "V2 previews and atomically edits 65 email memories" spec. It has been seen ten times on unrelated PRs, all on fleet runners. The stack dumps show an idle event loop, idle pool workers, and a few threads in filesystem calls. For about 23 s before the exit nothing wrote a log line, not even the watchdog's own daemon thread. So the process was either not being scheduled, or one thread held the GIL. The failure output today cannot tell those two cases apart: the job records nothing about the host.Why it matters
The flake reds unrelated PRs and has not reproduced locally in four tries. Without host-state data from the next sighting, each one is triaged from a stack dump that looks the same whether the runner froze or the gateway did.
Not a goal
src/kiro_crew/testing/,conftest.py), spec, retry, timeout, ceiling or skip.e2e, or putting its log into the existinge2e-playwright-failuresartifact.What changed (motivation → approach → change)
Symptom: a -14 exit with no record of the host's state → cause: the job samples no pressure, steal or cgroup counters → change, all in the
e2ejob of.github/workflows/ci.yml:Start the host pressure sampler, right beforeRun E2E and dedicated memory UI evidence in paralleland afterrun-as-runner, so it runs through the job's default ci-shell and sets noshell:. It starts a background subshell that, every 5 s, writes a UTC timestamp,/proc/pressure/{cpu,io,memory}(whole files,total=included),/proc/loadavg, the cgroup v2cpu.statandmemory.eventsfor the cgroup in/proc/self/cgroup, and thecpuline of/proc/stat(steal). The loop runs withset +e, and each failed read writesread failed: <path>to the log instead of ending the loop. stdin is/dev/nulland all output goes to$RUNNER_TEMP/e2e-host-pressure.log, so the step returns at once. The PID is saved to$RUNNER_TEMP/e2e-host-pressure.pid. A short comment gives the reason and the caveat:/proc/pressureand/proc/statare host-wide inside the container, and a gap in the log's own timestamps means the sampler was frozen too.Stop the host pressure sampler, right after the E2E step, runs whenever the sampler started:if: ${{ always() && steps.host-pressure-sampler.outcome == 'success' }}, the start step'sid. A job that fails beforerun-as-runnerhas no ci-shell yet, so an ungated run step there would fail too. It kills only the saved PID (no pattern match) and passes when the process is already gone.Upload host pressure samples,if: failure(), at the end of the job. It uploads${{ runner.temp }}/e2e-host-pressure.logas its owne2e-host-pressureartifact (if-no-files-found: ignore,retention-days: 7), with the same pinnedactions/upload-artifactline the job already uses.docs/ci/e2e-gate.md: one paragraph after thee2e-playwright-failuresparagraph that names the new artifact, what it records, and the caveat. That paragraph no longer calls the Playwright upload the final step.Backwards compatibility
Compatible: three steps are added to one CI job. The sampler only reads files and writes inside
$RUNNER_TEMP, every step tolerates failure, and the existing artifacts and their paths are unchanged.Removes nothing: this diff only adds steps and a doc paragraph.
Tests
No test added. The decision on #22314 limits this change to the workflow and its doc. The existing pins on this job pass with the new steps:
test/test_ci_nonroot_boundary.py(noshell:afterrun-as-runner, no file commands after it) andtest/test_e2e_ci_scheduling.py, along with the ci.yml-reading testslocal-gate.pyselects.Proof: not provable -- docs or config only
Manual verification
Host: Ubuntu 24.04.5 LTS (x86_64, Linux 7.0), non-root user.
The flake was not reproduced. The -14 kill was induced: a harness gateway (
spawn_feature_gateway("minimal"), isolated home) was frozen withSIGSTOPfor 35 s, past its 25 s watchdog exit budget, and then sentSIGCONT:The stack dump has the same shape as the CI sightings (idle pool workers, a thread in
pathlib.stat), and nothing in it or in the baseci.ymlrecords host state.Smoke test of the change: the sampler and stop
run:texts were taken from the modifiedci.ymlby parsing the YAML, then run underbash -eo pipefailwithRUNNER_TEMPset to a temp dir. CPU, disk and memory load ran for about 60 s:To exercise missing cgroup files, the same text was run with
/sys/fs/cgrouphidden behind an empty tmpfs in a user and mount namespace. Every sample logged the failed reads, and the loop kept sampling until the stop step ended it:The parse line above (step order, shells,
id,if:) is read from the current head, and the stop step's gate names the start step, which comes earlier in the same job. The three steps'run:andwith:texts are byte-identical to the ones the timings and samples above came from. The upload step's path is${{ runner.temp }}/e2e-host-pressure.log, the file the sampler writes to$RUNNER_TEMP.actionlint1.7.7 with shellcheck reports the same three findings on base and on a head with theserun:texts, all at other jobs' lines; none are new.Related Issues
Part of #22314
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.
🤖 Generated with Claude Code