Repository navigation
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This is a do-not-merge diagnostic, and each probe answers the stated question: who holds shard 7's extra ~3.5 GiB. The only finding is a knob CI never sets. Notes
What this change shipsInventory (6 items) — 5 justifiedIntent: find which tests and processes push backend shard 7 to the runner's memory limit. This is an ADDITION (diagnostic instrumentation).
[FIRST-PRINCIPLES-REVIEWED] 3c39d86 Description read[DESCRIPTION-READ] d51f76b6f79927aaf596e4f6314b8980abf5957be6bf274df7302b83debf397c That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a diagnostic-only probe that changes no selection or outcomes, is opt-in, and is easy to revert. Notes
[DESIGN-REVIEWED] 3c39d86 Description read[DESCRIPTION-READ] 985a59c6ec49ae8ccdbde08361fc3f48674ed22df34d571c936d878b8c91aa2b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Correction: closed too early by mistake and reopened. This diagnostic draft will be closed once #36653 merges. Branch kept for the probe script. |
Diagnostic only, not for merge. Backend shard 7 peaks within about 700 MiB of the runner memory and loses its runner several times a day. This records which tests grow a worker RSS and which processes hold memory at the host peak.
8b40043 to
3c39d86
Compare
|
Diagnosis only, done; the probe stays on branch diag/shard7-runner-loss. |
Goal: Measure which tests and processes hold memory on the backend shards. Diagnostic only, do not merge.
Problem / Motivation
Backend Tests (3.12, 7)loses its self-hosted runner several times a day ("The self-hosted runner lost communication with the server", no log). Surviving shard-7 jobs peak at 14,877-15,045 MiB of 15,754 MiB, about 3.5 GiB above every other shard (10-12 GiB) at the same 6 workers. The existing sampler only records the peak, not who holds it.Why it matters
Shard 7 red on unrelated PRs and on main, as lost runners and as xdist worker crashes.
Not a goal
Fixing anything. This PR is closed once the numbers are read; the fix ships separately.
What changed
scripts/ci_rss_probe.py: opt-in pytest plugin. Each xdist worker logs tests whose RSS, peak RSS or children's peak RSS grew by at least 32 MiB, plus a per-file RSS sum at session end.ci.ymlbackend-test job: loads the plugin, snapshotspsat each new host peak above 11 GiB, samples host memory every 30 s, and prints all of it in "Report peak memory".Tests
Diagnostic CI run only; static gates (black, flake8, isort) on the new script.