Skip to content

ci: record per-test worker rss on backend shards (diagnostic) - #36635

Closed
CrysisDeu wants to merge 1 commit into
mainfrom
diag/shard7-runner-loss
Closed

CrysisDeu wants to merge 1 commit into
mainfrom
diag/shard7-runner-loss

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Goal: Measure which tests and processes hold memory on the backend shards. Diagnostic only, do not merge.

Problem / Motivation

Backend Tests (3.12, 7) loses its self-hosted runner several times a day ("The self-hosted runner lost communication with the server", no log). Surviving shard-7 jobs peak at 14,877-15,045 MiB of 15,754 MiB, about 3.5 GiB above every other shard (10-12 GiB) at the same 6 workers. The existing sampler only records the peak, not who holds it.

Why it matters

Shard 7 red on unrelated PRs and on main, as lost runners and as xdist worker crashes.

Not a goal

Fixing anything. This PR is closed once the numbers are read; the fix ships separately.

What changed

  • scripts/ci_rss_probe.py: opt-in pytest plugin. Each xdist worker logs tests whose RSS, peak RSS or children's peak RSS grew by at least 32 MiB, plus a per-file RSS sum at session end.
  • ci.yml backend-test job: loads the plugin, snapshots ps at each new host peak above 11 GiB, samples host memory every 30 s, and prints all of it in "Report peak memory".

Tests

Diagnostic CI run only; static gates (black, flake8, isort) on the new script.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This is a do-not-merge diagnostic, and each probe answers the stated question: who holds shard 7's extra ~3.5 GiB. The only finding is a knob CI never sets.

Notes

  • Item 2 — zero consumers: parser.addoption("--ci-rss-min-mib", type=int, default=32) is never passed by ci.yml. The grep ci-rss-min-mib matches only its definition. Subtraction: delete the option and write 32 as a module constant in scripts/ci_rss_probe.py.

What this change ships

Inventory (6 items) — 5 justified

Intent: find which tests and processes push backend shard 7 to the runner's memory limit. This is an ADDITION (diagnostic instrumentation).

  1. Each backend xdist worker logs the tests that grew its RSS, peak RSS or children's peak RSS by 32 MiB or more — justified
  2. A new pytest option sets that 32 MiB threshold — zero consumers, CI always uses the default
  3. At session end, each worker writes its top-15 per-file RSS totals — justified
  4. A ps top-15 snapshot is taken at each new host peak above 11 GiB — justified
  5. Host memory is sampled every 30 s and printed — justified
  6. The "Report peak memory" step prints the largest per-test records and each worker's end record — justified

[FIRST-PRINCIPLES-REVIEWED] 3c39d86

Description read

[DESCRIPTION-READ] d51f76b6f79927aaf596e4f6314b8980abf5957be6bf274df7302b83debf397c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a diagnostic-only probe that changes no selection or outcomes, is opt-in, and is easy to revert.

Notes

  • RUSAGE_CHILDREN.ru_maxrss is the high-water mark of the single largest child that has been waited on, not a sum. So d_kids only moves when a new child beats every earlier one. It misses children that are still running or were never reaped, which are the likely holders of shard 7's extra 3.5 GiB. The ps snapshot at the host peak is the better signal; read the numbers with that in mind.
  • per_file adds up d_rss per test. Freed memory and allocator reuse make those sums noisy. Ranking files by the worker's RSS at the last test of each file would show held memory more directly.

[DESIGN-REVIEWED] 3c39d86

Description read

[DESCRIPTION-READ] 985a59c6ec49ae8ccdbde08361fc3f48674ed22df34d571c936d878b8c91aa2b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3c39d86

Verdict parsed from the review's SHA-scoped output markers for commit 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3c39d862bb32a3f6ee4e8405311e3bc5379cf77b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8b400438f1f3d5d481c8cd5eaeb5bbc1e989e082 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8b40043

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8b400438f1f3d5d481c8cd5eaeb5bbc1e989e082 false-positive: <reason, 20 to 500 characters>

@CrysisDeu

CrysisDeu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Correction: closed too early by mistake and reopened. This diagnostic draft will be closed once #36653 merges. Branch kept for the probe script.

@CrysisDeu CrysisDeu closed this Nov 13, 2026
@CrysisDeu CrysisDeu reopened this Nov 13, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
Diagnostic only, not for merge. Backend shard 7 peaks within about 700 MiB of the runner memory and loses its runner several times a day. This records which tests grow a worker RSS and which processes hold memory at the host peak.
@CrysisDeu
CrysisDeu force-pushed the diag/shard7-runner-loss branch from 8b40043 to 3c39d86 Compare November 13, 2026 19:12
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Diagnosis only, done; the probe stays on branch diag/shard7-runner-loss.

@CrysisDeu CrysisDeu closed this Nov 13, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant