Skip to content

test(platform_compat): poll the argv fingerprint read after Popen - #21033

Closed
jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/18310-argv-fingerprint-poll
Closed

jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/18310-argv-fingerprint-poll

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_argv_fingerprint_is_stable_and_tells_processes_apart stops failing at random on Linux CI, and a real failure says which read answered None.

The test fails with assert (None is not None) on unrelated PRs, always on Backend Tests (3.12, 4). The two runs I read (38520843428, 38581781524) both fail at the b assertion: the second child's fingerprint, read once right after its Popen returned.

Why it matters

13 unrelated PR runs went red on it from 10-09 to 10-14, and Coverage Gate and PR Readiness fail closed on each one.

Not a goal

  • No change to process_argv_fingerprint. Answering None for an empty argv is correct for its callers, which compare a recorded fingerprint and must never record a fingerprint of nothing.
  • No change to the sibling test_exact_argv_matches_and_near_misses_do_not, which test(platform_compat): poll the Windows exact-argv read #19197 already polls.

What changed (motivation → approach → change)

Symptom: the first fingerprint read after Popen sometimes answers None.

Root cause: a kernel window, not leaked test state. Popen returns once the child's close-on-exec pipe closes, and Linux closes it during exec before it publishes the new image's argv. A read in that window finds /proc/<pid>/cmdline empty, and process_argv_fingerprint answers None for an empty argv, as documented. Measured on a Linux host, reading /proc/<pid>/cmdline right after Popen([sleep, "300"]) returns:

  • 1 empty read in 3000 on an idle host;
  • 250 empty reads in 12000 with 8 such loops running at once;
  • zero exceptions in either case, so it is the empty-argv branch, not the exception branch.

The single shard is placement, not a predecessor: file sharding assigns test/test_platform_compat.py to shard 4 of 8 (sha256 of the path), so this test only ever runs there. In the failing runs, a read twice through the same code just before b failed, which rules out a patched pc.Path, sys.platform or IS_WINDOWS left behind by an earlier test.

Change, test only: the POSIX branch reads each child's fingerprint through kiro_crew.testing.wait.wait_until under the file's existing _ARGV_READ_CEILING_SECS lost-run ceiling. If it never reads, the failure names the branch: the exception the /proc read raised, or how many bytes it read (darwin: what darwin_process_argv answered). The stability (== a) and distinctness (b != a) assertions are unchanged. The Windows branch is unchanged apart from inlining the two reads.

Why not a readiness byte, as _spawn uses: this test deliberately spawns the plain sleep binary, which cannot signal, so that the argv the kernel reports is exactly the spawn argv on Linux and macOS.

Open #21000 moves process_argv_fingerprint and darwin_process_argv into platform_compat_identity.py and keeps both names on the platform_compat facade. This PR only calls them through pc., so it applies unchanged on top of #21000, and the test target does not move.

Backwards compatibility

Compatible: test-only change; no source file is touched.

Removes nothing: the change only adds a bounded retry and a diagnostic to one test.

Tests

  • test_the_fingerprint_read_retries_an_argv_that_is_not_published_yet: two None reads, then a value, give the value after exactly 3 reads.
  • test_a_fingerprint_that_never_reads_fails_naming_the_branch (Linux): at a 0 s ceiling the failure names the pid and the /proc branch (FileNotFoundError for an unallocatable pid).
  • Repro (scratch, not committed): the live test with the first read of each pid forced to None fails on origin/main and passes on this head.
  • Mutation: replacing the poll with a single read turns the retry pin and the repro red.
  • pytest -n 8 test/test_platform_compat.py plus the wait-discipline, flake-pattern and pid-reader ratchets: 653 passed, 29 skipped. All test_*ratchet*.py and test_*facade*.py plus importers of the module: 2734 passed, 11 skipped.
  • black, isort, flake8, comment-history, subprocess-encoding: pass. Semgrep semgrep/ on the file: no new finding (3 before, 3 after, all pre-existing).

Manual verification

N/A — test-only change. The race was measured directly on a Linux host as described above.

Related Issues

Closes #18310

Pattern harvest

Rule candidate: review-prompt
Pattern: a test reads a child's /proc/<pid> state once, right after Popen returns, assuming exec has published it; poll it with wait_until or have the child signal readiness.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): N/A
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Popen returns once the child's close-on-exec pipe closes, which Linux
does before it publishes the new image's argv, so /proc/<pid>/cmdline
can still read empty right after it and process_argv_fingerprint
correctly answers None. The live fingerprint test read each child once,
so it failed whenever that window was hit.

Read each child's fingerprint through wait_until under the existing
lost-run ceiling, and name in the failure which branch answered None:
the exception the /proc read raised, or how many bytes it read.

Closes kirodotdev#18310
@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 16, 2026 01:15
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 31b949557146b4fb5ff70782c851b5eea3114120 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 31b949557146b4fb5ff70782c851b5eea3114120 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 31b9495

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 31b949557146b4fb5ff70782c851b5eea3114120 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 31b949557146b4fb5ff70782c851b5eea3114120 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the production recorder at ssh_tunnel_manager.py:2280 reads the fingerprint after tunnel readiness, not right after Popen. If it reads earlier, the same race silently records "".

[DESIGN-REVIEWED] 31b9495

Description read

[DESCRIPTION-READ] f7e790d0bb68e5feb85e93c296ab8e1dde2a6fbfa612719652b2d5b70bf21794

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 31b949557146b4fb5ff70782c851b5eea3114120 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 31b9495

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @jeeshofone for this fix and the careful root-cause write-up on the exec/argv window.

This PR went idle and no longer applied: on main the test moved from test/test_platform_compat.py to test/test_platform_compat_process_identity.py. Your change is carried, unchanged in substance and with you as the commit author, in #21926, which is green. Closing this one in favour of it; your fork branch is untouched.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
KiroCrewCaptain pushed a commit that referenced this pull request Oct 18, 2026
…1926)

Popen returns once the child's close-on-exec pipe closes, which Linux
does before it publishes the new image's argv, so /proc/<pid>/cmdline
can still read empty right after it and process_argv_fingerprint
correctly answers None. The live fingerprint test read each child once,
so it failed whenever that window was hit.

Read each child's fingerprint through wait_until under the existing
lost-run ceiling, and name in the failure which branch answered None:
the exception the /proc read raised, or how many bytes it read.

Ported from #21033: the test moved from test/test_platform_compat.py to
test/test_platform_compat_process_identity.py on main, so the original
commit no longer applies; the change is the same.

(hand-ported from commit 31b9495)

Closes #18310

Co-authored-by: Will Laws <will.laws@me.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_argv_fingerprint_is_stable_and_tells_processes_apart returns None on Linux CI

2 participants