Repository navigation
fix(tests): isolate platform-dependent main test failures - #10506
Conversation
|
Intent: Make main-based test runs independent of fabricated host PIDs, mount ownership, host architecture, temporary path length and inherited runner options while preserving behavior and refusal assertions. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Host-state isolation done at the right layer — injected backends/clocks into real code paths, with each deleted timing pin replaced by a strictly stronger deterministic pin. The one deleted assertion that pinned behavior (the Suggestions
[DESIGN-REVIEWED] 86ecdd1 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
|
b8b4470 to
c6652d5
Compare
|
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only isolation fixes -- platform-dependent assertions in ten test modules are made independent of the host (browser launcher, gh binary discovery, sandbox argv, xdist escaped-failure guard), with no production file touched. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.
Problem / Motivation
Tests can fail because of the machine running them rather than the behavior they check. A macOS CI run treated an invented PID as a live host process. Windows CI exceeded a duration-accounting test's 4.5-second wall-clock ceiling. Main-based local runs exposed assumptions about filesystem owners, kernel headers, temporary paths and inherited runner options.
Why it matters
These failures make unrelated changes red and send contributors looking for product bugs that are not there. Tests should control their inputs while keeping behavior and refusal assertions real.
What changed (motivation → approach → change)
Keep host state out of unit-test fixtures. A real liveness oracle receives an explicit process backend, and a simulated host PID collision proves that no host probes run. The reset test uses a non-exhausting process probe while retaining its force-kill assertion. Ownership fixtures pin only ancestors above their temporary tree, preserving real fixture ownership and permission bits. Kernel headers are matched to their architecture, Unix sockets use the existing short-path helper, and nested pytest clears inherited PYTEST_ADDOPTS.
Check duration arithmetic with controlled time, not a runner-speed budget. Real pytest/xdist integration still checks report phases, fixture cleanup, timer disarming and duration-key completeness. Direct tests drive the real root-conftest guard with an isolated clock and deterministic CallInfo values to detect lost or double-counted time. The integration test needs neither a sleep nor an upper wall-clock limit.
Keep unavailable facilities separate from broken behavior. The real cgroup test skips only a user-bus connection failure; other scope failures still fail. Its subprocess call pins UTF-8. Testing conventions document these requirements. No production code, dependency declarations, security policy, CI configuration or UI changes.
Tests
Manual verification
Verified on Linux aarch64 with Python 3.12.13 and pinned project test tools. The PID-collision case uses an injected macOS-style backend; native macOS/Windows results come from CI. The real cgroup-limit test skips locally because the user bus is inaccessible; unrelated scope errors remain failures.
Six initial local failures were resolved by installing pip in the isolated venv and refreshing editable package metadata, without source edits. Electron's separate lockfile dependencies were installed for its tests.
Screenshots / video
N/A — tests and testing documentation only; no visual change.
Related Issues
no linked issue: these test-isolation failures were found during a common-main-failure investigation.
The separate dependency-lock create race tracked by #10425 is handled by #10464; this PR does not duplicate it. Main includes #10532's relaxed duration ceiling; this PR replaces that remaining wall-clock bracket with deterministic accounting tests.
Pattern harvest
Rule candidate: review-prompt
Pattern: isolate fabricated PIDs, ancestor ownership, nested pytest options and Unix-socket paths from the test host. Match architecture-specific headers to the architecture being checked. Verify exact duration accounting with controlled clocks rather than a fixed wall-time ceiling. Preserve real behavior and refusal assertions.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)