Skip to content

fix(tests): isolate platform-dependent main test failures - #10506

Merged
iamwhatever merged 2 commits into
mainfrom
fix/acp-recovery-test-pid-isolation
Sep 13, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/acp-recovery-test-pid-isolation

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Tests can fail because of the machine running them rather than the behavior they check. A macOS CI run treated an invented PID as a live host process. Windows CI exceeded a duration-accounting test's 4.5-second wall-clock ceiling. Main-based local runs exposed assumptions about filesystem owners, kernel headers, temporary paths and inherited runner options.

Why it matters

These failures make unrelated changes red and send contributors looking for product bugs that are not there. Tests should control their inputs while keeping behavior and refusal assertions real.

What changed (motivation → approach → change)

Keep host state out of unit-test fixtures. A real liveness oracle receives an explicit process backend, and a simulated host PID collision proves that no host probes run. The reset test uses a non-exhausting process probe while retaining its force-kill assertion. Ownership fixtures pin only ancestors above their temporary tree, preserving real fixture ownership and permission bits. Kernel headers are matched to their architecture, Unix sockets use the existing short-path helper, and nested pytest clears inherited PYTEST_ADDOPTS.

Check duration arithmetic with controlled time, not a runner-speed budget. Real pytest/xdist integration still checks report phases, fixture cleanup, timer disarming and duration-key completeness. Direct tests drive the real root-conftest guard with an isolated clock and deterministic CallInfo values to detect lost or double-counted time. The integration test needs neither a sleep nor an upper wall-clock limit.

Keep unavailable facilities separate from broken behavior. The real cgroup test skips only a user-bus connection failure; other scope failures still fail. Its subprocess call pins UTF-8. Testing conventions document these requirements. No production code, dependency declarations, security policy, CI configuration or UI changes.

Tests

  • Latest merge head 86ecdd1: all ten changed test files passed together — 1,727 passed, 7 skipped, zero failures. A focused verifier confirmed the conflict resolution preserves the previously reviewed deterministic test byte-for-byte and includes every other incoming main change. The elapsed-time file's seven tests passed separately. Isort, flake8, subprocess-encoding and comment-history checks passed; wheel build passed.
  • Deterministic PID collision reproduces the original CI evidence before fixture isolation and passes after it.
  • Duration tests use exact cases: protocol 10 minus logged setup 2 gives a synthetic call duration of 8; an already-logged call gives the replacement teardown only its own 1.5 duration. Lost-duration and duplicate-duration mutations both fail. The real root conftest is unchanged.
  • The force-kill assertion was mutation-checked: removing the force-kill branch makes it fail. Production code is absent from this diff.
  • Full backend run on precursor 438ccb0a2: 101,200 passed, 392 skipped, 9 xfailed, zero failures. All 50 local gate types were exercised; the one new encoding omission was fixed and that gate rechecked. Frontend build/type/lint/guard tests, Electron tests (1,872 total, zero failures), bundle checks and Python static checks passed.
  • On published precursor 330024f, remote macOS and Linux backend shards and the other Windows shards passed. Windows shard 4 identified the duration ceiling addressed here. The latest head needs its fresh CI run; the full 101,200-test local suite was not repeated for the duration-test-only revision.
  • GPT and Opus local reviews returned no findings on b8b4470.

Manual verification

Verified on Linux aarch64 with Python 3.12.13 and pinned project test tools. The PID-collision case uses an injected macOS-style backend; native macOS/Windows results come from CI. The real cgroup-limit test skips locally because the user bus is inaccessible; unrelated scope errors remain failures.

Six initial local failures were resolved by installing pip in the isolated venv and refreshing editable package metadata, without source edits. Electron's separate lockfile dependencies were installed for its tests.

Screenshots / video

N/A — tests and testing documentation only; no visual change.

Related Issues

no linked issue: these test-isolation failures were found during a common-main-failure investigation.

The separate dependency-lock create race tracked by #10425 is handled by #10464; this PR does not duplicate it. Main includes #10532's relaxed duration ceiling; this PR replaces that remaining wall-clock bracket with deterministic accounting tests.

Pattern harvest

Rule candidate: review-prompt
Pattern: isolate fabricated PIDs, ancestor ownership, nested pytest options and Unix-socket paths from the test host. Match architecture-specific headers to the architecture being checked. Verify exact duration accounting with controlled clocks rather than a fixed wall-time ceiling. Preserve real behavior and refusal assertions.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and Zedmor September 13, 2026 08:14
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Make main-based test runs independent of fabricated host PIDs, mount ownership, host architecture, temporary path length and inherited runner options while preserving behavior and refusal assertions.
Not a goal: Change production process liveness, executable trust, sandbox policy, or dependency-lock provisioning; the separate dependency-lock race is covered by PR #10464.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 13, 2026
@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 86ecdd1fa0c293f9cb95f45943995341ae601946 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Host-state isolation done at the right layer — injected backends/clocks into real code paths, with each deleted timing pin replaced by a strictly stronger deterministic pin.

The one deleted assertion that pinned behavior (the 3.0 <= victim_duration < 6.0 double-count bracket) is treated as a decision, not a gap: its single-count intent is re-pinned exactly (duration == 8.0, != 10.0, mutation-checked) against the real root-conftest generator, which is what the wall-clock bracket could only approximate. Refusal-side assertions in the ownership and liveness tests stay on real data; only accept-half positive controls get the simulated ancestor chain. Conventions doc updated in the same commit.

Suggestions

  • _trust_ancestors_above is now triplicated (test_service.py, test_issue_radar_gh_bin.py, test_source_providers.py); move one copy (the extra=-capable one) into a shared test/ helper module beside tmpdir_helpers.py so these security-fixture stats can't drift apart.

[DESIGN-REVIEWED] 86ecdd1

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 86ecdd1fa0c293f9cb95f45943995341ae601946 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 86ecdd1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 86ecdd1fa0c293f9cb95f45943995341ae601946: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 86ecdd1fa0c293f9cb95f45943995341ae601946 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 86ecdd1

Verdict parsed from the review's SHA-scoped output markers for commit 86ecdd1fa0c293f9cb95f45943995341ae601946.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 86ecdd1fa0c293f9cb95f45943995341ae601946: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 13, 2026
@bolichen97

bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=2696ffbbb6cb Root-conftest import safety

outcome: rebutted
This is a future-maintenance concern, not a present import-safety defect: the root pytest home floor is already installed before these synchronous tests run, the frame-recording environment key is already scrubbed, and the import-time symlink probe cleans its TemporaryDirectory.
The loaded module has an independent time binding and report dictionary; the CallInfo patch is restored inside MonkeyPatch.context before the test returns. Both local reviewers checked these concrete side effects, and the duration-loss/double-count mutations fail against the real guard.
Moving the host-floor hook into a new shared module or imposing a general re-execution contract expands the fixture repair without closing a demonstrated failure. The helper docstring explicitly identifies and explains the isolated root-conftest load; retain the current test-only boundary.

@bolichen97

bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=a4cff9a02342 File-local ancestor fixtures

outcome: rebutted
Each small file-local helper overrides only an explicit set of ancestors above its fixture root. Target-file ownership, permission bits and unrelated paths remain real. The service-only extra mapping expresses that suite’s existing root-owned-parent case rather than a diverging executable-trust policy.
Consolidation is optional maintenance, not required to fix a current assertion or security defect. Introducing a shared helper/import surface is outside this bounded common-failure repair; retain the local helpers and the tested refusal assertions. No production validator changes.

@bolichen97
bolichen97 force-pushed the fix/acp-recovery-test-pid-isolation branch from b8b4470 to c6652d5 Compare September 13, 2026 09:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Sep 13, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=fe4f285a07a2 Keep this repair scoped to host-independent fixtures

outcome: rebutted
The helpers are intentionally file-local positive-control fixtures. Each overrides only the explicit ancestor set above its own temporary root; target ownership, permission bits, unrelated paths and the refusal assertions remain real. The service-only extra mapping expresses its existing root-owned-parent case.
Shared extraction is optional maintenance, not a demonstrated correctness or security fix. Introducing another shared test import surface is disproportionate to this bounded common-failure repair. Retain the small local helpers, the documented isolation contract, and the passing cross-platform refusal tests without another code revision.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 13, 2026 12:05

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only isolation fixes -- platform-dependent assertions in ten test modules are made independent of the host (browser launcher, gh binary discovery, sandbox argv, xdist escaped-failure guard), with no production file touched. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.

@iamwhatever
iamwhatever merged commit 3e2bbd0 into main Sep 13, 2026
79 checks passed
@iamwhatever
iamwhatever deleted the fix/acp-recovery-test-pid-isolation branch September 13, 2026 12:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 13, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants