Skip to content

fix(ci): decline container_tests collection for directly-named files too - #10709

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/container-tests-direct-arg-guard
Sep 14, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
rubencu:fix/container-tests-direct-arg-guard

Conversation

@rubencu

@rubencu rubencu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backend Tests (3.12, 1) fails on every frontend-only PR with ModuleNotFoundError: No module named 'httpx' from container_tests/test_review_findings.py, which fails the Coverage Gate closed and reds PR Readiness. Red right now on at least #10678, #10679, #10680, #10681 and #10698.

Why it matters

No frontend-only PR can reach readiness: passed until this lands. The failing suite's deps (httpx/fastapi/uvicorn) are deliberately absent from the app CI env — the environment is right, the guard just has a hole.

What changed (motivation → approach → change)

The container suite's conftest.py (src/kiro_crew/apps/builtins/aws_control/crew/runtime/container_tests/conftest.py) already declines collection when the image-only deps are missing, via collect_ignore_glob. That glob only filters files pytest discovers by walking the directory. CI's reduced cross-surface path passes this suite's files as explicit pytest arguments, and pytest treats direct arguments as overriding every ignore mechanism — a pytest_ignore_collect hook included — so the modules were imported anyway and hit the missing dep.

Module construction is the one step every path to a test module shares — directory walk, explicit file argument, file.py::test node id. So the conftest now also substitutes a declining collector in pytest_pycollect_makemodule: it raises the module-level skip instead of importing, is scoped to this one directory, and returns None (pytest proceeds normally) wherever the suite may run. The CREW_CONTAINER_TESTS_REQUIRED=1 lane is untouched and still hard-errors when the environment cannot run the suite.

flowchart LR
  A[pytest args name a container test file]:::ctx --> B{collect_ignore_glob}:::ctx
  B -- "walk-discovered: filtered" --> C[not collected]:::ctx
  B -- "named directly: bypassed" --> D[import module -> ModuleNotFoundError]:::removed
  B -- "named directly" --> E[makemodule -> declining collector -> skip]:::added
  classDef added fill:#DCFCE7,stroke:#16A34A
  classDef removed fill:#FEE2E2,stroke:#DC2626,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7
Loading

🟩 added · 🟥 removed · 🟦 unchanged

Tests

Three pins added to test/test_ci_surface_tests.py, using the file's existing conftest-loading harness: with a dep missing, the hook substitutes the declining collector for an in-suite file; with all deps present, it returns None; and it leaves files outside the suite's directory alone. All 49 pre-existing pins in the file still pass.

Manual verification

At the tip of main, reproduced the failure and verified the fix for every invocation shape against a dep-less env: explicit file argument and :: node id now report the module skipped instead of failing; CI's exact reduced shape (-q -n auto --timeout=120 --no-cov <container files> <other files>) exits 0 with the other files' tests passing; the plain directory walk is byte-identical to before (glob path untouched); CREW_CONTAINER_TESTS_REQUIRED=1 still raises the refusal.

Pattern harvest

Rule candidate: review-prompt
Pattern: collect_ignore/collect_ignore_glob only covers walk-discovered files; any suite gated that way is still importable via explicit pytest arguments and needs a collector-level decline.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fixes #10700

@rubencu
rubencu requested review from a team and krishdhasmana September 14, 2026 05:37
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 14, 2026
@rubencu

rubencu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Intent: Make the container suite's existing may-this-environment-collect decline hold for files named explicitly on the pytest command line — the path CI's reduced cross-surface run uses — so frontend-only PRs stop failing the backend shard on the suite's deliberately-absent image deps (#10700).
Not a goal: Changing where the suite DOES run (deps-present envs, the CREW_CONTAINER_TESTS_REQUIRED lane), adding the image deps to the app env, or changing which files the cross-surface selector emits.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 14, 2026
collect_ignore_glob only filters files pytest discovers by walking the
directory. CI's reduced cross-surface path passes this suite's files as
explicit pytest arguments, which pytest treats as overriding every ignore
mechanism - so on every frontend-only PR the backend shard imported a module
whose deps (httpx/fastapi/uvicorn) are deliberately absent from the app env,
and failed with ModuleNotFoundError (issue kirodotdev#10700).

Module construction is the one step every path to a test module shares, so
the conftest now also substitutes a declining collector in
pytest_pycollect_makemodule: it raises the module-level skip instead of
importing, scoped to this directory, and returns None wherever the suite may
run. The REQUIRED lane (CREW_CONTAINER_TESTS_REQUIRED=1) is unchanged and
still hard-errors when the environment cannot run the suite.
@rubencu
rubencu force-pushed the fix/container-tests-direct-arg-guard branch from 6c44a96 to 9a60f3a Compare September 14, 2026 05:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 9a60f3af5fc909eb540024e35c68b0f93409ba30 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Fixes the decline-bypass at the one seam every collection path shares, preserving the required-lane's refusal-to-skip invariant; scoped and pinned.

Suggestions

  • The pin tests sentinel out _DeclinedModule, so collect()'s raise — the actual skip mechanism, whose pragma claims it is "exercised via the pin tests' subprocess" — never runs under test; one subprocess pytest run passing a declined file directly would prove pytest reports it as a skip, not an error, end-to-end.

[DESIGN-REVIEWED] 9a60f3a

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 9a60f3af5fc909eb540024e35c68b0f93409ba30 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims check out. The premise that explicit pytest args bypass ignore mechanisms is already documented as "verified, not assumed" on the base at scripts/ci-surface-tests.py:197-199; test_review_findings.py matches the cross-surface \bpackaging\b pattern so CI does name it explicitly; the Windows sibling is already handled at the selector, and the macOS list's one entry deliberately relies on direct naming (its dedicated CI step), so no unfixed siblings. The fix sits at the shared construction step (cause level), tests are added, and the issue plus five red PRs give the defect provenance. No product-shape change, no riders.

First-Principles-Verdict: PASS

Verify the hook fires in one real pytest run on a directly-named file — the new pins call it in isolation; only the author's manual runs cover the live path.

What this change ships

Inventory (3 items) — 3 justified

Intent: unblock frontend-only PRs whose Backend Tests shard fails because CI names container-suite files explicitly in an env deliberately missing their deps. FIX.

  1. Frontend-only PRs stop failing Backend Tests on the container suite's absent image deps — justified
  2. Naming a container test file directly in a declined env now reports a module skip instead of an import error — justified
  3. Three pins on the hook's decline/allow/scope decisions in test_ci_surface_tests.py — justified

[FIRST-PRINCIPLES-REVIEWED] 9a60f3a

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 9a60f3af5fc909eb540024e35c68b0f93409ba30 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a60f3a

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 9a60f3af5fc909eb540024e35c68b0f93409ba30 via the fork AI-review pipeline; updated in place on each push.

Review details

I've validated the single candidate against the file. Let me assess it.

The candidate hinges on two premises it explicitly cannot confirm (its own confidence is "low"):

  1. That pytest passes an unresolved module_path for a direct file argument.
  2. That the target CI runner checks out under a symlinked path.

Falsifying (a) — "a concrete input that occurs in practice":

  • The hook's decline path fires only in the app CI env lacking image deps (fastapi/httpx/uvicorn). On a developer laptop those deps are present (requirements-dev.txt), so _declined is None and the hook returns None regardless — no symlink comparison is even reached.
  • The direct-argument path is specifically CI's reduced cross-surface path (ci.yml passing explicit file lists). That runs on GitHub Actions Linux, where the checkout lives at /home/runner/work/<repo>/<repo> — no symlinked component. Path(__file__).resolve() equals the unresolved spelling there, so module_path.parent == _HERE holds and the decline fires correctly.
  • The "symlinked checkout (/tmp-rooted, macOS /private)" input the candidate needs is hedged as "if… symlinked" and does not occur on the environment this hook targets. The same .parent == _HERE idiom at line 209 is unchanged and, per the candidate itself, "presumably exercised without incident."

Input (a) is speculative ("if a caller were to"), not something occurring in practice on the path where the hook runs. The candidate drops under Step 1.

Step 2: the diff is test-infrastructure only (a pytest makemodule hook and its pins). Nothing in it presents a reachable, concrete defect meeting the 80+ bar.

No findings.

[OPUS-REVIEWED] 9a60f3a

@iamwhatever
iamwhatever merged commit 66cc779 into kirodotdev:main Sep 14, 2026
80 of 81 checks passed
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 14, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…too (kirodotdev#10709)

collect_ignore_glob only filters files pytest discovers by walking the
directory. CI's reduced cross-surface path passes this suite's files as
explicit pytest arguments, which pytest treats as overriding every ignore
mechanism - so on every frontend-only PR the backend shard imported a module
whose deps (httpx/fastapi/uvicorn) are deliberately absent from the app env,
and failed with ModuleNotFoundError (issue kirodotdev#10700).

Module construction is the one step every path to a test module shares, so
the conftest now also substitutes a declining collector in
pytest_pycollect_makemodule: it raises the module-level skip instead of
importing, scoped to this directory, and returns None wherever the suite may
run. The REQUIRED lane (CREW_CONTAINER_TESTS_REQUIRED=1) is unchanged and
still hard-errors when the environment cannot run the suite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backend Tests shard 1 fails on every frontend-only PR: container_tests escapes its own collect_ignore_glob

2 participants