Skip to content

test: sweep every mutating and parameterized route for its guard - #14290

Merged
iamwhatever merged 1 commit into
test/integration-route-sweepfrom
test/integration-route-sweep-2
Sep 27, 2026
Merged

iamwhatever merged 1 commit into
test/integration-route-sweepfrom
test/integration-route-sweep-2

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every mutating and every parameterized route the gateway serves is checked for its guard, and every parameterized GET for a non-5xx answer to an unknown id, on every CI run.

#14269 swept the parameter-less GETs. The other 865 routes -- every POST/PUT/PATCH/DELETE and every route with a path token -- still had no fleet-wide guard check, and no check that an unknown id answers as a client error rather than a server failure.

Why it matters

An unguarded mutating route is the worst kind of leak (it acts); a 5xx for an unknown id reads as an outage to any monitor or client.

Not a goal

What changed (motivation -> approach -> change)

Sixth slice of the in-process integration layer (harness #13621; memory seams #13736; spawn admission #14160; streaming #14258; parameter-less GET sweep #14269). Two more sweeps in test/integration/test_route_sweep.py cover the rest of the router: every route that is not a parameter-less GET (865 of 1248), with the stand-in sweep-probe substituted for every {token}.

Stacked on #14269 (base test/integration-route-sweep): it extends that PR's file and uses its gw.registered_routes(). Retarget as the stack lands. The diff shown is this slice alone.

Tracks #14288 (the two 5xx-for-unknown-id routes; the table entries come out when it is fixed, this PR does not close it).

The two new contracts

Contract Routes Assertion Exceptions (exact path + reason, in the test)
Guarded every mutating route and every parameterized route (865) unauthenticated, no body -> 401 or 403, so the guard is proven to run before any handler could act UNGUARDED_OTHER: GET /{name} -> 200 (SPA fallback); GET /browser-view/{tail} -> 404; GET /artifact-app/{slug}/{token}/{path} and GET /sandbox-doc/{doc_id}/{token} -> 404 (token in the URL is the credential); POST /api/auth/logout -> 200 (idempotent); POST /api/messaging/teams -> 503 (inbound webhook with its own signature auth; channel not enabled)
Unknown id every parameterized GET (127) authenticated, stand-in id -> below 500 (404 is the contract; 403/400/200 are how some routes say it) HELD_OPEN_PARAM: /api/sessions/{id}/agents/{agent_id}/stream (SSE); UNKNOWN_ID_ANSWERS_5XX: /api/remote-artifacts/{provider}/browse (503) and /api/remote-artifacts/{provider}/{external_id} (502) -> #14288

Observed distribution for the unknown-id sweep on a fresh home: 404 x57, 403 x36, 400 x17, 200 x14, plus the three listed.

What the coverage number means

pytest test/integration alone now requests 1235/1247 routes (99%); .github/workflows/ci.yml tightens --min from 25 to 95 (the only workflow change). The 12 the metric still misses are duplicate registrations of the same canonical path (the hit is credited to the first resource that matches). docs/system-specs/common/testing-conventions.md now states what the ratchet measures: every route has been asked for its guard and its fresh-home answer, not that every route's behaviour is pinned -- the seam tests beside the sweep (bugs 11-16, 20) are that, one route at a time.

Tests

  • Local, on the stack: pytest test/integration -n 2 --dist loadgroup -> 25 passed, 37s (the four sweeps together ~22s).
  • black / isort / flake8 / mypy on the touched file; comment-history gate; docs lint.
  • No route is called with credentials except the parameterized GETs; no mutating route is ever reached past the guard (the unauthenticated request is refused before the handler), so the sweep leaves the fresh home as it found it.

Backwards compatibility

Compatible: tests and a CI ratchet only; no product code changes.

@CrysisDeu
CrysisDeu requested review from a team and Zedmor and removed request for a team September 26, 2026 21:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of a703e93b8b7f05e24761c5d8ead594c235834a29 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the six UNGUARDED_OTHER entries pin intended contracts — especially POST /api/auth/logout answering 200 unauthenticated — not defects grandfathered as contract.

What this change ships

Inventory (6 items) — 6 justified

Intent: prove every mutating and parameterized route runs its auth guard and answers an unknown id sanely — an ADDITION (test coverage), sixth slice of a merged series.

  1. Every mutating/parameterized route (865) now proven to refuse unauthenticated requests — justified
  2. Six exact-path unauthenticated exceptions pinned with status and reason — justified
  3. Every parameterized GET (127) proven to answer an unknown id below 500 — justified
  4. Two known 5xx-on-unknown-id routes pinned as defects with tracking issue remote-artifacts routes answer 502/503 for an unknown publish provider #14288, entry forced out when fixed — justified
  5. CI route-coverage ratchet raised from 25% to 95% — justified
  6. testing-conventions.md now states the ratchet measures requested routes, not pinned behaviour — justified

[FIRST-PRINCIPLES-REVIEWED] a703e93

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a703e93b8b7f05e24761c5d8ead594c235834a29 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify none of the 14 parameterized GETs answering 200 to an unknown id should 404 — the below-500 floor blesses them silently.

Suggestions

  • Routes registered as method * are probed only as POST (wire = "POST" if method == "*"); a per-verb guard gap on other methods passes unchecked — probe at least one more verb per * route.

[DESIGN-REVIEWED] a703e93

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a703e93b8b7f05e24761c5d8ead594c235834a29 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a703e93

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a703e93b8b7f05e24761c5d8ead594c235834a29: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a703e93b8b7f05e24761c5d8ead594c235834a29 — this comment is updated in place on each push.

Review details

The route-coverage ratchet is raised onto a metric this PR's blanket sweeps saturate, so it can no longer name an untested route.

FINDING — .github/workflows/ci.yml:3310 — --min 95 lands on a metric the two new sweeps saturate: _note_hit runs before the request and credits a route whatever the status (test/integration/conftest.py:546), while _other_routes/_parameterized_get_routes enumerate every registered route from the live router, so any route added later is auto-credited by its own unauthenticated 403 and neither --min 95 nor --missing can go red for a route with zero end-to-end coverage — the thing the job comment calls "the share of registered routes the integration suite exercised end-to-end" → Fix: credit a hit only for a request that resolved into a handler, or gate the ratchet on the auth=True hit set, in test/integration/conftest.py / scripts/check_integration_route_coverage.py (both untouched here, so this cannot land inside this diff).

[OPUS-REVIEWED] a703e93

Verdict parsed from the review's SHA-scoped output markers for commit a703e93b8b7f05e24761c5d8ead594c235834a29.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a703e93b8b7f05e24761c5d8ead594c235834a29: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep branch from 178d25f to 579c68a Compare September 27, 2026 04:32
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep-2 branch from 60c70ef to 3b0735f Compare September 27, 2026 04:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep branch from 579c68a to 488c083 Compare September 27, 2026 05:02
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep-2 branch from 3b0735f to bdeb0f0 Compare September 27, 2026 05:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep branch from 488c083 to f6bcaed Compare September 27, 2026 05:13
Sixth slice of the in-process integration layer, extending the route
sweep to the rest of the router. Two more sweeps in
test/integration/test_route_sweep.py, over every route that is not a
parameter-less GET (865 of the 1248), each path token replaced by a
stand-in that matches nothing on a fresh home.

Guarded: without credentials every mutating route and every
parameterized route answers 401 or 403, the mutating ones sent with no
body, so the guard is proven to run before any handler could act. The
six that answer otherwise are listed with the status and the reason:
the SPA fallback, the browser-view route this build does not serve, two
routes authenticated by a token in the URL, the idempotent logout, and
the inbound Teams webhook that carries its own signature auth.

Unknown id: with credentials every parameterized GET asked for an id
that does not exist answers below 500. 404 is the contract; 403, 400
and 200 are how some routes say it; a 5xx never is. One SSE stream is
held open and listed; the two remote-artifacts routes that answer 502
and 503 for an unknown provider are listed against #14288 and come out
when the status moves below 500.

Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens
from 25% to 95%. The conventions doc now says what that number is:
every route asked for its guard and its fresh-home answer, not every
route's behaviour pinned -- that is what the seam tests beside the
sweep are for.
@CrysisDeu
CrysisDeu force-pushed the test/integration-route-sweep-2 branch from bdeb0f0 to a703e93 Compare September 27, 2026 05:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@iamwhatever
iamwhatever merged commit eb38c8d into test/integration-route-sweep Sep 27, 2026
31 of 32 checks passed
@iamwhatever
iamwhatever deleted the test/integration-route-sweep-2 branch September 27, 2026 07:12
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
CrysisDeu added a commit that referenced this pull request Sep 27, 2026
)

Sixth slice of the in-process integration layer, extending the route
sweep to the rest of the router. Two more sweeps in
test/integration/test_route_sweep.py, over every route that is not a
parameter-less GET (865 of the 1248), each path token replaced by a
stand-in that matches nothing on a fresh home.

Guarded: without credentials every mutating route and every
parameterized route answers 401 or 403, the mutating ones sent with no
body, so the guard is proven to run before any handler could act. The
six that answer otherwise are listed with the status and the reason:
the SPA fallback, the browser-view route this build does not serve, two
routes authenticated by a token in the URL, the idempotent logout, and
the inbound Teams webhook that carries its own signature auth.

Unknown id: with credentials every parameterized GET asked for an id
that does not exist answers below 500. 404 is the contract; 403, 400
and 200 are how some routes say it; a 5xx never is. One SSE stream is
held open and listed; the two remote-artifacts routes that answer 502
and 503 for an unknown provider are listed against #14288 and come out
when the status moves below 500.

Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens
from 25% to 95%. The conventions doc now says what that number is:
every route asked for its guard and its fresh-home answer, not every
route's behaviour pinned -- that is what the seam tests beside the
sweep are for.
CrysisDeu added a commit that referenced this pull request Sep 27, 2026
)

Sixth slice of the in-process integration layer, extending the route
sweep to the rest of the router. Two more sweeps in
test/integration/test_route_sweep.py, over every route that is not a
parameter-less GET (865 of the 1248), each path token replaced by a
stand-in that matches nothing on a fresh home.

Guarded: without credentials every mutating route and every
parameterized route answers 401 or 403, the mutating ones sent with no
body, so the guard is proven to run before any handler could act. The
six that answer otherwise are listed with the status and the reason:
the SPA fallback, the browser-view route this build does not serve, two
routes authenticated by a token in the URL, the idempotent logout, and
the inbound Teams webhook that carries its own signature auth.

Unknown id: with credentials every parameterized GET asked for an id
that does not exist answers below 500. 404 is the contract; 403, 400
and 200 are how some routes say it; a 5xx never is. One SSE stream is
held open and listed; the two remote-artifacts routes that answer 502
and 503 for an unknown provider are listed against #14288 and come out
when the status moves below 500.

Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens
from 25% to 95%. The conventions doc now says what that number is:
every route asked for its guard and its fresh-home answer, not every
route's behaviour pinned -- that is what the seam tests beside the
sweep are for.
iamwhatever pushed a commit that referenced this pull request Sep 27, 2026
* test: sweep every parameter-less GET route for its guard and a fresh-home answer

Fifth slice of the in-process integration layer. test/integration/
test_route_sweep.py reads every route the live router serves at boot
(the new gw.registered_routes(), the same reading the coverage ratchet
counts) and drives each parameter-less GET twice through the booted
gateway.

Guarded: without credentials the route answers 401 or 403. Every GET
the router serves is swept, the SPA shell and its assets included; the
eight that answer otherwise are listed with the status each answers and
the reason (the shell, favicon and logo an unauthenticated browser needs
for the login; the health, live and ready probes; the pre-login theme
flags; the browser-view route this build answers 404 to everyone). A
route that is not listed and answers outside 401/403 is a guard
regression, and one that is listed and moves off its status is a broken
probe or asset. Serves: with credentials the
route does not fail with a 5xx on a fresh home; the six that answer 503
on a fresh home are listed with what each names as unavailable, and a
500 is never a contract.

Exclusions are exact paths with a reason, never patterns: two routes
hold the connection open (the SSE stream and the log long-poll), one
fetches the official app catalog over the network (the rootdir conftest
fences it), and two shell the real cloud CLI (the preflight against the
operator's profile; the deploy profile listing). For that last one the harness now also points the CLI at
credential files that do not exist under the test home, so no test in
the layer can exercise a developer's AWS account. 383 routes are swept.

Route coverage moves from 9 to 383 of 1247 (30.7%); the ratchet
tightens from 0.5% to 25%. The conventions doc gains the sweep rule.

The harness also grants the disposable home unsandboxed-exec consent,
as the E2E suite does: the CI container has no sandbox backend, and
without it every child the gateway spawns is refused before the fake
model runs.

* test: sweep every mutating and parameterized route for its guard (#14290)

Sixth slice of the in-process integration layer, extending the route
sweep to the rest of the router. Two more sweeps in
test/integration/test_route_sweep.py, over every route that is not a
parameter-less GET (865 of the 1248), each path token replaced by a
stand-in that matches nothing on a fresh home.

Guarded: without credentials every mutating route and every
parameterized route answers 401 or 403, the mutating ones sent with no
body, so the guard is proven to run before any handler could act. The
six that answer otherwise are listed with the status and the reason:
the SPA fallback, the browser-view route this build does not serve, two
routes authenticated by a token in the URL, the idempotent logout, and
the inbound Teams webhook that carries its own signature auth.

Unknown id: with credentials every parameterized GET asked for an id
that does not exist answers below 500. 404 is the contract; 403, 400
and 200 are how some routes say it; a 5xx never is. One SSE stream is
held open and listed; the two remote-artifacts routes that answer 502
and 503 for an unknown provider are listed against #14288 and come out
when the status moves below 500.

Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens
from 25% to 95%. The conventions doc now says what that number is:
every route asked for its guard and its fresh-home answer, not every
route's behaviour pinned -- that is what the seam tests beside the
sweep are for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants