Repository navigation
test: sweep every mutating and parameterized route for its guard - #14290
Conversation
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the six What this change shipsInventory (6 items) — 6 justifiedIntent: prove every mutating and parameterized route runs its auth guard and answers an unknown id sanely — an ADDITION (test coverage), sixth slice of a merged series.
[FIRST-PRINCIPLES-REVIEWED] a703e93 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify none of the 14 parameterized GETs answering 200 to an unknown id should 404 — the below-500 floor blesses them silently. Suggestions
[DESIGN-REVIEWED] a703e93 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsThe route-coverage ratchet is raised onto a metric this PR's blanket sweeps saturate, so it can no longer name an untested route. FINDING — .github/workflows/ci.yml:3310 — [OPUS-REVIEWED] a703e93 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
178d25f to
579c68a
Compare
60c70ef to
3b0735f
Compare
579c68a to
488c083
Compare
3b0735f to
bdeb0f0
Compare
488c083 to
f6bcaed
Compare
Sixth slice of the in-process integration layer, extending the route sweep to the rest of the router. Two more sweeps in test/integration/test_route_sweep.py, over every route that is not a parameter-less GET (865 of the 1248), each path token replaced by a stand-in that matches nothing on a fresh home. Guarded: without credentials every mutating route and every parameterized route answers 401 or 403, the mutating ones sent with no body, so the guard is proven to run before any handler could act. The six that answer otherwise are listed with the status and the reason: the SPA fallback, the browser-view route this build does not serve, two routes authenticated by a token in the URL, the idempotent logout, and the inbound Teams webhook that carries its own signature auth. Unknown id: with credentials every parameterized GET asked for an id that does not exist answers below 500. 404 is the contract; 403, 400 and 200 are how some routes say it; a 5xx never is. One SSE stream is held open and listed; the two remote-artifacts routes that answer 502 and 503 for an unknown provider are listed against #14288 and come out when the status moves below 500. Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens from 25% to 95%. The conventions doc now says what that number is: every route asked for its guard and its fresh-home answer, not every route's behaviour pinned -- that is what the seam tests beside the sweep are for.
bdeb0f0 to
a703e93
Compare
eb38c8d
into
test/integration-route-sweep
) Sixth slice of the in-process integration layer, extending the route sweep to the rest of the router. Two more sweeps in test/integration/test_route_sweep.py, over every route that is not a parameter-less GET (865 of the 1248), each path token replaced by a stand-in that matches nothing on a fresh home. Guarded: without credentials every mutating route and every parameterized route answers 401 or 403, the mutating ones sent with no body, so the guard is proven to run before any handler could act. The six that answer otherwise are listed with the status and the reason: the SPA fallback, the browser-view route this build does not serve, two routes authenticated by a token in the URL, the idempotent logout, and the inbound Teams webhook that carries its own signature auth. Unknown id: with credentials every parameterized GET asked for an id that does not exist answers below 500. 404 is the contract; 403, 400 and 200 are how some routes say it; a 5xx never is. One SSE stream is held open and listed; the two remote-artifacts routes that answer 502 and 503 for an unknown provider are listed against #14288 and come out when the status moves below 500. Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens from 25% to 95%. The conventions doc now says what that number is: every route asked for its guard and its fresh-home answer, not every route's behaviour pinned -- that is what the seam tests beside the sweep are for.
) Sixth slice of the in-process integration layer, extending the route sweep to the rest of the router. Two more sweeps in test/integration/test_route_sweep.py, over every route that is not a parameter-less GET (865 of the 1248), each path token replaced by a stand-in that matches nothing on a fresh home. Guarded: without credentials every mutating route and every parameterized route answers 401 or 403, the mutating ones sent with no body, so the guard is proven to run before any handler could act. The six that answer otherwise are listed with the status and the reason: the SPA fallback, the browser-view route this build does not serve, two routes authenticated by a token in the URL, the idempotent logout, and the inbound Teams webhook that carries its own signature auth. Unknown id: with credentials every parameterized GET asked for an id that does not exist answers below 500. 404 is the contract; 403, 400 and 200 are how some routes say it; a 5xx never is. One SSE stream is held open and listed; the two remote-artifacts routes that answer 502 and 503 for an unknown provider are listed against #14288 and come out when the status moves below 500. Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens from 25% to 95%. The conventions doc now says what that number is: every route asked for its guard and its fresh-home answer, not every route's behaviour pinned -- that is what the seam tests beside the sweep are for.
* test: sweep every parameter-less GET route for its guard and a fresh-home answer Fifth slice of the in-process integration layer. test/integration/ test_route_sweep.py reads every route the live router serves at boot (the new gw.registered_routes(), the same reading the coverage ratchet counts) and drives each parameter-less GET twice through the booted gateway. Guarded: without credentials the route answers 401 or 403. Every GET the router serves is swept, the SPA shell and its assets included; the eight that answer otherwise are listed with the status each answers and the reason (the shell, favicon and logo an unauthenticated browser needs for the login; the health, live and ready probes; the pre-login theme flags; the browser-view route this build answers 404 to everyone). A route that is not listed and answers outside 401/403 is a guard regression, and one that is listed and moves off its status is a broken probe or asset. Serves: with credentials the route does not fail with a 5xx on a fresh home; the six that answer 503 on a fresh home are listed with what each names as unavailable, and a 500 is never a contract. Exclusions are exact paths with a reason, never patterns: two routes hold the connection open (the SSE stream and the log long-poll), one fetches the official app catalog over the network (the rootdir conftest fences it), and two shell the real cloud CLI (the preflight against the operator's profile; the deploy profile listing). For that last one the harness now also points the CLI at credential files that do not exist under the test home, so no test in the layer can exercise a developer's AWS account. 383 routes are swept. Route coverage moves from 9 to 383 of 1247 (30.7%); the ratchet tightens from 0.5% to 25%. The conventions doc gains the sweep rule. The harness also grants the disposable home unsandboxed-exec consent, as the E2E suite does: the CI container has no sandbox backend, and without it every child the gateway spawns is refused before the fake model runs. * test: sweep every mutating and parameterized route for its guard (#14290) Sixth slice of the in-process integration layer, extending the route sweep to the rest of the router. Two more sweeps in test/integration/test_route_sweep.py, over every route that is not a parameter-less GET (865 of the 1248), each path token replaced by a stand-in that matches nothing on a fresh home. Guarded: without credentials every mutating route and every parameterized route answers 401 or 403, the mutating ones sent with no body, so the guard is proven to run before any handler could act. The six that answer otherwise are listed with the status and the reason: the SPA fallback, the browser-view route this build does not serve, two routes authenticated by a token in the URL, the idempotent logout, and the inbound Teams webhook that carries its own signature auth. Unknown id: with credentials every parameterized GET asked for an id that does not exist answers below 500. 404 is the contract; 403, 400 and 200 are how some routes say it; a 5xx never is. One SSE stream is held open and listed; the two remote-artifacts routes that answer 502 and 503 for an unknown provider are listed against #14288 and come out when the status moves below 500. Route coverage moves to 1235 of 1247 (99%) and the ratchet tightens from 25% to 95%. The conventions doc now says what that number is: every route asked for its guard and its fresh-home answer, not every route's behaviour pinned -- that is what the seam tests beside the sweep are for.
Problem / Motivation
Goal: Every mutating and every parameterized route the gateway serves is checked for its guard, and every parameterized GET for a non-5xx answer to an unknown id, on every CI run.
#14269 swept the parameter-less GETs. The other 865 routes -- every POST/PUT/PATCH/DELETE and every route with a path token -- still had no fleet-wide guard check, and no check that an unknown id answers as a client error rather than a server failure.
Why it matters
An unguarded mutating route is the worst kind of leak (it acts); a 5xx for an unknown id reads as an outage to any monitor or client.
Not a goal
What changed (motivation -> approach -> change)
Sixth slice of the in-process integration layer (harness #13621; memory seams #13736; spawn admission #14160; streaming #14258; parameter-less GET sweep #14269). Two more sweeps in
test/integration/test_route_sweep.pycover the rest of the router: every route that is not a parameter-lessGET(865 of 1248), with the stand-insweep-probesubstituted for every{token}.Stacked on #14269 (base
test/integration-route-sweep): it extends that PR's file and uses itsgw.registered_routes(). Retarget as the stack lands. The diff shown is this slice alone.Tracks #14288 (the two 5xx-for-unknown-id routes; the table entries come out when it is fixed, this PR does not close it).
The two new contracts
UNGUARDED_OTHER:GET /{name}-> 200 (SPA fallback);GET /browser-view/{tail}-> 404;GET /artifact-app/{slug}/{token}/{path}andGET /sandbox-doc/{doc_id}/{token}-> 404 (token in the URL is the credential);POST /api/auth/logout-> 200 (idempotent);POST /api/messaging/teams-> 503 (inbound webhook with its own signature auth; channel not enabled)GET(127)HELD_OPEN_PARAM:/api/sessions/{id}/agents/{agent_id}/stream(SSE);UNKNOWN_ID_ANSWERS_5XX:/api/remote-artifacts/{provider}/browse(503) and/api/remote-artifacts/{provider}/{external_id}(502) -> #14288Observed distribution for the unknown-id sweep on a fresh home: 404 x57, 403 x36, 400 x17, 200 x14, plus the three listed.
What the coverage number means
pytest test/integrationalone now requests 1235/1247 routes (99%);.github/workflows/ci.ymltightens--minfrom 25 to 95 (the only workflow change). The 12 the metric still misses are duplicate registrations of the same canonical path (the hit is credited to the first resource that matches).docs/system-specs/common/testing-conventions.mdnow states what the ratchet measures: every route has been asked for its guard and its fresh-home answer, not that every route's behaviour is pinned -- the seam tests beside the sweep (bugs 11-16, 20) are that, one route at a time.Tests
pytest test/integration -n 2 --dist loadgroup-> 25 passed, 37s (the four sweeps together ~22s).Backwards compatibility
Compatible: tests and a CI ratchet only; no product code changes.