feat: quarantine known flakes, webhook notifications and test-tracking docs (P4)

[calvin-archastro]

Merge order: #152 is merged; nothing else first. After merging: deploy aster-srv first, then release aster and bump ASTER_VERSION (see Rollout).

P4 of test tracking: T22 quarantine store and feed, T23 enforcement, T26 webhooks, T25 docs.

Rebased onto #149: quarantine matching uses #149's shard-aware test identity (stream::test_id takes the report's project), so a quarantined test matches on any shard; the guide documents balanced partitions.

What changes

classifier pass ──▶ quarantine (enter / release / expire / broken) ──▶ quarantine_log
      │                                                                   │
      └──▶ webhooks.deliveries (same txn) ──▶ sender, signed, backoff     │
aster (CI) ◀── GET /v1/test-results/quarantine (versioned by log id) ◀────┘
   + aster-quarantine.toml ──▶ target passes "(quarantined: N)" if all 4 conditions hold

1. Server (T22), at the end of each classifier pass:
   • A test that is not broken enters after definite flakes from 2 runs within 7 days.
   • It leaves after 20 consecutive passes, when it breaks, or after 30 days. After leaving, only later events count toward re-entry.
   • Settings: ASTER_SRV_QUARANTINE_RUNS (0 = off), _RELEASE_PASSES, _DAYS.
   • Every change is logged; the newest log id is the list's version.
   • Also new: GET /v1/test-results/quarantine, tests?state=quarantined, and a real /ui/quarantine page.
2. Client (T23): with capture on, Aster loads the server list and aster-quarantine.toml at start and prints the list's version and count.
   • The server list is cached in .aster/quarantine.json; the cache is used for up to 24h when the server is down.
   • A failed target passes only if all of these hold:
     • its results are complete;
     • every test whose last attempt failed is quarantined;
     • it exited with its runner's "tests failed" code (ExUnit 2, nextest 100, others 1; never Gradle, Maven or cargo test);
     • ASTER_QUARANTINE is not off.
   • When a target passes this way:
     • its dependents run, and the pass is not cached;
     • each let-through failure gets a ::warning and a step-summary row;
     • the results file marks it "quarantined": true.
3. Webhooks (T26): ASTER_SRV_WEBHOOKS lists endpoints and the events each receives (test.quarantined|released|expired|broken).
   • Deliveries are recorded in the classifier's transaction (outbox).
   • Each is signed with X-Aster-Signature: sha256=… and retried from 30s, doubling to 6h, then abandoned after 12 attempts.
4. Docs (T25):
   • New docs/test-tracking.md.
   • Updated: the README's quarantine section, GETTING_STARTED, and the server README (quarantine, webhooks, settings, runbook).
   • Tests parse the documented TOML and JSON examples.

Open choice

The auto-quarantine threshold is still open in the spec. I built the proposal (2 runs) and made it configurable: ASTER_SRV_QUARANTINE_RUNS=1 quarantines after one proven flake, without a release.

Rollout

1. Merge, then deploy aster-srv. Nothing needs setting; it starts filling the list. Check /ui/quarantine for a day.
   • If the list looks wrong, set ASTER_SRV_QUARANTINE_RUNS=0.
2. Release aster, then bump firstlanding's ASTER_VERSION. CI then honours the list.
   • ASTER_QUARANTINE=off is the kill switch for a job.
3. Optional: set ASTER_SRV_WEBHOOKS as a Pulumi secret.

Verification

• Checks: cargo fmt, clippy -D warnings and cargo doc -D warnings are clean.
• Server tests against Postgres: all pass, including 10 new ones in tests/quarantine.rs:
  • entry, the 7-day window and the threshold;
  • broken tests are never quarantined, and a quarantined test leaves when it breaks;
  • release and re-entry, and expiry;
  • repository isolation;
  • signed delivery to a local receiver, backoff then abandonment, and unsubscribed endpoints.
• Client end to end with real go test:
  • quarantined-only failures pass and their dependent runs;
  • mixed failures still fail and say why;
  • the off switch works;
  • a bad file is an error.
• Unit tests: the conditions, exit codes, repo-file parsing, cache fallback, a fetch from a local server, stream marking and summary labels.
• Full workspace: all tests pass except the known local dev_services flake.
• Perf: on the 11.7M-result copy, 2,000 quarantined tests with six days of history take about 1.8s per pass (about 1 ms per test), and entry for 2,000 tests takes under 0.5s.
• Not done: I did not run this against production or a real ExUnit suite; the ExUnit exit code (2) comes from ExUnit's documented behaviour.

🤖 Generated with Claude Code

[calvin-archastro]

ArchDev is watching this pull request. State: watching (cycle 3 of 3 for head fb75912).

when	trigger	decision	result
2026-10-15T13:48:53.823Z	review feedback	enqueue job_782c44596856f1b636f4f784	published e3f67d7
2026-10-15T14:26:31.855Z	review feedback	enqueue job_9c8087862028ebeada38cf88	published 06b81ec
2026-10-15T19:30:18.333Z	merge conflict	enqueue job_bc92db1e40c62b552c756308	cancelled pull request converted to draft

Paused by the no-autofix label or a draft; never resolves reviewer threads. Jobs: job_782c44596856f1b636f4f784, job_9c8087862028ebeada38cf88, job_bc92db1e40c62b552c756308.