Epic: cascade as the verified-default route-setup path (churn-safe, leak-free, soak-proven)

[0pcom]

Goal

Make cascade the verified-default route-setup path: multihop, mux, and rotating-policy routes carry traffic reliably and reclaim their rules/IDs under sustained churn — proven by a live soak harness, with cascade matching or beating legacy through the same intermediates.

Subsumes the parked "multihop route-setup liveness-gate", "route-setup robustness under deploy-churn", and "routing examination" threads.

Context (from a full routing-system audit)

• Cascade and legacy install byte-identical rules — the "cascade multihop data-plane bug" in pkg/visor/visorconfig/v1.go is not a cascade rule defect; the "rules installed but no traffic" symptom is one of the generic data-plane modes below and affects legacy too.
• The audit surfaced real, cascade-independent bugs that bite the current default (legacy) path, plus prerequisites for safely enabling cascade.

"Rules installed but no traffic" — enumerated (cascade-independent)

(a) dead onward transport at an intermediary; (b) black-hole beyond hop 0 (source can't see it — the cascade report); (c) rule/ID mismatch; (d) appended mux leg's handshake lost → leg never scheduled; (e) keepalive gap → idle rule expiry; (f) consume-rule resolves before route-group registration (datagram drops); (g) mux selector desync after prune → flow hashed to a removed leg index.

Acceptance = one soak harness going green

A soak run over a ≥3-hop path with mux=4 + spread-bw (15s rotation), for 30–60 min, asserting:

1. Continuity — a steady traffic probe sees zero black-hole windows across rotations/self-heals.
2. No leak — per-relay rule count and nextID high-water stay bounded over the whole run.
3. Bounded setup — p99 route-setup latency under a ceiling; no minutes-long stalls.
4. A/B parity — cascade through a set of intermediates carries traffic iff legacy does through the same intermediates.

Phases (each a PR set, each gated by the harness)

Phase 0 — Observability + soak harness (prerequisite)

• Instrument routing-table nextID high-water + per-relay rule counts (exposed via RPC/stats).
• Route-continuity probe (steady traffic, black-hole-window detection).
• Soak script: multihop + mux(4) + spread-bw rotation, collects the 4 metrics; runs the cascade A/B and definitively closes the "is it a bug" question.

Phase 1 — Churn correctness (cascade-independent; ship now)

• Rotation/prune leg retirement sends a Close packet downstream so intermediaries reclaim rules immediately instead of waiting 10 min (route_group.go:1404-1460) — the dominant churn leak. Mainnet milestone1 #1
• Reserved-ID un-reserve / TTL so partial-setup reservations don't permanently consume nextID (id_reserver.go).
• Mux schedule-rebuild barrier so no flow is scheduled onto a pruned leg index (mode g).
• Bounded interactive-dial ctx (app dials pass context.Background() today — rpc_ingress_gateway.go:246).
• Gate: harness metric 2 flat, metric 1 no black-holes, metric 3 bounded — on the current legacy default.

Phase 2 — Cascade hardening (prereq to flipping the default)

• Split cascade/legacy into bounded sub-budgets within the caller's remaining ctx (they share one today → timeout stacking, wrappers.go:194-206).
• Give cascade ACK waits a ctx.Done() case (cascade_handler.go:287 ignores ctx).
• Typed cascade failure classification (definitive → fast-fail, no legacy; transient → fall back) + make the dmsg/vstream fallback paths symmetric (wrappers.go:194-211 vs :240-249).
• Gate: harness metric 4 (A/B parity) + metric 3 with cascade on.

Phase 3 — SetupPolicy + default flip

• Caller-aware SetupPolicy threaded into the setup loop (interactive = fast-fail/tight/bounded; background heal/rotation = patient + bounded concurrency + warm alternate pool), replacing hardcoded maxRetries=6 + fixed timeouts.
• Flip EnableCascadeRouteSetup default → true.
• Gate: full soak green with cascade as default, no regression vs the Phase-1 baseline.

Because merge = fleet deploy, each phase self-validates on real traffic within ~10 min of merging; the harness is the pass/fail signal.