diff --git a/.gitattributes b/.gitattributes index fb927743..d46549a0 100644 --- a/.gitattributes +++ b/.gitattributes @@ -48,3 +48,18 @@ # entire value is that one specific byte survives. scripts/ci-check-pr-separation.js binary tests/watcher/sonnet/run-tests.js binary + +# scripts/gateway/chain.js also carries a literal NUL byte -- the hash-chain's own +# field-separator sentinel inside a .join("\0") call, so no legitimate field value +# can collide across two different joins. Unlike the two fixtures above, this +# file's whole point is that its CODE gets reviewed, so it must stay diffable. +# `binary` (the directive used for those two files) is the wrong tool here: it is +# shorthand for `-diff -merge -text`, so copying it verbatim would leave chain.js +# permanently in the exact same "Binary files ... differ" state the plain +# NUL-byte auto-detection already produces today -- it would not fix the +# reviewability problem, it would lock it in. `diff` (bare, positive) is narrower: +# it does not touch -text/-merge, it only tells git's diff engine to treat this +# one path as text for diff purposes despite the byte that trips the binary +# heuristic, so a real line-based diff renders on GitHub. Verified locally via +# `git check-attr` and `git diff` before this attribute was added and after. +scripts/gateway/chain.js diff diff --git a/KNOWN-LIMITATIONS.md b/KNOWN-LIMITATIONS.md index 359e738c..83d5e7fc 100644 --- a/KNOWN-LIMITATIONS.md +++ b/KNOWN-LIMITATIONS.md @@ -115,5 +115,67 @@ There is no `fs.flock`, no `fcntl`, no `O_EXLOCK`, and no `LOCK_*` constant on a *Control:* documented here, in `scripts/gateway/agent-transport.js`'s own header comment, and in the project's decision log (external panel brief + all five responses). +## 11. Gateway crash-recovery and idempotency fencing has a deliberately narrow guarantee, an operator-in-the-loop failure mode, and one untested layer (added 2026-09-09) + +**Why this exists.** Two Codex findings on PR #29 exposed real gaps in `scripts/gateway/`: (Finding 1) a `tools/call` result lived only in memory until the whole agent connection closed, so a gateway crash between "downstream returned a result" and "the agent disconnects" silently dropped that evidence from the signed chain; (Finding 2) nothing tied a retried `tools/call` to its original attempt, so a timeout-then-retry on a side-effecting tool (email, payment, ...) could execute the downstream side effect twice. Per governance decision, this was not fixed unilaterally by the team that wrote the original code — an independent 5-model external panel (openai/gpt-6-astra, x-ai/grok-4.5, google/gemini-3.1-pro-preview, deepseek/deepseek-v3.2, qwen/qwen3-max) reviewed the problem and candidate designs first; this project's decision log and `option-c-hardened-design.md` carry the full brief, all five raw responses, and the synthesized design's documented divergences from each panelist. + +**What's implemented.** `scripts/gateway/recovery.js`: a separate, unsigned, local-only store under `state_dir/gateway-recovery/` — an append-only per-connection WAL (`active/.jsonl`) recording `SESSION_START`/`INITIALIZE`/`CALL_START`/`CALL_RESULT`/`CLOSING` events, and one intent file per logical `(connectionId, tool, canonicalized-arguments)` operation (`intents/.json`) with a two-state terminal model — `dispatched -> completed | ambiguous` (no "failed": an explicit downstream error does not prove no side effect occurred, since a tool can commit an effect and still report failure). `proxy.js` consults an existing intent before ever dispatching a `tools/call`: a `completed` intent replays its cached result only if the retry presents the same caller-supplied idempotency key (`params._meta.idempotencyKey`) the original dispatch carried (JSON-RPC error `-32080 GATEWAY_AMBIGUOUS_RETRY` if still `dispatched`, `-32081 GATEWAY_DOWNSTREAM_OUTCOME_UNKNOWN` if `ambiguous`, `-32082 GATEWAY_TOOL_QUARANTINED_PENDING_OPERATOR_REVIEW` if a different connection this gateway's startup recovery flagged `pendingOperatorReview` still has an unresolved intent for this same tool — see the 2026-09-10 update below for both). `gateway.js#recoverCrashedSessions` runs once at startup, after the writer-claim is acquired and before any downstream connection starts, replaying any WAL a crashed prior instance left behind through `session.js`'s real, unmodified recorder functions; a call with no `CALL_RESULT` in the WAL is resolved from its real intent record if `completed` (this is precisely how Finding 1's gap is closed — from proof, never a guess), and left for an operator if not. A freshly-crashed unresolved call's intent is typically still `dispatched` at this point, not yet `ambiguous` — `buildHealthStatus`'s `recovery.pending_operator_review` field lists only `ambiguous` intents (an intent this gateway itself later fenced, e.g. via `closeConnection` on a live retry), while a `dispatched` one is counted in `recovery.in_flight` instead. The startup log's own `RECOVERY_AMBIGUOUS_INTENT` line (naming the connection and giving the exact `recovery-resolve`/`recovery-abandon` invocation) is the authoritative discovery path for this specific case, not `pending_operator_review` — which remains the right place to check for anything already known-ambiguous from ongoing operation between restarts. `downstream.js` also propagates the same intent key to the downstream as a conventional `graphsmith/idempotency_key` `_meta` field on both stdio and HTTP legs, for a downstream that itself understands one. + +**What this deliberately does NOT guarantee — disclosed, not hidden.** + +- **Not exactly-once execution.** The gateway will not knowingly issue a second downstream call for an operation whose outcome it cannot yet prove, from first dispatch until an operator explicitly resolves it. A different, uncoordinated caller hitting the same downstream directly, or a downstream that retries internally, is outside this gateway's control either way — where a downstream honors the propagated idempotency key, protection is stronger; where it's a black box, "halt and require a human" is the guarantee, not "guaranteed no duplicate." +- **The `dispatched`/`ambiguous` blocking fence is scoped to `(connectionId, tool, canonicalized-arguments)` only**, not (per one panelist's broader proposal) every effectful tool call on a downstream server considered together. Two different tools, or the same tool with even slightly different arguments, are never fenced against each other. A downstream whose side effects have cross-call interactions (e.g. two differently-worded calls that both happen to charge the same invoice) is not protected by this mechanism. (`completed`-intent replay is scoped differently as of the 2026-09-10 update below — see there.) +- **An `ambiguous` intent is never auto-expired by a TTL.** This is deliberate (every panelist proposed some auto-expiry; this design rejects all of them) — an unresolved ambiguous intent durably blocks any identical retry until an operator runs `recovery-resolve` (with real knowledge of what actually happened) or `recovery-abandon` (an explicit "I don't know and I'm giving up on finding out" override). The failure mode this trades into: an operator who never runs either command leaves that one specific operation permanently un-retryable through this gateway, indefinitely. That is the intended trade — silently expiring the fence would reopen exactly the double-dispatch risk this feature exists to close. +- **`recovery-abandon` deletes the intent fence it gives up on.** If the abandoned operation actually did execute downstream, a later identical call is no longer recognized as a retry and can re-run it. This is a knowing, per-connection, operator-invoked tradeoff (never automatic, never global) — documented here and in `gateway.js#abandonConnection`'s own header comment — for the case where waiting for certainty is no longer worth it. +- **Downstream `_meta` propagation is advisory, not enforced.** The gateway does not verify a downstream actually honors `graphsmith/idempotency_key`; a downstream that ignores it is unprotected at that layer, though still protected by the gateway-side fence, which is the actual mechanism this guarantee depends on. +- **Tool-level quarantine (2026-09-10 update, cluster E2) is a startup snapshot, not a live-updated list.** `pendingOperatorReviewConnections` is computed once, at this gateway process's own startup, from `recoverCrashedSessions`' return value. Since `recovery-resolve`/`recovery-abandon` require the writer-claim this live process already holds, nothing else can actually change those connections' outcome while this process stays up (see `recovery.js#resolveIntentNotExecuted`'s own header comment) — so the snapshot cannot go stale in practice, but this is a property of the writer-claim's single-writer model, not something the quarantine check itself enforces or re-verifies per call. It also does not close the gap for a tool that becomes ambiguous from a live (not-yet-recovered) crash between this process's own startup and its own eventual restart — only a crash a PRIOR process's startup already discovered and recovered is quarantined this way. +- **Scoped to real `tools/call` dispatch only.** An agent-initiated `sampling/createMessage` request forwarded through the same dispatch branch is explicitly out of scope for both the fence and the `_meta` propagation (see `proxy.js`'s and `downstream.js`'s own comments on this) — this only matters for the (unusual) case of an agent, rather than a downstream server, sending a sampling request through this gateway. +- **Recovery is proven at the unit/integration level, not against a real killed OS process.** `tests/gateway/recovery/run-tests.js` exercises `recoverCrashedSessions`/`abandonConnection` against WAL and intent fixtures built with the exact same `recovery.js` primitives `proxy.js` itself uses, replayed through the real `session.js`/`chain.js` code (not a parallel test-only implementation) — including the specific Finding-1 gap shape (intent completed, WAL missing the `CALL_RESULT` line) and a simulated "recovery crashed again mid-cleanup" repeat run. What is NOT covered is a true end-to-end test that spawns the real gateway CLI, `SIGKILL`s it mid-call, and asserts a second real process recovers it: `scripts/writer-claim.js`'s lease staleness window is not exposed as a `gateway-config.json` field, so such a test would either need to wait out the real (multi-heartbeat) staleness window — slow and disclosed as deliberately out of scope for this PR's test suite — or add a config passthrough to `writer-claim.js`/`config.js`, which this PR deliberately does not touch (out of scope: this fix is `recovery.js` + its call sites, not a writer-claim redesign). The gap this leaves untested is narrow: whether an actual `SIGKILL`'d process leaves the same durable, fsync'd WAL/intent files on disk that these tests construct directly — the same durability primitives (`state-store.js`'s `atomicCreateExclusive`/`atomicOverwriteFile`, and `appendDurableLine`) are already covered end-to-end for `chain.jsonl` by `tests/gateway/chain/run-tests.js`'s own "process killed mid-append" case (SS8 test plan item 12). + +*Control:* documented here, in `option-c-hardened-design.md` (full panel brief, all five raw responses, and the synthesized design's point-by-point divergence from each), in `recovery.js`'s and `gateway.js#recoverCrashedSessions`/`abandonConnection`'s own header comments, and exercised by `tests/gateway/recovery/run-tests.js` and the Option-C-specific additions to `tests/gateway/proxy/run-tests.js`. + +**Update (2026-09-09) — round-2 hardening from this PR's own review.** Six gaps found in the first implementation (all local to `recovery.js`/`gateway.js`/`proxy.js`'s own bookkeeping, none touching the identity/fencing model itself) are fixed: `walPath`/`intentPath` now reject a path-traversal-shaped `connectionId`/`intentKey` (relevant only to the `recovery-resolve`/`recovery-abandon` CLI's own flags and this module's exported API, not the gateway's own internally-generated IDs); the recovery directory is fsynced after every WAL/intent write, not just the file itself, matching `state-store.js`'s own convention; a `GATEWAY_BUNDLE_ID_COLLISION` during startup recovery is now content-verified (comparing the on-disk bundle's real `execution_trace` hash) before being trusted as "already durably appended," rather than assumed from the id match alone — `sealBoundaryBundle`'s `bundle_id` is a coarse fingerprint (`{init, grantedTools, n: calls.length}`) that two genuinely different crashed sessions could share; one connection's unreadable WAL/intent state (a permissions or I/O error, not a content one) no longer aborts recovery for every other connection in the same startup pass, and `recovery-abandon` can now quarantine (rename aside, not delete) such a connection without needing to re-read the state that made it unreadable in the first place; `recovery-resolve --confirmed executed` now requires `--result-file` (a missing one used to resolve with a `null` cached result, later replayed as an unverified clean success) and itself acquires the writer-claim (mirroring `recovery-abandon`, since both mutate the same intent files a live gateway's dispatch guard reads and writes); and `recovery-resolve --confirmed not-executed` now persists a terminal `not_executed` intent state instead of deleting the intent immediately, since deleting it left a crashed connection's still-pending WAL replay with nothing to distinguish "operator confirmed not executed" from "still unresolved" on the next restart — it would have been re-flagged for operator review forever. A dispatch blocked by the fence (`GATEWAY_AMBIGUOUS_RETRY`/`GATEWAY_DOWNSTREAM_OUTCOME_UNKNOWN`) is now also durably WAL-logged (a new `ANOMALY` event, replayed through `session.recordAnomaly` on recovery) rather than recorded in memory only, and both a blocked and a replayed call now get the same structured `gateway_call_completed` log line every other handled call gets. + +**Update (2026-09-10) — E1/E2 hardening from PR #33's own review, escalated to the frontier panel per project governance.** PR #33's AI review (CodeRabbit + Codex) raised two architectural findings against this built implementation that were escalated to the same 5-model panel rather than fixed unilaterally (see `graphsmith-pr33-panel-verdict-pr29-round4-halt-status-2026-09-10.md`); Paul decided both in favor of the panel's narrower-scoped option. + +- **Cluster E1 (identical-arguments over-fencing).** Two reviewers found that `(connectionId, tool, canonicalized-arguments)` alone cannot distinguish a genuine retry from two deliberate, intentional identical calls (e.g. two real `increment` calls) — the second used to silently receive the first's cached result instead of executing. Fixed: `completed`-intent replay now additionally requires the retry's `params._meta.idempotencyKey` to match the key the original dispatch carried. Without a match (no key, or a different key), the call dispatches as a new, independent operation — the completed intent record is superseded in place with a bumped `generation` counter under the same `intentKey` (still fenced against an in-flight/crashed retry of this new attempt), and the downstream-facing `graphsmith/idempotency_key` `_meta` value is suffixed with the generation (`.g`) so a downstream implementing its own dedup does not mistake this new, independent call for the earlier, unrelated one. The `dispatched`/`ambiguous` blocking branches are unconditional and unaffected by any key — an unproven-outcome retry is a real double-dispatch risk independent of caller intent, which is why the panel (and Paul) rejected requiring a key everywhere. +- **Cluster E2 (reconnect bypasses the fence).** A reviewer found that a reconnecting agent gets a brand-new `connectionId` after a crash, so its retry of "the same" logical operation computes a different `intentKey` than the still-unresolved intent its crashed predecessor left behind — the exact-key lookup never sees it. Fixed narrowly rather than via a full admission block or a stable cross-restart identity (both rejected — see the panel synthesis doc): before dispatching a genuinely new `intentKey`, the gateway also checks whether the SAME tool (any arguments, any connection) has an unresolved (`dispatched`/`ambiguous`) intent belonging to a connection this gateway's own startup recovery flagged `pendingOperatorReview` — if so, the tool is quarantined (`-32082 GATEWAY_TOOL_QUARANTINED_PENDING_OPERATOR_REVIEW`) until an operator resolves that connection. Deliberately scoped to `pendingOperatorReview` connections only: normal concurrent use of the same tool across unrelated, healthy live connections is unaffected. + +*Control:* `tests/gateway/proxy/run-tests.js`'s idempotency-key replay/no-key-supersede/mismatched-key and tool-quarantine cases (56/56 passing, along with the full existing `tests/gateway/*` and `tests/graphlint/gemini` suites, independently re-run against this exact change). + +**Update (2026-09-12) — round-3 hardening from PR #33's re-review after E1/E2 landed.** All local implementation gaps in `recovery.js`/`gateway.js`/`proxy.js`'s own bookkeeping (none second-guessing the E1/E2 design itself): the `INITIALIZE`, `ANOMALY`, and `CALL_RESULT` WAL appends were unguarded (unlike the already-guarded `CALL_START`/`SESSION_START` appends) — a WAL failure on any of them could escape `handleMessage`'s own "never throws" contract; `INITIALIZE` and `SESSION_START` (`openConnection`) now persist their WAL event *before* committing the corresponding in-memory state, rolling back cleanly on failure instead of leaving a "ghost" session or an unrecoverable initialized flag, while `ANOMALY`/`CALL_RESULT` (recorded after the real outcome is already known, with no "don't dispatch" option left) are now best-effort: logged, never escaping. If E1's generation-supersede overwrites a `completed` intent and the new dispatch's own `CALL_START` WAL append then fails, the rollback now *restores* the exact pre-supersede record (full idempotency key/cached result/generation) instead of deleting it outright — a plain delete would have let a later request carrying the ORIGINAL idempotency key dispatch again instead of replaying it. The E2 tool-quarantine scan (`recovery.listAllIntents`) now fails closed (blocks via the existing `-32082` response) rather than letting an unrelated corrupt/unreadable intent file anywhere on disk reject the whole request with a generic `-32603`. An agent reusing an in-flight JSON-RPC id for a different operation is now rejected before a durable intent is ever created for it (previously: intent created, then `session.recordCallStart`'s own duplicate-id check threw uncaught, orphaning that intent). An intent an operator resolved `not-executed` (but whose owning crashed connection has not yet finished being cleaned up) now gets its own dedicated block code (`-32084 GATEWAY_OPERATOR_CONFIRMED_NOT_EXECUTED`) instead of silently falling through two wasted attempts into the generic `-32080 GATEWAY_AMBIGUOUS_RETRY`. `recovery-abandon`'s own `GATEWAY_BUNDLE_ID_COLLISION` handling now applies the same content-verification `recoverCrashedSessions` already does — it used to treat every collision as an expected repeated attempt and could silently discard a genuinely different session's only remaining record. `listActiveConnections` now returns a sorted (not raw `readdirSync`) order, since `recoverCrashedSessions` consumes it directly for chain-append sequencing. `gateway-recovery/`'s directories and files (WAL/intent records, holding raw goals/arguments/results) are no longer created with the umask-dependent, world-readable `0755`/`0644` default. (Originally `0700`/`0600`; as of the 2026-09-17 round-2 review pass these use the same shared access model `startup-permissions.js` applies to those exact paths -- `DEFAULT_DIR_MODE` `0750` / `DEFAULT_FILE_MODE` `0640`, group-readable for same-group ops tooling -- because the per-write `0700`/`0600` re-tightening was silently undoing that startup pass on every subsequent WAL/intent/signature write.) `doStop`'s shutdown loop now isolates a per-connection `closeConnection` failure (e.g. a corrupt intent file) so the writer-claim is still released and other sessions still close, instead of that one failure escaping shutdown entirely and leaking a stale claim. `buildHealthStatus`'s `recovery` field and the startup `RECOVERY_AMBIGUOUS_INTENT` log now both name the actual intent key(s) involved (a new `recovery.dispatched` list alongside the existing `in_flight` count; real keys in the log's `recovery-resolve` command instead of a literal `` placeholder) rather than requiring an operator to hand-parse raw recovery files to find them. + +*Control:* `tests/gateway/proxy/run-tests.js` (84/84), `tests/gateway/recovery/run-tests.js` (106/106), and `tests/gateway/e2e/run-tests.js` (43/43, including a new real-process SIGTERM test that corrupts an intent file mid-shutdown and asserts the writer-claim is still released) — along with the full existing `tests/gateway/*` suite — independently re-run against this exact change. + +**Update (2026-09-16) — anomaly-cap boundary tests, and a WAL/disk-growth gap confirmed still open, not closed by this cap.** Round-1 fix-round follow-up on `MAX_ANOMALIES_PER_SESSION` (`session.js`), the cap on `session.anomalies` referenced above. Four things previously stated as design intent or left ambiguous by earlier review rounds are now directly tested against the real code, not assumed: + +- **The cap boundary itself, on `recordAnomaly` directly** (the earlier test suite only drove `recordUnmatchedResponseAnomaly` through it): the cap admits exactly `MAX_ANOMALIES_PER_SESSION` (1000) real entries plus one `ANOMALY_CAP_REACHED` terminal marker, then returns `false` and stops growing `session.anomalies` for every call after. +- **`recordAnomaly` consumes the cap's budget before, and independently of, the WAL append** — settling a point earlier review rounds disagreed on. `proxy.js`'s blocked-retry branch calls `session.recordAnomaly(...)` first, then attempts `recovery.appendWalEvent(..., { type: "ANOMALY" })` in its own guarded `try`/`catch`. Proven directly by forcing that WAL append to fail (simulated `ENOSPC`) and confirming the anomaly was already recorded in `session.anomalies` regardless — the in-memory/sealed-bundle accounting does not depend on the WAL append succeeding, or on it happening at all. +- **The cap bounds the in-memory/sealed-bundle list only — it does NOT bound the WAL, and this is by design, not an oversight** (the 2026-09-14 Cluster H panel's unanimous recommendation: the WAL is the durable, uncapped source of truth for crash recovery and must never tombstone an entry the cap evicted from memory). Proven directly against the real dispatch path: driving `MAX_ANOMALIES_PER_SESSION + 5` blocked-retry attempts on one connection plateaus `session.anomalies` at the cap while the WAL's `ANOMALY` event count keeps growing 1:1 with every attempt. +- **A reconnect resets the budget.** Per Cluster E2 above, a reconnecting agent gets a brand-new `connectionId` and therefore a brand-new `session` object with its own empty `anomalies` array — an exhausted session does not poison a later one. Proven directly: a session driven to its cap, followed by a fresh `createSession` for a new connectionId, starts at zero and fills its own full, independent budget. +- **LIMITATION, confirmed rather than assumed: the cap can suppress high-value evidence.** `MAX_ANOMALIES_PER_SESSION` is a plain event-count cap with no severity or evidentiary-value weighting — filling it with `MAX_ANOMALIES_PER_SESSION` low-value `UNMATCHED_RESPONSE` entries and then recording one `GATEWAY_AMBIGUOUS_RETRY` (a blocked-duplicate-dispatch attestation, exactly the kind of anomaly Option C's design depends on reaching the sealed bundle) shows the high-value entry is silently dropped, indistinguishable in outcome from one more piece of noise. This is not fixed by anything in this update; it is disclosed here as a real, demonstrated limitation of a plain count-based cap, not treated as closed. +- **The broader WAL/disk-growth risk is explicitly NOT closed by any of the above.** The blocked-retry branches skip `MAX_COMPLETED_CALLS_PER_SESSION` entirely (they return before that counter is ever touched) and still perform an unconditional, uncapped `ANOMALY` WAL append on every attempt; combined with the per-connection (not per-agent, not per-`state_dir`) scope of `MAX_ANOMALIES_PER_SESSION` and the fact that neither an event-count cap nor rotation alone bounds disk usage when event SIZE varies, total WAL disk growth remains unbounded for a client that loops a fenced retry (optionally reconnecting to reset any per-connection accounting). Tracked as **SAT-1119** ("WAL/disk-growth: no aggregate byte-budget or admission-policy bound on gateway-recovery WAL"), a real Linear ticket scoping the actual fix (WAL rotation/segmenting vs. a byte budget vs. a retry-path admission policy) — separate from, and explicitly not satisfied by, SAT-1096. SAT-1119 is scoped to the gateway-recovery WAL specifically; the chain-of-custody file (`chain.jsonl`) has its own, separate disk-growth story documented in §12 below — the two are not the same ticket and fixing one does not touch the other. + +*Control:* `tests/gateway/session/run-tests.js` (`recordAnomalyRespectsCapWithTerminalMarker`, `reconnectStartsWithAFreshAnomalyBudget`, `cappedAnomaliesCanSuppressAHigherValueLaterEvent`) and `tests/gateway/proxy/run-tests.js` (`blockedRetryAnomaliesAreUncappedInTheWalDespiteTheSessionCap`, plus the added assertion in `anomalyWalAppendFailureDoesNotEscapeHandleMessage`), along with the full existing `tests/gateway/*` suite, independently re-run against this exact change. SAT-1119 (Linear) for the open WAL/disk-growth gap. + +## 12. `chain.jsonl`'s read-side memory ceiling is removed; disk growth, rotation, and the cached-replay WAL gap remain open (added 2026-09-18) + +**Why this exists.** A 5-model frontier panel review of PR #33 (`graphsmith-pr33-round5-frontier-panel-verdict-2026-09-18.md`) surfaced two separate, real findings against `scripts/gateway/chain.js` and `scripts/gateway/proxy.js`, decided on and built in this round: + +- **Finding #8 — every read of `chain.jsonl` (`readChain`, `readChainForValidation`, and therefore `reconcileHead` and `gateway.js#buildHealthStatus`'s periodic health check) used to slurp the entire file into one JS string, split it, and JSON-parse every line into an in-memory array before doing anything else.** For a chain large enough, this could exceed V8's own maximum string length (roughly 512MB–1GB depending on Node version) and crash the process outright — not a hypothetical, since nothing rotates or bounds `chain.jsonl`'s own growth (see the SAT-1119 cross-reference above). Separately, the read itself was unrenewed: `reconcileHead`'s startup walk held the writer-claim across the whole read with zero lease-renewal coverage during that phase (renewal existed only for the per-entry hash-chain walk that runs *after* the read completes), so a sufficiently large chain could let the lease go stale and a competing claimant take over mid-read. +- **Finding #10 (bonus, not in the original panel scope) and Finding #5 — see below.** + +**What's implemented for Finding #8.** `chain.js#streamChainFileLines` (new, shared primitive) reads `chain.jsonl` in fixed-size byte chunks (default 1MB) and splits on the raw newline byte rather than decoding the whole file to a string first — peak memory for the read phase is `O(chunk size)`, not `O(file size)`, regardless of how large the chain grows. `readChain` and `readChainForValidation` (and therefore `reconcileHead` and `buildHealthStatus`) are rebuilt on top of it. An `onProgress(bytesReadSoFar)` hook fires after every physical read syscall — both call sites (`reconcileHead`'s startup walk and `buildHealthStatus`'s periodic health check, plus a third call site inside `verifyAndRepairBundleCollision` found during this same fix) now wire this into the writer-claim's `maybeRenew()`, so the lease is kept alive throughout the read itself, not only during the walk that follows it. A line whose byte length exceeds `MAX_ENTRY_BYTES` is now rejected with a distinct `GATEWAY_CHAIN_ENTRY_TOO_LARGE` code (previously mislabeled `CORRUPT_GATEWAY_CHAIN`, the catch-all for genuine content corruption, which would have sent an operator chasing tampering instead of an oversized-record/resource-exhaustion problem); a malformed-JSON line still throws `CORRUPT_GATEWAY_CHAIN`, unchanged. An unreadable file (real I/O failure other than "does not exist") throws `GATEWAY_CHAIN_READ_FAILED`, also a distinct code, not `CORRUPT_GATEWAY_CHAIN`. + +**What this deliberately does NOT fix.** Removing the read-side memory ceiling and adding read-phase lease renewal does not bound `chain.jsonl`'s own on-disk growth — nothing rotates, segments, or archives it, and there is no byte budget on the file itself (distinct from, and not addressed by, SAT-1119's WAL-specific scope above). A chain large enough to be slow to read is still read in full on every `reconcileHead` startup call and on every `buildHealthStatus` health-check tick; this fix makes that read memory-safe and lease-safe, not fast or bounded. Rotation/archival for `chain.jsonl` remains an open, unscoped future item. + +**A real, pre-existing correctness bug was found (not introduced) while writing this checklist's own tests, and is fixed as part of this same update.** `streamChainFileLines` read its target buffer into ONE `Buffer` allocated once and reused across every physical read syscall. `Buffer#slice` never copies -- it returns a view into the same backing memory -- so on the very first physical read of a stream, if that chunk did not end exactly on a newline, the carried-over `pending` bytes were left as a view into the SAME buffer the next `fs.readSync` call then overwrote in place, silently corrupting the assembled line the next time it was used (observed directly: `"schema_version"` decoded back as `"chechema_version"`). This could only ever manifest on a chain large enough to require more than one physical chunk read (the exact scenario this whole fix exists to make safe) — a chain small enough to fit in a single 1MB read, which is what every pre-existing `reconcileHead`/`readChain` test used, could never trigger it, which is why it shipped undetected until a test was written that deliberately forced many small physical chunks. Fixed by copying (`Buffer.from(...)`) instead of viewing whenever a chunk-derived buffer must survive past the next physical read; verified by reverting the one-line fix and confirming the new tests fail with exactly this corruption, then restoring it and confirming they pass. + +*Control:* `tests/gateway/chain/run-tests.js` — `streamChainFileLines`-level tests for multi-byte UTF-8 split across a chunk boundary (the test that caught the buffer-aliasing bug above), per-chunk renewal across many physical chunks, immediate abort when a mid-read renewal detects lost ownership (the competing-claimant-after-stale-lease case), a file with no trailing newline, and ENOENT/absent-file handling; `readChain`-level tests pinning the new `GATEWAY_CHAIN_ENTRY_TOO_LARGE` code for an oversized record and confirming genuine JSON corruption still reports `CORRUPT_GATEWAY_CHAIN`; and an integration-level test through the default public `readChain` API proving a real file spanning more than one physical 1MB chunk parses every entry correctly. Existing `reconcileHeadRenewsTheClaimPerEntryAcrossTheStartupWalk`/`reconcileHeadRenewsTheClaimAgainImmediatelyBeforeTheRepairWrite` tests were updated (not weakened) to account for the one additional read-phase renewal call a chain fitting in a single 1MB stream chunk now performs. Full existing `tests/gateway/*` suite (777 tests as of this update, up from 762 before this section's own 15 new chain tests) independently re-run against this exact change with zero regressions. + +**Finding #5 — the cached-replay WAL gap is disclosed and made observable, not closed.** `proxy.js`'s idempotency-replay branch (a duplicate/idempotent `tools/call` served from cache rather than re-dispatched) records its `CALL_START`/`CALL_RESULT` WAL events in-memory-first, WAL-second — deliberately fail-open, since the side effect already happened and this gateway has nothing left to refuse. If the `CALL_RESULT` append fails after `CALL_START` succeeded (or the process crashes between the two), a crash-recovery replay sees an orphaned `CALL_START` with no way to prove whether the agent actually received the cached result. Per Paul's explicit decision (choosing the "make the gap observable" design over closing it structurally, which would require inventing new durable state and was assessed by the panel as `CONTESTED`, not worth further panel spend): the replayed step's identity (`call_seq`) is now attested directly in the live `ANOMALY` event when a WAL append fails on this path, and — the actual fix, since crash recovery's own `recoverCrashedSessions` never seals a connection it flags for operator review — `gateway.js#abandonConnection` (the `recovery-abandon` CLI, the only path that unconditionally seals such a connection) now records an explicit `GATEWAY_RECOVERY_INCOMPLETE_REPLAYED_STEP` anomaly citing the exact step number for any pending call whose `CALL_START` was a cached replay. This does not eliminate the underlying gap — a crashed connection's replayed-but-unconfirmed step is still an open question an operator must resolve by hand — it ensures that gap is visible in the sealed bundle rather than silently indistinguishable from an ordinary interrupted call. + +*Control:* `tests/gateway/recovery/run-tests.js#abandonConnectionRecordsIncompleteMarkerForOrphanedReplayedStep` — constructs an orphaned replayed `CALL_START`, confirms `recoverCrashedSessions` flags it for operator review without sealing anything, then confirms `recovery-abandon` actually produces a sealed bundle whose `decision_record.md` contains the `GATEWAY_RECOVERY_INCOMPLETE_REPLAYED_STEP` marker citing the correct step. Verified to actually catch the gap it targets: temporarily reverting the fix makes this specific assertion fail, not just "some test somewhere." + --- *This document is publication-hygiene clean. See also the worked offline-verifier transcript in [`docs/examples/offline-verify.md`](docs/examples/offline-verify.md).* diff --git a/checks/register-gateway-sessions.js b/checks/register-gateway-sessions.js index f45b70ac..9bc1bc10 100644 --- a/checks/register-gateway-sessions.js +++ b/checks/register-gateway-sessions.js @@ -22,23 +22,114 @@ * "something was tampered with" (test plan item 17's explicit requirement). * It also checks HEAD.json consistency against the chain's own tail (test plan item 12: a * crash between chain.jsonl's append and HEAD.json's update, per SS3.6's specified write - * order, must be reported as an incomplete append needing recovery, never silently treated - * as "chain is fine, just short") and, when a bundle-existence check is supplied, that every - * chained bundle_id has a corresponding bundle file on disk. + * order, must be reported explicitly, never silently treated as "chain is fine, just short") + * and, when a bundle-existence check is supplied, that every chained bundle_id has a + * corresponding bundle file on disk. + * + * Round-2 fix-round follow-up (docs/contracts/chain-validity.md, "C2 -- Canonical Chain + * Validity", SS5 "one implementation, not three copies"): the structural walk below -- + * per-entry hash/link/sequence/genesis verification and the HEAD-vs-tail comparison -- no + * longer hand-rolls that logic. It delegates entirely to `chain.validateChain` (and, via + * that function, `chain.readChainTail`'s own torn-record handling), the SAME canonical + * validator `chain.reconcileHead` (startup reconciliation) and the append-time check inside + * `chain.appendSession` / `chain.repairMissingChainEntry` already use. C2 SS5 names this walk + * as the third of three call sites that must never quietly diverge on what "valid" means; + * this was the last of the three still re-deriving its own subset of the definition. + * `entryShapeOk` and `headShapeOk` below stay as intentionally-duplicated, hand-rolled + * copies of chain.js's own functions of the same name -- C2 SS5 explicitly permits this + * ("provably identical hand-rolled copies," the same convention this codebase already uses + * for register-retention.js vs register-gateway-sessions.js) -- they are used here only to + * filter the entries this module's own bundle-existence check (see below, outside C2's + * structural scope) walks, not to re-derive any hash/link/sequence verdict + * chain.validateChain already owns. + * + * A NOTE ON THE LEASE-KEEPALIVE LOOP BELOW (round-1 fix-plan item 1, landed the same + * round as this delegation; round-3 fix pass corrects a defect an independent verifier + * found in the round-2 landing described below): before the round-2 delegation, + * `ctx.maybeRenew()` was called once per chain entry from INSIDE this module's own + * hand-rolled hash/link/sequence loop. Round-2 replaced that loop with a small, + * decoupled pre-pass over `ctx.chain`, run BEFORE `chain.validateChain` was ever + * called -- reasoning that `chain.validateChain` should not be handed a caller-specific + * lease-renewal callback. That was wrong in practice: `maybeRenew()` is time-gated + * (scripts/writer-claim.js fires it at most once per heartbeatMs interval), so calling + * it N times in a tight synchronous pre-pass loop performs at most ONE real renewal, and + * the actual expensive validation work inside `chain.validateChain` then ran with + * effectively zero renewal coverage -- exactly the lease-starvation this fix exists to + * prevent. + * + * Fixed here (round-3) by giving `chain.validateChain` an optional `onEntry` hook (see + * its own doc comment in scripts/gateway/chain.js) that IT invokes once per entry, in + * order, from INSIDE its own per-entry hash/link/sequence loop -- `ctx.maybeRenew` is + * passed straight through as that hook below. This still keeps `chain.validateChain` the + * one canonical structural validator (C2 SS5) free of any lease-specific logic of its + * own: an injected callback is exactly the same shape `ctx.computeEntrySha256` and + * `ctx.bundleExists` already are at this call site, not a new mechanism. Renewal is now + * genuinely interleaved with the real per-entry work (see + * tests/gateway/chain/run-tests.js's new + * `walkGatewaySessionsMaybeRenewInterleavedWithValidateChainNotFrontLoaded`, which proves + * the call count is bounded by how far the walk actually got, not by `rawChain.length`), + * and a thrown lease-loss error aborts the walk at exactly the entry `onEntry` was called + * for -- STRICTLY no later, and often earlier, than the round-2 pre-pass could ever + * abort, so the pre-existing "abort immediately, mid-walk, on a detected lease loss" + * contract (`walkGatewaySessionsCallsMaybeRenewOncePerEntry` and + * `walkAbortsOnLeaseTakeoverMidWalkNotDowngradedToAFailedReport`) holds more strongly + * than before, not less. * * Discipline (mirrors register-retention.js exactly): - * - fail-closed: any broken link, sequence gap, bad shape, or head mismatch => failed. + * - fail-closed for genuine structural/integrity problems: bad entry/HEAD shape, a + * sequence gap, a broken or tampered link, an invalid genesis, a fork (equal seq with + * differing hash), HEAD ahead of a verified tail, or any HEAD lag worse than the exact + * single-step/hash-matching case => `failed`. A lagging or rebuildable HEAD that IS a + * genuine, hash-verified single-step ancestor of the verified chain tail -- or its + * torn-final-record counterpart -- is NOT corruption (docs/contracts/chain-validity.md + * (C2) SS2 rows 1/3/4/7/9, SS3's two-failure-class split): reported as `reconcilable`, + * distinct from both `failed` and `verified`, so a caller (an operator, or this same + * round's own divergence test comparing this walk against the append-time check) never + * conflates a self-healing pointer lag with genuine corruption. This walk never writes + * anything either way -- see the next bullet. * - no clock/randomness in the decision path; timestamps are evidence only, if present. * - Report contract: { status, evidence[], assumptions[], failure_domain? }; pure. + * - Round-1 fix-plan commit 6 / docs/contracts/chain-validity.md SS5: this walk (run + * by gateway.js#buildHealthStatus roughly every STATUS_WRITE_INTERVAL_MS) reports + * EVIDENCE ONLY and never mutates chain.js's own process-local admission latch + * (getChainIntegrityFailure) -- only chain.js's own detectors (the O(1) append-time + * checkHeadAgainstTailOrRepair, and reconcileHead's full walk at startup) ever latch. + * Concretely: a HEAD/tail POINTER divergence this walk finds is redundant with (and + * slower than) the append-time check, which already catches it immediately -- and, + * since round-2, is guaranteed to receive the identical classification, because both + * now delegate to the same chain.validateChain. Mid-chain INTERIOR corruption (a + * tampered/gap/broken-link entry where HEAD and the chain's own tail already agree) + * IS visible in this walk's own evidence, but nothing consumes that to stop admission + * -- an operator reading the written status output has to notice and act (restart, + * which re-runs reconcileHead's own full walk and hard-fails startup per Paul's + * 2026-09-15 decision), or a future, separately-scheduled deep walk would have to be + * built to close that gap. This is not an oversight to route around here: mixing a + * second, independent classification into the admission-latch path would violate + * SS5's "one authoritative structural validator" contract. + * - Round-1 fix-plan item 1 (generalized keepalive), round-2 fix pass: a lease loss + * DETECTED by this walk's own maybeRenew() call (see below) must not be downgraded to + * this module's own ordinary fail-closed "report it, evidence only" contract -- that is + * exactly the ban createLeaseGuard's own doc comment (scripts/gateway/gateway.js) + * states: a detected lease loss must never be relabeled as "flag for operator review, + * continue" handling. `ctx.isLeaseError`, injected by the same caller that injects + * `ctx.maybeRenew`, lets that caller's own leaseGuard recognize (by reference identity, + * not error.code) that a caught exception IS the one its own maybeRenew() just threw, + * and rethrow it so it propagates out of this walk entirely rather than being absorbed + * into a status object, mirroring recoverCrashedSessions'/reconcileHead's own + * rethrow-not-downgrade handling of the identical error class. * - Honest limit (A6, same as register-retention.js's): a privileged local attacker who * rewrites both the chain and its own HEAD.json is out of scope for THIS verifier alone * -- SG-FR-6 (remote anchoring, not yet implemented -- see scripts/gateway/chain.js's * pushChainTailToRemoteAnchor stub) is the intended mitigation, same class as * register-retention.js's own stated A6 limit. - * Zero-dep CJS, Node >= 18. Schema: schemas/gateway-session-entry.schema.json. + * Zero external-package dependencies (Node core plus the sibling scripts/gateway/chain.js + * for the canonical validator -- see above), Node >= 18. Schema: + * schemas/gateway-session-entry.schema.json. */ "use strict"; +const chain = require("../scripts/gateway/chain.js"); + const HEX64 = /^[0-9a-f]{64}$/; const ENTRY_KEYS = ["schema_version", "seq", "bundle_id", "prev_entry_sha256", "entry_sha256"]; @@ -59,11 +150,10 @@ function entryShapeOk(e) { const HEAD_KEYS = ["schema_version", "seq", "bundle_id", "entry_sha256"]; /** Same strict, fail-closed shape discipline as entryShapeOk (and scripts/gateway/ - * chain.js's own headShapeOk, which this deliberately mirrors -- this module stays - * dependency-free, per its own header, so it hand-rolls rather than requires it). Without - * this, a HEAD.json with a missing/wrong schema_version, extra fields, or an otherwise - * malformed shape could still slip through the tail comparison below and receive a clean - * "verified" verdict as long as its seq/bundle_id/entry_sha256 happened to match. */ + * chain.js's own headShapeOk, which this deliberately mirrors -- an intentionally- + * duplicated, provably-identical copy per C2 SS5, kept here only so this module's own + * bundle-existence pass below can be self-contained; it plays no part in the + * hash/link/sequence/HEAD verdict itself, which is entirely chain.validateChain's). */ function headShapeOk(h) { if (!h || typeof h !== "object") return false; for (const k of HEAD_KEYS) { @@ -77,13 +167,61 @@ function headShapeOk(h) { return true; } +/* Which of this module's two failure_domains a chain.validateChain "refuse" class maps + * to. "trusted-core" (mirrors this module's own pre-existing convention for a HEAD.json + * problem): the on-disk HEAD pointer itself disagrees with a verified tail in a way that + * is not the auto-catch-up case -- empty chain naming a HEAD, HEAD claiming a seq past the + * real tail, or a lag worse than the single exact-hash-verified step. "untrusted-input" + * (the fail() default, kept explicit here for clarity): a problem in chain.jsonl's own + * entries -- shape, genesis, sequence, link, hash, or a fork. */ +const REFUSE_DOMAIN_BY_CLASS = { + "empty-chain-with-head": "trusted-core", + "head-ahead-of-tail": "trusted-core", + "multi-step-lag": "trusted-core", + "head-not-ancestor": "trusted-core", + // Round-3 fix pass (verifier-flagged gap): not currently emitted by + // chain.validateChain itself (only by checkHeadAgainstTailOrRepair's append-time + // check and chain.reconcileHead's startup walk, neither of which this status walk + // calls) -- classified explicitly anyway so this map never silently falls through to + // the untrusted-input default for either class below if that ever changes. + // operational/trusted-core: chain.jsonl's tail could not be read at all -- a + // HEAD/tail-plumbing problem, mirroring this module's own HEAD.json convention, not + // evidence of tampered chain.jsonl content. + "tail-unreadable": "trusted-core", + "invalid-genesis": "untrusted-input", + "sequence-gap": "untrusted-input", + "broken-link": "untrusted-input", + "tampered": "untrusted-input", + "fork": "untrusted-input", + "malformed-interior": "untrusted-input", + // untrusted-input/refuse: a torn physical tail is chain.jsonl CONTENT that only + // startup reconciliation, under the exclusive writer-claim lease, may repair. + "torn-tail-requires-manual-truncation": "untrusted-input", +}; + /* ctx = { * chain: [gateway-session-entry], // ordered append-only log, as read from chain.jsonl * head: { schema_version, seq, bundle_id, entry_sha256 } | null, // HEAD.json's content - * computeEntrySha256: (entry) => hex64, // scripts/gateway/chain.js's function, injected - * // rather than required(), keeping this module - * // dependency-free like register-retention.js + * computeEntrySha256: (entry) => hex64, // scripts/gateway/chain.js's function. No + * // longer used internally (chain.validateChain + * // uses its own copy) -- still required on ctx + * // for API back-compat with every existing + * // caller/test, and as a basic shape guard. * bundleExists?: (bundle_id) => boolean, // optional: checked against each chain entry + * // that survived chain.validateChain's own walk + * // (outside C2's structural scope -- this + * // module's own addition, same as before). + * maybeRenew?: () => void, // optional, round-1 fix-plan item 1: the caller's own + * // writer-claim keepalive (via gateway.js's createLeaseGuard), + * // passed straight through as chain.validateChain's own + * // onEntry hook (round-3 fix pass) so it fires once per chain + * // entry, genuinely interleaved with chain.validateChain's own + * // per-entry work -- same injection discipline as + * // computeEntrySha256/bundleExists above, never required() here + * isLeaseError?: (error) => boolean, // optional, paired with maybeRenew: lets the + * // catch-all below recognize (by reference identity) that a + * // caught exception IS a detected lease loss and rethrow it + * // rather than report it as an ordinary failed walk * } */ function walkGatewaySessions(ctx) { const evidence = []; @@ -94,66 +232,111 @@ function walkGatewaySessions(ctx) { const fail = (msg, domain) => ({ status: "failed", evidence, assumptions, failure_domain: domain || "untrusted-input", reason: msg }); try { if (!ctx || typeof ctx !== "object") return fail("no context"); - const chain = ctx.chain; - if (!Array.isArray(chain)) return fail("chain must be an array"); + const rawChain = ctx.chain; + if (!Array.isArray(rawChain)) return fail("chain must be an array"); if (typeof ctx.computeEntrySha256 !== "function") return fail("ctx.computeEntrySha256 function is required"); - if (chain.length === 0) { - if (ctx.head) return fail("HEAD.json names a chain tail but chain.jsonl is empty -- incomplete/corrupt state", "trusted-core"); + // The one canonical structural validator (C2 SS1/SS5) -- the same function + // chain.reconcileHead and the append-time check (chain.appendSession / + // chain.repairMissingChainEntry) already delegate to. `ctx.head` is passed through + // as-is; chain.validateChain's own documented contract collapses "absent," "not an + // object," and "fails headShapeOk" to the same `null` case, exactly like its other + // two callers already do (see chain.js#reconcileHead's own head-read try/catch). + // + // Round-1 fix-plan item 1 (lease keepalive, generalized), round-3 fix pass (fixes + // the defect described in this file's header note above): `ctx.maybeRenew`, when + // provided, is passed straight through as chain.validateChain's own `onEntry` hook + // rather than run in a separate, disconnected pre-pass -- so it fires once per + // entry, genuinely interleaved with chain.validateChain's own hash/link/sequence + // work as that work actually happens, and a thrown lease-loss error aborts the walk + // at exactly the entry chain.validateChain had reached, never after the fact. + const validateOptions = typeof ctx.maybeRenew === "function" ? { onEntry: ctx.maybeRenew } : undefined; + // C2 rows 3/4: "absent", "not an object", and "fails headShapeOk" are one input case. + const result = chain.validateChain(rawChain, headShapeOk(ctx.head) ? ctx.head : null, validateOptions); + + if (result.status === "empty") { return { status: "not-applicable", evidence, assumptions, reason: "empty gateway-session log -- nothing to verify" }; } - let prevHash = null; - let prevSeq = null; - for (let i = 0; i < chain.length; i++) { - const e = chain[i]; - if (!entryShapeOk(e)) return fail(`entry[${i}] has an invalid shape/type -- refusing (fail-closed)`); - - const recomputed = ctx.computeEntrySha256({ schema_version: e.schema_version, seq: e.seq, bundle_id: e.bundle_id, prev_entry_sha256: e.prev_entry_sha256 }); - if (recomputed !== e.entry_sha256) { - return fail(`entry[${i}] (seq=${e.seq}) entry_sha256 does not match its own recomputed content -- TAMPERED entry, distinct from a sequence gap`, "untrusted-input"); - } + if (result.status === "refuse") { + return fail(result.reason, REFUSE_DOMAIN_BY_CLASS[result.class]); + } - if (i === 0) { - /* Board decision 2026-09-04, PR #29 review "require the verified chain to start - * at sequence one": a first entry with seq > 1 and a null predecessor passes - * every OTHER check here (its own hash recomputes correctly, and there is no - * prior entry for a broken-link check to compare against) even though it is - * exactly the missing-genesis-prefix failure mode this schema's contiguous, - * monotonic sequence contract exists to catch -- e.g. a chain.jsonl truncated - * from the front. Both halves of "this is genuinely the chain root" must hold. */ - if (e.seq !== 1) return fail(`entry[0] has seq=${e.seq}, expected 1 -- chain does not start at its genesis entry (missing prefix), distinct from a broken hash link`); - if (e.prev_entry_sha256 !== null) return fail("entry[0].prev_entry_sha256 must be null (chain root)"); - } else { - if (e.seq !== prevSeq + 1) { - return fail(`entry[${i}] seq=${e.seq} expected ${prevSeq + 1} -- SEQUENCE GAP (an entry was removed), distinct from a broken hash link`, "untrusted-input"); - } - if (e.prev_entry_sha256 !== prevHash) { - return fail(`entry[${i}] does not chain to entry[${i - 1}] -- broken/mutated link, distinct from a sequence gap`, "untrusted-input"); + // result.status is "valid" or "reconcilable" here: the chain itself hashed, linked, + // and sequenced correctly end to end (C2 SS1) -- only HEAD's relationship to that + // verified tail still differs between the two. Bundle-file existence is entirely + // this module's own addition, outside C2's structural scope (a moved/archived + // bundle says nothing about the CHAIN's own integrity) -- checked here, against + // every entry that survived chain.validateChain's own walk. entryShapeOk mirrors + // validateChain's own torn-tail carve-out (only the physical last element is ever + // exempt from the shape check), so this reproduces exactly the same "effective + // chain" validateChain itself just verified, without re-deriving its verdict. + const survivingEntries = rawChain.filter(entryShapeOk); + evidence.push( + `hash chain intact across ${survivingEntries.length} entry(ies); seq contiguous (append-only)` + + (survivingEntries.length > 0 ? `, starting at ${survivingEntries[0].seq}.` : ".") + ); + if (typeof ctx.bundleExists === "function") { + for (const e of survivingEntries) { + if (!ctx.bundleExists(e.bundle_id)) { + return fail(`entry (seq=${e.seq}) references bundle_id "${e.bundle_id}" with no corresponding bundle file on disk -- incomplete append or a deleted bundle`, "trusted-core"); } } - if (typeof ctx.bundleExists === "function" && !ctx.bundleExists(e.bundle_id)) { - return fail(`entry[${i}] (seq=${e.seq}) references bundle_id "${e.bundle_id}" with no corresponding bundle file on disk -- incomplete append or a deleted bundle`, "trusted-core"); - } - prevHash = e.entry_sha256; - prevSeq = e.seq; } - evidence.push(`hash chain intact across ${chain.length} entry(ies); seq contiguous (append-only), starting at ${chain[0].seq}.`); - const last = chain[chain.length - 1]; - if (!ctx.head) { - return fail(`chain.jsonl has ${chain.length} entry(ies) but HEAD.json is missing -- incomplete append (crash between the chain.jsonl write and the HEAD.json update), needs recovery`, "trusted-core"); - } - if (!headShapeOk(ctx.head)) { - return fail("HEAD.json has an invalid shape/type -- refusing (fail-closed)", "untrusted-input"); + if (result.status === "valid") { + evidence.push(`HEAD.json matches the chain tail (seq=${result.tail.seq}, bundle_id=${result.tail.bundle_id}).`); + return { status: "verified", evidence, assumptions }; } - if (!(ctx.head.seq === last.seq && ctx.head.bundle_id === last.bundle_id && ctx.head.entry_sha256 === last.entry_sha256)) { - return fail("HEAD.json does not match the chain's own tail entry -- incomplete append (crash between the chain.jsonl write and the HEAD.json update), needs recovery, never treated as merely short-but-fine", "trusted-core"); - } - evidence.push(`HEAD.json matches the chain tail (seq=${last.seq}, bundle_id=${last.bundle_id}).`); - return { status: "verified", evidence, assumptions }; + // result.status === "reconcilable": a lagging/rebuildable HEAD, or a torn final + // record, that chain.validateChain itself classifies as NOT corruption (C2 SS2 rows + // 1/3/4/7/9; SS3's second bullet, "repair and continue, loudly"). Reported as its + // own distinct status, never folded into `failed` -- this walk still never repairs + // anything itself (only chain.reconcileHead, under the writer-claim lease, writes + // HEAD.json); it only makes sure this state is visible, and visibly DIFFERENT from + // genuine corruption, per C2 SS3's "two failure classes stay distinct everywhere." + if (result.anomaly) evidence.push(`anomaly: ${result.anomaly}`); + + // Round-3 fix pass (defect 2, docs/contracts/chain-validity.md C2 SS2 row 4): a + // headAction of "rebuild" means chain.validateChain found HEAD.json absent/unusable + // with no prior pointer to advance -- row 4 says only STARTUP RECONCILIATION, under + // the exclusive writer-claim lease (chain.reconcileHead), may treat that as + // repairable; every other call site, including this lease-less, periodic status + // walk, must REFUSE it instead. chain.reconcileHead's own use of "rebuild" stays + // correct and is untouched by this -- the mapping below exists only at THIS call + // site's own reporting boundary, per C2's explicit per-call-site distinction. The + // row-7 single-step-lag case (headAction: "advance", a genuine hash-verified + // ancestor with a real prior pointer to advance) is unaffected and stays + // "reconcilable" below, exactly as before. + if (result.headAction === "rebuild") { + return fail(result.reason, "trusted-core"); + } + return { + status: "reconcilable", + evidence, + assumptions, + reason: result.reason, + headAction: result.headAction, + truncatedTail: Boolean(result.truncatedTail), + }; } catch (e) { + /* Round-1 fix-plan item 1 (generalized keepalive), round-2 fix pass: a lease loss + * DETECTED by this walk's own maybeRenew() call above must not be downgraded to this + * function's own ordinary fail-closed "report it, evidence only" contract -- that is + * exactly the ban createLeaseGuard's own doc comment (scripts/gateway/gateway.js) + * states: a detected lease loss must never be relabeled as "flag for operator + * review, continue" handling. `ctx.isLeaseError`, injected by the same caller that + * injects `ctx.maybeRenew`, lets that caller's own leaseGuard recognize (by + * reference identity, not error.code -- see createLeaseGuard's own doc comment for + * why) that THIS exception is the one its own maybeRenew() just threw, and rethrow + * it here so it propagates out of this walk entirely rather than being absorbed into + * a status object, exactly mirroring recoverCrashedSessions'/reconcileHead's own + * rethrow-not-downgrade handling of the identical error class. Optional, like + * `ctx.maybeRenew` itself: a caller with no writer-claim in play never sets it, and + * every exception (this walk's own, or any other injected function's) keeps falling + * through to the ordinary fail-closed report below, unchanged. */ + if (typeof ctx.isLeaseError === "function" && ctx.isLeaseError(e)) throw e; return { status: "failed", evidence, assumptions, failure_domain: "trusted-core", reason: "exception during gateway-session walk -- failing closed: " + (e && e.message ? e.message : String(e)) }; } } @@ -182,29 +365,34 @@ if (require.main === module) { const e1 = mk(1, null, "gsa-0000000000000001"); const e2 = mk(2, e1.entry_sha256, "gsa-0000000000000002"); const e3 = mk(3, e2.entry_sha256, "gsa-0000000000000003"); - const chain = [e1, e2, e3]; + const chainEntries = [e1, e2, e3]; const head = { schema_version: "1.0", seq: 3, bundle_id: e3.bundle_id, entry_sha256: e3.entry_sha256 }; - const good = check.run({ chain, head, computeEntrySha256 }); + const good = check.run({ chain: chainEntries, head, computeEntrySha256 }); // tamper the middle entry's own hash (does not recompute). const tampered = [e1, { ...e2, entry_sha256: h(9) }, e3]; const brokenSelf = check.run({ chain: tampered, head, computeEntrySha256 }); // delete the middle entry -> sequence gap, distinct failure text from brokenSelf. const gapChain = [e1, e3]; const gap = check.run({ chain: gapChain, head: { ...head, seq: 3 }, computeEntrySha256 }); - // HEAD.json missing -> incomplete append. - const noHead = check.run({ chain, head: null, computeEntrySha256 }); - // HEAD.json stale (crash before update) -> incomplete append. - const staleHead = check.run({ chain, head: { schema_version: "1.0", seq: 2, bundle_id: e2.bundle_id, entry_sha256: e2.entry_sha256 }, computeEntrySha256 }); + // HEAD.json missing entirely, but the chain itself is fully valid -- C2 row 3/4: + // chain.validateChain itself classifies this as headAction "rebuild", but this + // call site's own round-3 defect-2 fix (see above) maps that to `failed`: only + // startup reconciliation under the exclusive writer-claim lease may treat an absent + // HEAD as repairable, never this lease-less status walk. + const noHead = check.run({ chain: chainEntries, head: null, computeEntrySha256 }); + // HEAD.json stale by exactly one hash-verified step (crash before the HEAD update) + // -- C2 row 7: reconcilable (auto-advance), not corruption. + const staleHead = check.run({ chain: chainEntries, head: { schema_version: "1.0", seq: 2, bundle_id: e2.bundle_id, entry_sha256: e2.entry_sha256 }, computeEntrySha256 }); // bundle file missing for a chained entry. - const missingBundle = check.run({ chain, head, computeEntrySha256, bundleExists: (id) => id !== e2.bundle_id }); + const missingBundle = check.run({ chain: chainEntries, head, computeEntrySha256, bundleExists: (id) => id !== e2.bundle_id }); const pass = good.status === "verified" && brokenSelf.status === "failed" && /TAMPERED/.test(brokenSelf.evidence.join(" ")) && gap.status === "failed" && /SEQUENCE GAP/.test(gap.evidence.join(" ")) && - noHead.status === "failed" && /incomplete append/.test(noHead.evidence.join(" ")) && - staleHead.status === "failed" && /incomplete append/.test(staleHead.evidence.join(" ")) && + noHead.status === "failed" && /HEAD\.json is absent/.test(noHead.evidence.join(" ")) && + staleHead.status === "reconcilable" && missingBundle.status === "failed" && /no corresponding bundle file/.test(missingBundle.evidence.join(" ")); console.log( @@ -213,7 +401,7 @@ if (require.main === module) { "tampered-detected=" + (brokenSelf.status === "failed"), "gap-detected=" + (gap.status === "failed"), "no-head-detected=" + (noHead.status === "failed"), - "stale-head-detected=" + (staleHead.status === "failed"), + "stale-head-reconcilable=" + (staleHead.status === "reconcilable"), "missing-bundle-detected=" + (missingBundle.status === "failed") ); process.exit(pass ? 0 : 1); diff --git a/docs/contracts/chain-validity.md b/docs/contracts/chain-validity.md new file mode 100644 index 00000000..3088bf77 --- /dev/null +++ b/docs/contracts/chain-validity.md @@ -0,0 +1,239 @@ +# C2 — Canonical Chain Validity + +> **Status:** binding design contract for PR #33's crash-recovery/idempotency +> follow-up work. Referenced by commit 1 (lease keepalive scope), commit 3 (chain +> validator + `readChainTail`), commit 4 (`reconcileHead` + append-time check), and +> the existing status/health walk (`checks/register-gateway-sessions.js`). Locked by +> the 2026-09-15 fix-round panel (see +> `claude/graphsmith-fix-round-plan-round1-2026-09-15.md` in the GraphSmith project), +> including Paul's 2026-09-15 binding decisions on startup posture and mid-run +> lagging HEAD referenced throughout. Do not reopen without a new decision record. + +This contract defines **one** canonical notion of "is the gateway-session chain +valid," to be used identically by three call sites that must never quietly diverge +on the definition of "valid": + +1. **Startup reconciliation** (`chain.reconcileHead`, commit 4) — runs once, after + `writerClaim.acquire()`, before the heartbeat starts and before anything else can + append. +2. **The append-time check** (inside `chain.appendSession` / + `chain.repairMissingChainEntry`, commit 4) — runs on the hot append path, cheap, + checking only the tail against HEAD. +3. **The status/health walk** (`checks/register-gateway-sessions.js`'s + `walkGatewaySessions`, existing) — runs periodically (currently piggybacked on + the ~10s status-write timer) and reports operator-facing evidence. + +Hash-and-link verification alone (each entry's `entry_sha256` recomputes, and each +entry's `prev_entry_sha256` matches the prior entry's `entry_sha256`) is **not +sufficient** to call a chain valid — a chain can be internally hash-consistent while +still containing sequence gaps, duplicate sequence numbers, a wrong genesis, or an +invalid interior shape. The validator below is the full check; no call site is +permitted to reimplement a subset of it under its own logic. + +## 1. The canonical structural validator + +Given `chain` (the ordered array of parsed `gateway-session-entry` records read from +`chain.jsonl`, schema `schemas/gateway-session-entry.schema.json`) and `head` (the +parsed contents of `HEAD.json`, or `null` if the file does not exist), a chain is +**valid** iff all of the following hold: + +- **Per-record schema.** Every entry has exactly the five required fields + (`schema_version`, `seq`, `bundle_id`, `prev_entry_sha256`, `entry_sha256`), no + extra fields, `schema_version` equal to the current schema version, `bundle_id` a + non-empty string, `prev_entry_sha256` either `null` or a 64-hex-char string, and + `entry_sha256` a 64-hex-char string. (Mirrors `chain.js#entryShapeOk` and + `register-gateway-sessions.js#entryShapeOk`, which must stay byte-for-byte + identical checks — see §5.) +- **Valid integer seqs.** Every `seq` is a positive integer (`Number.isInteger`, + `>= 1`). A non-integer, negative, zero, or non-numeric `seq` fails the record's + own shape check above and is never reached by the contiguity check below. +- **Correct genesis seq and prev.** `chain[0].seq === 1` and + `chain[0].prev_entry_sha256 === null`. Both conditions are required + independently: a first entry with `seq > 1` and `prev_entry_sha256 === null` + passes every other check (it hash-recomputes correctly, and there is no prior + entry for a link check to compare against) while still being exactly the + "missing genesis prefix" failure mode (e.g. a `chain.jsonl` truncated from the + front) this rule exists to catch. +- **Contiguity.** For every `i > 0`, `chain[i].seq === chain[i-1].seq + 1`. A gap + (e.g. seq 5 then seq 7) is reported as a **sequence gap**, distinct from a broken + link, so an operator can tell "something was deleted" from "something was + tampered with." +- **Uniqueness.** Contiguity from a fixed genesis already implies no two entries + share a `seq` in a well-formed chain; an implementation that checks contiguity + incrementally (as `register-gateway-sessions.js` does today) gets uniqueness for + free and does not need a separate pass. An implementation must not weaken this by + checking hash links alone. +- **Correct link hashes.** For every `i > 0`, `chain[i].prev_entry_sha256 === + chain[i-1].entry_sha256`. A mismatch is a **broken/mutated link**, distinct from a + sequence gap. +- **Correct entry hashes.** For every entry, recomputing `entry_sha256` from its + own other fields (`chain.js#computeEntrySha256`) reproduces the stored + `entry_sha256`. A mismatch is a **TAMPERED entry**, distinct from a sequence gap, + even when it happens to also break the next entry's link. +- **HEAD correctness.** `head` is present, shape-valid, and `head.seq === + tail.seq && head.bundle_id === tail.bundle_id && head.entry_sha256 === + tail.entry_sha256`, where `tail` is `chain[chain.length - 1]`. Divergence here + is handled by the explicit cases in §2, not folded into a single boolean. + +A validator that reports **valid** must have executed every one of the above +checks over the **entire** chain, not merely re-verified hashes locally around a +single edited record. This is what makes `chain[HEAD.seq - 1]` safe to index by any +caller afterward: a chain that has passed this whole validator cannot contain gaps +or duplicate seqs, so `seq - 1` really is that entry's array position. + +## 2. Explicit behavior for each named state + +Every call site listed above must implement — or delegate to a single shared +function that implements — exactly this behavior for each state. "Refuse" means: +throw/report failure and take no chain-mutating action; do not guess a repair. + +| # | State | Behavior | +|---|---|---| +| 1 | **Both `chain.jsonl` and `HEAD.json` absent.** | Not a failure. This is the genesis-of-genesis state (no session has ever been appended). Reconciliation and the append-time check both short-circuit and perform **zero disk writes**. The status walk reports `not-applicable` ("empty gateway-session log — nothing to verify"), not `verified` and not `failed`. | +| 2 | **Empty chain with `HEAD.json` present.** | Refuse. `HEAD.json` names a tail but `chain.jsonl` has no entries to be that tail — this is incomplete/corrupt state (e.g. the chain file was lost or replaced), never treated as "empty chain, fine." | +| 3 | **`HEAD.json` malformed (unparseable JSON, or parses but fails `headShapeOk`).** | Refuse, **unless** the chain independently and fully validates (per §1) and the *lagging-HEAD* condition in row 7 below applies — a syntactically broken `HEAD.json` with a fully verified chain is one of the ways the lagging-HEAD case can present after a crash mid-write, and reconciliation must attempt that repair before giving up. If the chain itself does not fully validate, or `HEAD.json`'s content (once parsed) does not point at a genuine ancestor of the tail, refuse. | +| 4 | **`HEAD.json` missing entirely (distinct from malformed).** | Same disposition as row 3: reconciliation treats it as the lagging-HEAD case and attempts to rebuild `HEAD` from the verified tail; the append-time check and status walk refuse (they do not repair; only reconciliation, under the exclusive lease, is permitted to write `HEAD`). | +| 5 | **Equal `seq` with differing hash** (two entries, or `HEAD` and the entry it claims to be, share a `seq` but not an `entry_sha256`). | Refuse as a **fork**, always — this is never the lagging-HEAD case (row 7) and must never be auto-repaired. A fork means two candidate histories exist for the same position; picking one silently is exactly the split-brain outcome this contract exists to prevent. | +| 6 | **Invalid genesis** (`chain[0].seq !== 1`, or `chain[0].prev_entry_sha256 !== null`, or an otherwise malformed first record). | Refuse. Per §1, this is checked as a compound condition, not "hash recomputes so it's fine." | +| 7 | **Mid-run lagging HEAD: chain tail exactly one step ahead of a HEAD that is a genuine, hash-verified ancestor of that tail** (`tail.seq === head.seq + 1 && tail.prev_entry_sha256 === head.entry_sha256`, with the rest of the chain fully valid per §1). | **This is not corruption.** It is the signature of a crash between `appendSession`'s chain-append step and its HEAD-update step (or, at startup, the equivalent state found during reconciliation). Per Paul's 2026-09-15 decision: **auto-catch-up** — advance `HEAD` in place to the verified tail, emit a loud anomaly recording that the catch-up happened, and continue. This applies identically whether the state is found by the append-time check mid-run or by `reconcileHead` at startup. Anything worse than this exact single-step, hash-matching lag (multi-step lag, HEAD *ahead* of the tail, or a lag where the claimed ancestor does not actually hash-link to the tail) is **not** this case — it falls through to refuse-and-latch. | +| 8 | **Malformed interior record** (any entry other than the physical last line fails its shape check, or fails to `JSON.parse` at all). | Refuse, always, as chain corruption. Only the **final** record gets the torn-record accommodation in row 9 — an interior record can only be malformed through data loss or tampering, never through an in-flight crash (a crash can only ever be caught mid-write on the line being written *last*; every earlier line already completed its own open/write/fsync/close cycle before the process could move on to appending the next one). | +| 9 | **A torn/unparseable FINAL record.** | **Not corruption; the on-disk-tear counterpart of row 7, and must not be treated as unrecoverable.** A crash during `chain.jsonl`'s own append (as opposed to the later HEAD-update step covered by row 7) can leave a truncated, non-JSON-parseable last line. The last **complete** record — i.e. the one before the torn tail — is authoritative. Under the exclusive writer-claim lease, and only under that lease, isolate or truncate the torn final record and proceed as if the chain's tail were the last complete record. This determination must use a real, enforced `MAX_ENTRY_BYTES` bound when scanning backward/growing a read window to find the last complete line — not an assumption that entries are "typically" some size. A file with no newline boundary found within the capped window must fail closed (refuse) rather than fall back to reading the entire file into memory. This accommodation applies during startup reconciliation exactly as it would mid-run; there is no separate "torn tail is fine at startup but not mid-run" rule. | + +## 3. Two failure classes stay distinct everywhere + +Every message and every anomaly/evidence string produced by any of the three call +sites must make it possible for an operator to tell these two classes apart, per +row 5 vs. row 7/9 above: + +- **Structural/integrity failure** (fork, tamper, sequence gap, invalid genesis, + malformed interior record, HEAD ahead of a verified tail, or any lag that does + not satisfy the exact single-step/hash-matching test in row 7): **refuse and + latch.** This is genuine corruption or an adversarial condition; no call site + repairs it silently, and it does not self-heal without an operator action + (restore from backup, truncate at the last known-good entry, or manually rebuild + `HEAD` — see the recovery runbook this contract's commits must ship alongside it, + per Paul's startup-posture decision below). +- **Lagging-pointer / torn-tail catch-up** (rows 1, 3, 4, 7, 9 as scoped above): + **repair and continue, loudly.** This is the expected shape of "the process died + between two of its own writes," which the write ordering in `chain.js`'s own + header comment exists specifically to make detectable-and-recoverable rather than + silently accepted. + +Conflating these two — treating every divergence as unrecoverable corruption, or +treating every divergence as silently fine — was independently flagged by every +reviewer of the pre-panel draft as the central defect to avoid. + +## 4. Startup posture on a genuine structural-failure verdict + +Per Paul's binding 2026-09-15 decision: when reconciliation (or the walk it runs +before the heartbeat starts) reaches a **structural/integrity failure** verdict +(§3, first bullet) at startup, the gateway must: + +1. **Hard-fail startup.** Stop accepting new sessions / do not start the service. + Nothing runs on top of confirmed corruption. +2. **Surface it loudly, not silently.** Emit a structured top-level error, exit + with a clear non-zero exit code, and write a diagnostic artifact to disk (a + status file does not require a running gateway). +3. **Point directly at the recovery runbook** (restore from backup / + truncate-at-last-known-good / rebuild `HEAD` from the verified tail) so the + diagnostic always ends in a concrete next step — never a stopped process with no + documented way out. +4. **Release the writer-claim lease** on every initialization failure path, so a + subsequent operator-run recovery command is not itself blocked by a lease this + failed startup is still holding. + +This posture applies only to a genuine structural-failure verdict. The +lagging-pointer/torn-tail states in §2 are, by definition, resolved by +reconciliation itself before this posture would ever trigger — they are not +"structural corruption" and must never route through this hard-fail path. + +## 5. One implementation, not three copies + +`chain.reconcileHead`, the append-time check inside `chain.appendSession` / +`chain.repairMissingChainEntry`, and `checks/register-gateway-sessions.js`'s +`walkGatewaySessions` must all call into the **same** structural-validation logic +for the checks in §1 (shared code, or, at minimum, provably identical hand-rolled +copies with a test that fails if they diverge — this codebase's own established +convention, per `chain.js#entryShapeOk` and +`register-gateway-sessions.js#entryShapeOk` already being intentionally-duplicated +but must-stay-identical). A commit that changes what "valid" means at one call site +without changing it identically at the other two violates this contract. + +The three call sites differ only in **what they do with the verdict**: +reconciliation may repair (rows 1/3/4/7/9) and writes `HEAD`; the append-time check +never writes `HEAD` outside of `appendSession`'s own normal flow and, on rows 7/9, +performs the same in-place advance under the same lease; the status walk only +reports evidence and never writes anything. + +## 6. Admission latch (commit 6) + +`chain.getChainIntegrityFailure(): {status, class, reason, at} | null` is the single, +process-local surface every genuine structural failure this contract's detectors +(reconciliation and the append-time check, §1/§5) latch into. Deliberately a +detail-object accessor, never a bare `(): boolean` -- a boolean cannot carry the +diagnostic reason a human needs to act on. + +**Primary enforcement point: session admission.** `GatewayProxy#openConnection` +consults `getChainIntegrityFailure()` first, before its existing +writer-claim-lost/draining check, and refuses every new session the instant it is +non-null. This is **belt-and-braces on top of the append-time check** (§1/§5), which +remains the primary correctness guarantee -- even a session that somehow got admitted +would still have its own eventual `chain.appendSession` refused. Refusing at +admission is what makes a **runtime** transition into failure behave identically to +detecting it at startup: nothing separate has to "notice" the latch and call a +stop-accepting method: the very next `openConnection` call already sees it. + +**`recoverCrashedSessions` aborts immediately once latched.** If an earlier +connection's own repair path (`verifyAndRepairBundleCollision` -> +`checkHeadAgainstTailOrRepair`) latches a genuine failure partway through startup +recovery's per-connection loop, every connection not yet processed is guaranteed to +hit the identical failure the moment its own `chain.appendSession` runs -- the +corruption is a property of the shared chain, not of any one connection's WAL. +Continuing to parse and replay their WALs (with attendant fsyncs and lease renewals) +only to rediscover the same already-known failure is pure waste; this function checks +the latch once per loop iteration and stops instead, leaving their WAL/intent state +untouched for the next restart (or an explicit `recovery-abandon`). + +**Classification, not a one-off exception list.** A failure only latches when it is +genuinely structural/content-level per §1/§3's own taxonomy (fork, tamper, sequence +gap, invalid genesis, malformed interior record, a lag worse than the single-step +case, or a tail `readChainTail` could open and read but that does not parse/shape- +check). An **operational** failure never latches: + +- `readChainTail` failing to even **open** `chain.jsonl` with `EACCES`/`EPERM` (a + misapplied or in-progress commit-2 group-readable chmod pass, or an operator's own + filesystem-permission change) is refused for that one append + (`GATEWAY_CHAIN_TAIL_UNREADABLE_OPERATIONAL`) but does **not** engage the latch -- + a permission problem says nothing about the chain's own integrity. +- `checks/register-gateway-sessions.js`'s own periodic status walk (run roughly every + `STATUS_WRITE_INTERVAL_MS` by `gateway.js#writeStatusFile`) reports + `session_chain_integrity` as **evidence only** and never writes to the latch at all, + per §5's "one authoritative structural validator" contract -- its own missing- + bundle-file finding (an operator archiving/moving a sealed bundle) is exactly the + false positive this rule exists to keep out of the latch. + +**Known limitation, stated here and in the written `gateway-status.json` itself** +(`session_chain_integrity_cadence_note`): because the status walk never latches, its +~10s cadence is not what stops admission on any finding. A HEAD/tail **pointer** +divergence is instead caught immediately by the append-time check. Mid-chain +**interior** corruption -- an entry whose own hash no longer recomputes, a sequence +gap, or a broken link, where HEAD and the chain's own tail already agree -- is +structurally invisible to that O(1), tail-only check; the status walk's own evidence +field may show it, but nothing currently *acts* on that evidence mid-run. Detecting +**and stopping admission for** that class depends on `reconcileHead`'s full walk +running again at the next startup (which hard-fails per §4), or on an explicit, +separately-scheduled deep walk that does not exist in this build. + +## Summary (for code comments) + +> Per `docs/contracts/chain-validity.md` (C2): one structural validator (seqs, +> genesis, contiguity, uniqueness, schema, link+entry hashes) used identically by +> reconciliation, the append-time check, and the status walk. A single-step, +> hash-matching HEAD lag (or its torn-final-record counterpart) is not corruption — +> auto-catch-up under the lease with a loud anomaly. Everything else refuses and +> latches. Structural failure at startup hard-fails loudly with a runbook pointer, +> per Paul's 2026-09-15 decision. The latch (`chain.getChainIntegrityFailure()`) is +> enforced primarily at session ADMISSION (`GatewayProxy#openConnection`), belt-and- +> braces on top of the append-time check; classified, not a one-off list -- an +> EACCES/EPERM opening chain.jsonl, or the status walk's own missing-bundle finding, +> is OPERATIONAL and never latches. diff --git a/docs/contracts/wal-append-failure-semantics.md b/docs/contracts/wal-append-failure-semantics.md new file mode 100644 index 00000000..b7e470d5 --- /dev/null +++ b/docs/contracts/wal-append-failure-semantics.md @@ -0,0 +1,147 @@ +# C1 — WAL Append-Failure Semantics + +> **Status:** binding design contract for PR #33's crash-recovery/idempotency follow-up +> work. Referenced by commit 1 (lease keepalive), commit 5 (sampling `CALL_RESULT` +> fail-closed), and any future code that calls `recovery.appendWalEvent` or reads a +> connection WAL. Locked by the 2026-09-15 fix-round panel (see +> `claude/graphsmith-fix-round-plan-round1-2026-09-15.md` in the GraphSmith project); +> do not reopen items 1-5 below without a new decision record. + +This contract governs `scripts/gateway/recovery.js#appendWalEvent` (and its private +helpers `appendDurableLine`/`writeFullySync`) and every caller of it. It exists +because the original PR #33 follow-up plan modeled a WAL append failure as "nothing +was written" and built retry/recovery logic on that assumption; three independent +reviewers (Fable, GPT-6 Astra, Grok-4.6) found the assumption false and cross-agreed +on why. This document is the corrected model. + +## 1. A throw from `appendWalEvent` means UNKNOWN persistence, not absence + +`appendWalEvent` is `ensureDir` → `openSync(path, "a", 0o600)` → `writeFullySync` +(loops `writeSync` until every byte is written) → `fsyncSync` → `closeSync` → +`fsyncDir`. A caller that catches a throw from this call **must not** assume the +event was not durably recorded. Exactly three outcomes are possible, and they are +indistinguishable from the caller's stack frame alone: + +1. **Nothing was written.** The `openSync` or the first `writeSync` failed before + any bytes landed (e.g. `ENOSPC`, `EROFS` at open time). +2. **A torn/partial line was written.** `writeFullySync`'s own truncate-on-stall + path (see its header comment in `recovery.js`) tries to undo a partial write by + `ftruncateSync`-ing back to the pre-append size, but that truncate is itself a + filesystem call and can fail (the exact failure mode this contract exists to + cover). A torn line can therefore remain on disk after the throw. +3. **The line was written completely, and a later step failed.** `fsyncSync` (data) + or the caller's own subsequent `fsyncDir` can throw *after* `writeFullySync` has + already returned successfully and the complete line is sitting in the file + (possibly not yet durable across a real crash, but present and syntactically + valid to any reader that opens the file right now). + +Because outcome 3 is possible, **a "failed append" and "the event was not recorded" +are not the same fact.** Any code that decides what to tell a downstream caller, or +what to write into an in-memory session record, must say "not confirmed durable," +never "not recorded" or "discarded." + +## 2. On any append failure, the connection's WAL is poisoned in memory + +The moment `appendWalEvent` throws for a given `connectionId`, that connection's +in-memory session state must be marked **poisoned** (a boolean or reason string on +the session record is sufficient) and **every subsequent call to +`appendWalEvent` for that same connection must be refused before it touches the +filesystem** — the caller should treat the refusal exactly like a second append +failure (see §3), without re-attempting the write. + +Rationale: per §1 outcome 2, the file may already end in an unterminated, torn JSON +line. `readWalEvents`' replay contract (`recovery.js`, see its header comment) is +"stop at the first line that fails to `JSON.parse`, keep everything before it" — +which exists precisely because a crash can only tear the *last* line in flight. +That contract silently assumes nothing is ever appended to the file again after a +tear. If a second, unrelated append is allowed to proceed on a torn WAL, its bytes +concatenate directly onto the torn line, and the result is a single malformed line +that swallows both the original partial content and the new event — the new event +is now unparseable too, and is lost to replay along with everything the caller +believed it just durably recorded. Poisoning the connection converts "silently lose +a real event" into "loudly refuse further writes on this connection," which is the +correct fail-closed direction. + +The poison flag is in-memory only and connection-scoped. It is not written to disk, +does not affect any other connection's WAL, and clears when the connection is +closed/reopened (a new connection gets a fresh WAL file and a fresh poison state). + +## 3. No best-effort second append after a possible tear + +Do not attempt a second, best-effort `appendWalEvent` call to record an error +result (or anything else) after the first append for that event already failed — +neither immediately, nor later on the same poisoned connection. This reverses an +earlier draft of the plan, which proposed a best-effort second append "for the +error CALL_RESULT" so that in-memory state and on-disk replay state would agree. + +Two independent, cross-confirmed failure modes make that unsafe: + +- **Concatenation onto a torn line** (§2): the second append can land directly + after an unterminated line from the first attempt, corrupting both. +- **Two conflicting terminal records for the same correlation key.** If the first + append actually completed (§1 outcome 3) and only a later fsync step threw, a + second append for the *same logical event* — even an "error" record instead of + the original "success" record — creates two `CALL_RESULT` lines on disk for one + correlation key: one recording success, one recording failure. Replay then has + two candidate terminal records for the same call, and which one "wins" was left + unspecified by the earlier draft. §4 defines that resolution rule now, but the + conflict itself is avoidable simply by never attempting the second write. + +The correct sequence when an append fails is: mark the connection poisoned (§2), +record the caller-facing outcome (e.g. a JSON-RPC error to the downstream caller) +and the in-memory session state, and stop. Do not touch the WAL again for that +connection. + +## 4. Replay conflict rule for terminal records + +`readWalEvents` can, despite §2-3, still encounter two `CALL_RESULT` events for the +same correlation key on legitimate historical WALs (e.g. a build that predates this +contract, or any other path that ever produced a duplicate before this rule +existed). Replay must resolve that deterministically: + +> **First complete `CALL_RESULT` per correlation key wins.** Every later +> `CALL_RESULT` for the same correlation key is treated as a duplicate: it is not +> applied to the replayed session state, and it is recorded as an anomaly (not +> silently dropped and not silently preferred over the first one). + +"Complete" here means the record parsed successfully as JSON and passed its own +shape check — i.e. it is one of the events `readWalEvents` returns at all (a torn +line never reaches this rule because it stops the read before ever producing an +event for that line, per its own stop-at-first-bad-line contract). + +This rule must be implemented once, in whatever function replays a connection's WAL +into session state, and covered by a test that constructs exactly this state (two +complete `CALL_RESULT` events for one correlation key) and asserts: the replayed +session reflects the first one, the second is present in the anomaly list, and +replay does not throw. + +## 5. `discarded_result_sha256`: exact attempted bytes, or omit the field + +When code records that a result could not be durably written (e.g. the sampling +`CALL_RESULT` fail-closed path), it may attach a `discarded_result_sha256` field so +a later auditor can correlate "the agent produced X" with "the gateway withheld X." +That field is only trustworthy under one rule: + +> **Hash the exact serialized buffer that was attempted for the WAL append** (the +> same `Buffer`/string passed to `writeFullySync`/`appendDurableLine`), **or omit +> the field entirely.** Never hash a fresh `JSON.stringify(resultObject)` call made +> after the fact for the sole purpose of populating this field. + +Rationale: `JSON.stringify` output depends on object key insertion order, which is +not guaranteed to be stable across two separate serializations of "the same" +logical object (a field added, reordered, or populated by a different code path on +a retry can silently change the byte string while the object's meaning is +unchanged). A digest computed from a *second*, independent serialization is +therefore not a reliable commitment to what was actually attempted on disk — it can +disagree with the real attempted bytes for reasons that have nothing to do with the +result's content. If the exact attempted buffer is not available in scope where the +digest is computed, the field must be left off the record rather than populated +with a digest of something else. A missing field is honest; a digest of the wrong +bytes is not. + +## Summary (for code comments) + +> Per `docs/contracts/wal-append-failure-semantics.md` (C1): a WAL append throw +> means unknown persistence, not absence. Poison the connection on failure; never +> retry. Replay keeps the first complete terminal record per correlation key. +> `discarded_result_sha256` hashes the exact attempted buffer or is omitted. diff --git a/docs/runbooks/gateway-chain-corruption-recovery.md b/docs/runbooks/gateway-chain-corruption-recovery.md new file mode 100644 index 00000000..26aee053 --- /dev/null +++ b/docs/runbooks/gateway-chain-corruption-recovery.md @@ -0,0 +1,145 @@ +# Runbook: gateway-session chain corruption recovery + +> Referenced from every `GATEWAY_CHAIN_STRUCTURAL_FAILURE` diagnostic this gateway +> writes (`/gateway-startup-failure.json`) and from every +> `GATEWAY_CHAIN_HEAD_DIVERGED` refusal it logs mid-run. Ships alongside round-1 +> fix-plan commit 4 (`chain.reconcileHead` + the classified append-time HEAD/tail +> check) per Paul's 2026-09-15 startup-posture decision +> (`claude/graphsmith-fix-round-plan-round1-2026-09-15.md`, +> `docs/contracts/chain-validity.md` SS4): a hard-failed startup must always end in a +> concrete next step, never just a stopped process. Read this top to bottom before +> touching `chain.jsonl` or `HEAD.json` by hand -- every option below is destructive in +> a different way, and picking the wrong one can destroy attested history that was +> otherwise recoverable. + +## 1. Is this actually corruption, or just a lagging pointer? + +Not every divergence between `HEAD.json` and `chain.jsonl` needs this runbook. Two +classes of divergence are **not** corruption and are already handled automatically, +under the writer-claim, by this gateway itself: + +- **A single-step lagging `HEAD`** (the chain's own tail is exactly one entry ahead of + `HEAD`, and the link hash matches) is auto-repaired the moment the next session tries + to append, or at the next startup by `chain.reconcileHead`. This is the ordinary + signature of a crash between `chain.jsonl`'s own append and `HEAD.json`'s update. You + do not need this runbook for it -- just restart the gateway (or let the next session + close normally) and check `gateway.js status` afterward. +- **Any size of lagging `HEAD`, found at startup**, where the chain independently + verifies end-to-end and `HEAD` names a genuine, hash-verified entry somewhere earlier + in that same chain, is repaired by `chain.reconcileHead` before the gateway ever + starts accepting sessions. + +If the gateway is refusing to start with a `GATEWAY_CHAIN_STRUCTURAL_FAILURE` +diagnostic (check `/gateway-startup-failure.json`), or a running gateway is +refusing every session close with `GATEWAY_CHAIN_HEAD_DIVERGED` / +`GATEWAY_CHAIN_INTEGRITY_FAILED`, you are past the automatic cases. Read the +diagnostic's `message`/`reason` field first -- it names exactly which of the cases below +you are in (a fork, a broken hash link, a sequence gap, an invalid genesis, HEAD ahead +of the chain's own tail, or an unreadable/torn tail) before you pick a recovery option. + +## 2. Before doing anything else + +1. **Stop the gateway process** if it is somehow still running (it should already have + refused to start, or be about to exit after a runtime refusal -- confirm it is not + silently retrying). +2. **Back up the entire `/gateway-sessions/` directory** (`chain.jsonl`, + `HEAD.json`, every `.json`, and the `quarantine/` subdirectory if present) + to a separate location before touching anything. Every option below assumes you can + get back to exactly this state if you make a mistake. +3. **Do not delete or hand-edit `chain.jsonl`/`HEAD.json` directly** until you have + picked one of the three options below and understand its consequences. A single + `sed`/manual JSON edit that produces a shape-valid-looking record is exactly how a + silent, undetected fork gets introduced. + +## 3. Option A -- restore from backup + +**Use this when** you have an independent, trusted backup of `/gateway- +sessions/` taken before the corruption occurred (a nightly snapshot, a filesystem +snapshot, etc.), and the corrupted window since that backup is small enough that losing +the sessions recorded in it is acceptable. + +1. Confirm the backup itself verifies: run the chain-integrity check + (`checks/register-gateway-sessions.js`, or `gateway.js status` once the gateway can + start against a copy of the backup in a scratch `state_dir`) against the backup + **before** restoring it over the live directory. +2. Stop the gateway (if not already stopped), replace the live `gateway-sessions/` + directory with the verified backup, and restart. +3. Any bundles/WALs newer than the backup are now orphaned. Move them to a separate + `manual-review/` directory outside `state_dir` for a human to inspect -- do not + delete them; they may still be useful evidence even though they can no longer be + chain-appended automatically. + +**Cost:** every session sealed after the backup was taken is gone from the attested +chain, even if it was otherwise legitimate. + +## 4. Option B -- truncate at the last known-good entry + +**Use this when** the failure is localized to the chain's own **tail** (the last one or +two records) -- e.g. a torn/unparseable final record from a crash mid-write, or a +tampered/malformed final entry -- and every entry before it verifies cleanly. This is +the right choice when you do **not** have a recent backup but most of the chain's +history is still trustworthy. + +1. Run the structural validator against `chain.jsonl` up to (but not including) the + suspect final record(s) to confirm everything before that point is genuinely intact + (hashes, links, sequence, genesis). Do not assume -- verify. +2. Truncate `chain.jsonl` to end exactly after the last entry that verified. Use a tool + that truncates at a byte offset you have confirmed corresponds to a complete, + newline-terminated record -- never a text editor's own line-ending guess. +3. Rebuild `HEAD.json` from that new, truncated tail (see the exact record shape in + Option C step 2 below). +4. Move the bundle file(s) for any entries you truncated away into `manual-review/` + (outside `state_dir`) rather than deleting them. +5. Restart the gateway. `chain.reconcileHead` will confirm the truncated chain verifies + and that `HEAD.json` matches its new tail before accepting any session. + +**Cost:** every entry from the first bad record onward is removed from the attested +chain, whether or not each individual one was itself legitimate (a sequence gap is not +repairable in place -- an entry that comes after a bad one can never be re-attached +without also being re-verified from scratch). + +## 5. Option C -- rebuild `HEAD` from the validated chain tail + +**Use this when** `chain.jsonl` itself is fully intact and verifies end-to-end, and the +**only** thing wrong is `HEAD.json` -- but the divergence is worse than the single-step +or verified-ancestor cases this gateway already repairs automatically (for example, +`HEAD.json` was manually edited to a bogus value, or points at a `seq` beyond the +chain's own real tail, or was lost/replaced by something that does not correspond to any +real entry in the chain at all). + +1. Run the structural validator against the **entire** `chain.jsonl` with `head` + omitted/ignored to confirm it independently verifies (correct genesis, contiguous + `seq`, every link hash and entry hash correct) end-to-end. If it does **not** + verify, stop -- you are actually in Option B or Option A's territory, not this one. +2. Take the chain's own last entry (the highest `seq`, confirmed correct by step 1) and + write `HEAD.json` as exactly: + ```json + { + "schema_version": "1.0", + "seq": , + "bundle_id": "", + "entry_sha256": "" + } + ``` + Do not hand-construct any field other than by copying it verbatim from the verified + tail entry. +3. Restart the gateway. `chain.reconcileHead` re-verifies the same thing you just did by + hand before accepting any session, so a mistake here fails startup again rather than + silently taking effect. + +**Cost:** none to the chain's own history -- this option only ever repairs the pointer, +never the chain -- but it requires you to have correctly confirmed step 1's "fully +intact, end-to-end" precondition; skipping that verification and rebuilding `HEAD` over +a chain that does **not** actually verify silently converts a detectable corruption into +an falsely-"healthy"-looking gateway. + +## 6. After recovery + +- Confirm `gateway.js status` reports `session_chain_integrity: { status: "verified" }` + before considering the incident closed. +- Investigate *why* the corruption happened (disk full, a manual edit, a filesystem + fault, a bug) before it recurs -- none of the three options above address root cause. +- If you used Option A or Option B, the sessions you removed from the attested chain are + gone from `SG-FR-7`'s guarantee even though their raw bundle/WAL files may still exist + in `manual-review/`; do not re-append them later by hand -- that would itself be + exactly the kind of undetectable, unattested mutation this chain exists to prevent. diff --git a/scripts/gateway/agent-transport.js b/scripts/gateway/agent-transport.js index a75ea91f..42a1de74 100644 --- a/scripts/gateway/agent-transport.js +++ b/scripts/gateway/agent-transport.js @@ -155,7 +155,53 @@ function runStdioAgentTransport(ctx) { reject(new Error("agent stdio disconnected before responding to a pushed request")); } pendingPushed.clear(); - await ctx.proxy.closeConnection(connectionId, "agent stdio disconnected"); + /* Pre-existing gap surfaced by merging PR #29's gateway.js doStop() fix ("await + * close() so a downstream ... is actually confirmed gone" -- Codex PR #29 review + * round 8): this listener runs as an async callback on the readline "close" EVENT, + * which nothing awaits or catches -- a rejection here (e.g. closeConnection's own + * intent-fencing loop hitting an unreadable/corrupt recovery intent, see + * proxy.js#closeConnection) becomes an unhandled promise rejection that crashes the + * WHOLE gateway process. This is not specific to shutdown: any ordinary stdio + * disconnect that races an unreadable intent for this connection hits the same + * path. gateway.js's own doStop() loop already guards its own (redundant, in the + * shutdown case) call to closeConnection for exactly this failure mode -- mirror + * that guard here so a single connection's fs-level intent corruption can never take + * the process down. Note this failure throws before closeConnection reaches its own + * onSealFailure/recordAnomaly reporting (both live further down, past the + * intent-fencing loop that threw), so outside of a shutdown that also runs + * gateway.js's own logged loop for this same connectionId, this catch previously had + * no logging surface of its own -- silently swallowing was a deliberate "never let + * one connection's fs corruption take the whole process down" choice, not a claim + * that the failure is reported elsewhere. + * + * Round-N fix (frontier-panel finding #1's own follow-on): proxy.js#closeConnection + * now guarantees this.sessions/this.agentInitialized are always cleaned up (a + * try/finally around the same intent-fencing loop this comment describes), which + * removes the ghost session that previously let gateway.js's own shutdown loop + * redundantly retry (and loudly log) THIS SAME failure a second time during a + * shutdown -- so this catch is now, in the ordinary case, the ONLY place this + * failure is ever surfaced at all, not just the no-shutdown case the paragraph + * above originally scoped this to. ctx.log (when provided -- see gateway.js's own + * call site) closes exactly the gap that paragraph named as "a reasonable follow-up + * but out of scope"; still best effort, so a throwing logger cannot turn an + * already-swallowed close failure into a crash. */ + try { + await ctx.proxy.closeConnection(connectionId, "agent stdio disconnected"); + } catch (error) { + // best effort only -- matches this file's own process.stdout "error" listener + // above, which absorbs a comparable stream-level failure for the same reason. + if (typeof ctx.log === "function") { + try { + ctx.log( + `CONNECTION CLOSE FAILURE for connection "${connectionId}" (agent stdio disconnected): ` + + `${error.message} (${error.code || "no code"}) -- session state was still cleaned up; ` + + "this connection will not be retried during shutdown." + ); + } catch (logError) { + // never let a failing logger turn an already-swallowed close failure into a crash. + } + } + } resolve(); }); }); @@ -230,12 +276,30 @@ function runHttpAgentTransport(ctx, listenConfig, token) { const httpSessions = new Map(); // sessionId -> { idleTimer } -- transport-level bookkeeping proxy.js has no reason to know about. + /* Round-N fix (frontier-panel finding #1's own follow-on): same rationale as the stdio + * transport's own closeConnection catch above -- proxy.js#closeConnection's + * try/finally now guarantees session cleanup even when the intent-fencing loop throws, + * but this call site previously swallowed that failure with no logging surface at all. + * ctx.log (best effort; never lets a throwing logger turn this into a crash) closes the + * same gap here. */ function sealSession(sessionId, reason) { const entry = httpSessions.get(sessionId); if (!entry) return; clearTimeout(entry.idleTimer); httpSessions.delete(sessionId); - ctx.proxy.closeConnection(sessionId, reason).catch(() => {}); + ctx.proxy.closeConnection(sessionId, reason).catch((error) => { + if (typeof ctx.log === "function") { + try { + ctx.log( + `CONNECTION CLOSE FAILURE for connection "${sessionId}" (${reason}): ` + + `${error.message} (${error.code || "no code"}) -- session state was still cleaned up; ` + + "this connection will not be retried during shutdown." + ); + } catch (logError) { + // never let a failing logger turn an already-swallowed close failure into a crash. + } + } + }); } function touchSession(sessionId) { diff --git a/scripts/gateway/chain.js b/scripts/gateway/chain.js index d60a4fa0..dffc6757 100644 Binary files a/scripts/gateway/chain.js and b/scripts/gateway/chain.js differ diff --git a/scripts/gateway/downstream.js b/scripts/gateway/downstream.js index 0528b080..7adcdcff 100644 --- a/scripts/gateway/downstream.js +++ b/scripts/gateway/downstream.js @@ -84,6 +84,18 @@ const META_CLIENT_INFO_KEY = "io.modelcontextprotocol/clientInfo"; const META_CLIENT_CAPABILITIES_KEY = "io.modelcontextprotocol/clientCapabilities"; const STATELESS_META_METHODS = new Set(["tools/list", "tools/call", "server/discover"]); +/* Option C (Codex PR #29 Finding 2, external-panel-reviewed design -- see + * option-c-hardened-design.md, "Both transports" paragraph): the gateway's own durable + * per-operation identity (scripts/gateway/recovery.js#computeIntentKey), propagated to + * the downstream as a conventional idempotency key so a downstream that itself + * recognizes one gets genuine at-most-once execution too. A downstream that ignores this + * key is unaffected -- the gateway-side dispatch fence in proxy.js is what actually + * protects against a double dispatch either way; this is defense in depth, not the + * mechanism the guarantee depends on. Scoped to exactly the calls proxy.js itself fences + * (real `tools/call` dispatch only, per that file's own "deliberately scoped" comment) -- + * call() below only ever receives this for those. */ +const META_IDEMPOTENCY_KEY = "graphsmith/idempotency_key"; + /* Mirrors mcp-server/src/protocol.js's own STATELESS_PROTOCOL_VERSION -- a DIFFERENT * literal from MCP_PROTOCOL_VERSION above. That constant is the legacy handshake * version this client negotiates via the top-level `initialize` call (mcp-server's own @@ -377,10 +389,21 @@ function connectStdio(endpoint, options = {}) { closeError = fail(`downstream process failed to start: ${error.message}`, "GATEWAY_DOWNSTREAM_DISCONNECTED"); }); - function call(method, params, timeoutMs, onIdAssigned) { + function call(method, params, timeoutMs, onIdAssigned, idempotencyKey) { if (closed) return Promise.reject(closeError || fail("downstream connection is closed", "GATEWAY_DOWNSTREAM_DISCONNECTED")); const id = nextId++; if (typeof onIdAssigned === "function") onIdAssigned(id); + /* See META_IDEMPOTENCY_KEY's header comment above. Unlike the HTTP leg's stateless + * _meta block, stdio needs no per-request protocolVersion/clientInfo (those are + * negotiated once, at this connection's own initialize) -- this is the only _meta + * merge stdio ever needs, and only when a key is actually supplied. Placed LAST in + * the merge (after any `_meta` the caller's own params happened to carry) so this + * gateway-internal correlation id always wins -- unlike buildStatelessMeta's own + * agent-overridable fields, this one is not a negotiable protocol value the agent is + * meant to be able to supply or override. */ + const effectiveParams = idempotencyKey + ? { ...(params || {}), _meta: { ...((params && params._meta) || {}), [META_IDEMPOTENCY_KEY]: idempotencyKey } } + : params; return new Promise((resolve, reject) => { const timer = setTimeout(() => { pending.delete(id); @@ -388,7 +411,7 @@ function connectStdio(endpoint, options = {}) { }, timeoutMs || DEFAULT_REQUEST_TIMEOUT_MS); if (typeof timer.unref === "function") timer.unref(); pending.set(id, { resolve, reject, timer }); - child.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n"); + child.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params: effectiveParams }) + "\n"); }); } @@ -477,7 +500,7 @@ function connectHttp(endpoint, options = {}) { let reachable = true; const notificationHandlers = []; // never fired: plain request/response HTTP has no server push - function call(method, params, timeoutMs, onIdAssigned) { + function call(method, params, timeoutMs, onIdAssigned, idempotencyKey) { if (closed) return Promise.reject(fail("downstream connection is closed", "GATEWAY_DOWNSTREAM_DISCONNECTED")); const id = nextId++; if (typeof onIdAssigned === "function") onIdAssigned(id); @@ -485,9 +508,22 @@ function connectHttp(endpoint, options = {}) { /* See buildStatelessMeta's header comment: this leg is stateless per-request, so * every request under a method the stateless MCP protocol requires _meta on gets * one attached here, centrally, rather than at each call site -- a caller-supplied - * _meta (if any) wins over the generated one, field by field. */ - const effectiveParams = isStatelessMetaMethod - ? { ...(params || {}), _meta: { ...buildStatelessMeta(options), ...((params && params._meta) || {}) } } + * _meta (if any) wins over the generated one, field by field. The idempotency key + * (see META_IDEMPOTENCY_KEY's header comment) is spread in LAST, after the caller's + * own _meta, so -- unlike the fields above it -- it always wins: it is this gateway's + * own internal correlation id, never a value the agent is meant to supply or + * override, so it is merged even on a call whose method isn't itself one of the + * stateless-protocol methods (defensive; in practice proxy.js only ever supplies one + * for "tools/call", which already is). */ + const effectiveParams = isStatelessMetaMethod || idempotencyKey + ? { + ...(params || {}), + _meta: { + ...(isStatelessMetaMethod ? buildStatelessMeta(options) : {}), + ...((params && params._meta) || {}), + ...(idempotencyKey ? { [META_IDEMPOTENCY_KEY]: idempotencyKey } : {}), + }, + } : params; /* The mirrored header (below) must agree with whatever protocol version this * request's own body actually declares -- STATELESS_META_PROTOCOL_VERSION for a diff --git a/scripts/gateway/gateway.js b/scripts/gateway/gateway.js index 28bcf723..48a27be6 100644 --- a/scripts/gateway/gateway.js +++ b/scripts/gateway/gateway.js @@ -30,6 +30,7 @@ const modeGate = require("./mode-gate.js"); const gatewayConfig = require("./config.js"); const chain = require("./chain.js"); const session = require("./session.js"); +const recovery = require("./recovery.js"); const { GatewayProxy, MAX_PENDING_CALLS_PER_SESSION, MAX_COMPLETED_CALLS_PER_SESSION } = require("./proxy.js"); const downstream = require("./downstream.js"); const { runStdioAgentTransport, runHttpAgentTransport } = require("./agent-transport.js"); @@ -37,6 +38,7 @@ const writerClaimModule = require("../writer-claim.js"); const { WriterClaim } = writerClaimModule; const registerGatewaySessions = require("../../checks/register-gateway-sessions.js"); const stateStore = require("../state-store.js"); +const { tightenStateDirPermissions } = require("./startup-permissions.js"); /** SS3.7's bounded drain: waits (polling) until every open session on `proxy` has no * calls still in flight, or `timeoutMs` elapses, whichever first. Extracted as its own @@ -133,6 +135,23 @@ function withTimeout(promise, timeoutMs, makeError) { * see agent-transport.js's header). Any other case (an http agent transport, or no agent * currently connected) gets a real JSON-RPC error naming exactly why, rather than the * silent drop this was before. */ +/* Per docs/contracts/wal-append-failure-semantics.md (C1 SS1-2): a throw from + * recovery.appendWalEvent means UNKNOWN persistence (possibly nothing written, + * possibly a torn line, possibly a complete line that only failed a later fsync/close), + * never confirmed absence. Any of those three outcomes is grounds to poison the + * connection: marks `session.walPoisoned` on the FIRST such failure only (a later + * failure on an already-poisoned connection is not a new fact worth overwriting the + * original reason with), so every later appendWalEvent attempt on this connection is + * refused before touching the filesystem instead of risking a second write landing on + * top of a possibly-torn line. Shared by every appendWalEvent call site in this + * function (CALL_START and both CALL_RESULT branches) so the contract is enforced + * uniformly rather than re-implemented per call site. */ +function poisonWalOnFailure(s, walError, proxy) { + if (s && !s.walPoisoned) { + s.walPoisoned = { reason: walError.message, at: proxy ? proxy.now() : Date.now() }; + } +} + function forwardDownstreamRequestToAgent(msg, agentPusher, log, proxy, serverName) { if (msg.method !== "sampling/createMessage") { return Promise.resolve({ jsonrpc: "2.0", id: msg.id, error: { code: -32601, message: `This gateway does not forward downstream-initiated method "${msg.method}" to the agent.` } }); @@ -212,7 +231,63 @@ function forwardDownstreamRequestToAgent(msg, agentPusher, log, proxy, serverNam }, }); } + /* Per docs/contracts/wal-append-failure-semantics.md (C1 SS2): once an earlier call on + * this connection poisoned its WAL (see the CALL_START/CALL_RESULT append catches + * below, which set s.walPoisoned on the very first append failure), every later + * appendWalEvent attempt on the SAME connection must be refused "before it touches the + * filesystem" -- not attempted-and-caught again. Checking here, before recordCallStart + * or any WAL write, is what makes that true for a brand-new sampling request: it never + * reaches the try/catch below at all. The poison is in-memory and connection-scoped + * (cleared by a fresh connection getting a fresh session object), so this never blocks + * a different connection or survives a reconnect. */ + if (s && s.walPoisoned) { + return Promise.resolve({ + jsonrpc: "2.0", + id: msg.id, + error: { + code: -32000, + message: + "This connection's WAL was poisoned by an earlier append failure " + + `(${s.walPoisoned.reason}) and is refusing every further durable write on this ` + + "connection rather than risk concatenating onto a possibly-torn line -- per " + + "docs/contracts/wal-append-failure-semantics.md (C1). Not admitting this " + + "sampling/createMessage request; close and reopen the connection to get a fresh WAL.", + }, + }); + } + /* Codex PR #33 review "persist sampling calls in the recovery WAL": this forward is + * recorded into the in-memory session (above) but, before this fix, NEVER into + * recovery.js's WAL at all -- unlike proxy.js's own tools/call path. A crash after the + * agent answers this sampling request but before the connection's own close would + * silently omit that model invocation from a recovered/sealed bundle: recoverCrashedSessions + * replays this connection's WAL from scratch and previously had no event type for it, + * and even hard-coded isModelCall:false on every CALL_START it did replay. This is + * deliberately NOT the same gap as the disclosed one in KNOWN-LIMITATIONS.md item 11 + * ("an agent-initiated sampling/createMessage forwarded through proxy.js's own dispatch + * branch is out of scope for the fence") -- that disclosure is scoped to the opposite + * direction (an AGENT sending sampling/createMessage INTO this gateway) and says so + * explicitly ("rather than a downstream server"); this is a downstream SERVER's own + * request being forwarded UP to the agent, a different code path (this function, not + * proxy.js's handleMessage) with no matching disclosure. Reuses the same CALL_START/ + * CALL_RESULT WAL event shape proxy.js's tools/call path uses (same recoverCrashedSessions/ + * abandonConnection replay code handles both), with isModelCall:true now carried through + * so replay can tell the two apart instead of assuming every replayed call is a plain + * tool call. No idempotency-intent fencing is added here (Finding 2's fence is + * deliberately scoped to tools/call only, unchanged) -- only Finding 1's WAL/attestation + * coverage is extended to this path. + * + * Ordering note (Cluster H, frontier-panel review 2026-09-14, 5/5 unanimous): the + * MAX_COMPLETED_CALLS_PER_SESSION refusal above runs FIRST and returns early, so a + * refused call is never admitted, recorded in session.calls, or forwarded to the agent + * -- it correctly never reaches the WAL either. The admission gate and the durability + * write share one boundary, not two: there is no "evicted call still needs a WAL + * entry" case on this path, since eviction here means outright refusal, not eviction + * of an already-recorded call from memory. The WAL append below remains unconditional + * on whatever this admission check decided; it is never itself capped or tombstoned. */ + let walCallSeq = null; + let walAppendFailed = null; if (s) { + walCallSeq = s.nextCallSeq; session.recordCallStart(s, correlationKey, { tool: "sampling/createMessage", server: recordedServerName, @@ -220,6 +295,52 @@ function forwardDownstreamRequestToAgent(msg, agentPusher, log, proxy, serverNam isModelCall: true, ts: startTs, }); + try { + recovery.appendWalEvent(proxy.stateDir, agentPusher.connectionId, { + type: "CALL_START", + call_seq: walCallSeq, + tool: "sampling/createMessage", + server: recordedServerName, + arguments: msg.params, + isModelCall: true, + ts: proxy.now(), + }); + } catch (walError) { + poisonWalOnFailure(s, walError, proxy); + walAppendFailed = walError; + } + } + if (walAppendFailed) { + /* Codex PR #33 review "refuse sampling when its CALL_START cannot be saved": this + * append happens strictly BEFORE agentPusher.current() is ever invoked -- nothing + * has been sent to the agent yet at this point, so forwarding anyway is not "already + * in flight," it is choosing to forward work this gateway just proved it cannot + * durably attest. + * + * Corrected per docs/contracts/wal-append-failure-semantics.md (C1): this comment + * used to say the CALL_RESULT append further below has "no 'don't forward' option + * left" because the downstream call had already completed. That is no longer true -- + * the CALL_RESULT success-branch append below is now fail-closed too (see its own + * comment): the gateway withholds the real result and returns an error instead when + * that append fails, exactly the same "don't hand back what we can't durably attest" + * choice made here. The two branches differ only in WHEN the choice is available: a + * CALL_START failure can refuse to forward at all (this branch); a CALL_RESULT + * failure can only refuse to RETURN a result the agent already produced -- it cannot + * un-invoke the agent. + * + * A crash before this connection's own close would then replay with no CALL_START + * for this call at all, recreating exactly the attestation gap Finding 1 exists to + * close. Roll back the in-memory pending-call entry (there is nothing durable to + * undo) and return a real JSON-RPC error to the downstream server instead, matching + * proxy.js's own "not dispatched -- safe to retry" contract for the identical + * failure mode. */ + if (s) s.pendingCalls.delete(correlationKey); + log(`Refusing to forward a downstream-initiated sampling/createMessage request to the agent: its CALL_START could not be durably recorded (${walAppendFailed.message}).`); + return Promise.resolve({ + jsonrpc: "2.0", + id: msg.id, + error: { code: -32000, message: `Failed to durably record this sampling request before forwarding it to the agent: ${walAppendFailed.message}. Not forwarded -- safe to retry.` }, + }); } /* Codex PR #29 review round 3 "log completed downstream-initiated sampling steps": * proxy.js's own agent-initiated call path (GatewayProxy#handleMessage) emits a @@ -269,6 +390,83 @@ function forwardDownstreamRequestToAgent(msg, agentPusher, log, proxy, serverNam * entry the gateway itself already removed. */ const correlatedNow = s && !s.finalized && s.pendingCalls.has(correlationKey); if (correlatedNow) { + /* Fail-closed CALL_RESULT WAL append, per docs/contracts/wal-append-failure- + * semantics.md (C1) and the 2026-09-15 fix-round decision record. Reordered to + * attempt the WAL append BEFORE session.recordCallResult -- matching CALL_START's + * own append-before-commit shape above -- rather than the other way around. + * recordCallResult moves this call out of pendingCalls into the permanent s.calls + * array that toSealableSession later seals verbatim; appending first means that if + * the append fails, nothing durable-looking has been committed to the in-memory + * record yet, and the caller below can choose what to commit instead of having to + * undo an already-sealed-shaped success entry. + * + * On failure: mark this connection's WAL poisoned (no best-effort second append -- + * C1 SS3 -- the agent's real result is deliberately NOT re-attempted onto a + * possibly-torn line), attest the failure with the SAME anomaly kind the tools/call + * path already uses for the identical failure mode (GATEWAY_RECOVERY_WAL_APPEND_ + * FAILED at proxy.js:344/795/932 -- reusing it, not minting a new kind, keeps one + * grep-able string for anything that keys on it), and record the call itself as an + * error rather than the real result -- the sealed bundle must not attest a success + * this gateway could not durably record. + * + * `discarded_result_sha256` is deliberately OMITTED here rather than populated: per + * C1 SS5, that field is only trustworthy when it hashes the EXACT serialized buffer + * recovery.appendWalEvent attempted (recovery.js computes that buffer internally, + * stamping its own recorded_at, and does not expose it to this caller) -- hashing a + * fresh, separate JSON.stringify(result) here would be a digest of different bytes + * than what was actually attempted on disk, which C1 says is worse than no digest. + * + * Honesty correction (this failure mode is NOT "safe to retry"): the agent already + * ran real inference and produced `result` by the time this append is attempted -- + * unlike the CALL_START failure above, there is no "nothing happened yet" to fall + * back on. The message below says so plainly: the gateway did not durably record + * the result, the underlying model call may already have executed, and retrying it + * may re-execute it -- never "safe to retry." + * + * Asymmetry with tools/call, disclosed rather than silently copied: proxy.js's own + * CALL_RESULT append (its `correlatedNow` branch) stays fail-OPEN on the identical + * failure mode -- it logs/attests but still returns the real result, relying on the + * intent file's own cached_result as recovery's fallback source of truth. This path + * has no such intent fence (sampling is deliberately unfenced, per Finding 2's own + * scope), so there is no fallback to lean on here and fail-closed is the only honest + * option. Whether proxy.js's fail-open choice is itself still correct now that this + * sibling path no longer matches it is out of scope for this commit and is tracked + * as SAT-1118, not left as only this comment. */ + let walError = null; + if (walCallSeq !== null) { + try { + recovery.appendWalEvent(proxy.stateDir, agentPusher.connectionId, { type: "CALL_RESULT", call_seq: walCallSeq, result, isError: false, ts: proxy.now() }); + } catch (e) { + walError = e; + } + } + if (walError) { + poisonWalOnFailure(s, walError, proxy); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: "sampling/createMessage", detail: walError.message }); + session.recordCallResult(s, correlationKey, { + result: { + error: + `This sampling call's result was not durably recorded (${walError.message}). ` + + "The gateway did not durably record this result; the underlying model call may " + + "already have executed, and retrying it may re-execute it.", + }, + isError: true, + ts: proxy.now(), + }); + logCompletion(true); + log(`Refusing to return a downstream-initiated sampling/createMessage result: its CALL_RESULT could not be durably recorded (${walError.message}).`); + return { + jsonrpc: "2.0", + id: msg.id, + error: { + code: -32000, + message: + `Failed to durably record this sampling call's result: ${walError.message}. ` + + "The gateway did not durably record the result; the underlying model call may " + + "already have executed, and retrying may re-execute it.", + }, + }; + } session.recordCallResult(s, correlationKey, { result, isError: false, ts: proxy.now() }); logCompletion(false); } @@ -298,6 +496,27 @@ function forwardDownstreamRequestToAgent(msg, agentPusher, log, proxy, serverNam * closeConnection/handleDownstreamDisconnect already recorded, not a second one. */ if (correlatedNow) { session.recordCallResult(s, correlationKey, { result: { error: error.message }, isError: true, ts: proxy.now() }); + if (walCallSeq !== null) { + try { + recovery.appendWalEvent(proxy.stateDir, agentPusher.connectionId, { type: "CALL_RESULT", call_seq: walCallSeq, result: { error: error.message }, isError: true, ts: proxy.now() }); + } catch (walError) { + /* Mirror-image of the success branch above, per docs/contracts/ + * wal-append-failure-semantics.md (C1): the downstream already received (or + * is about to receive, via the return below) a real JSON-RPC error for this + * call, so there is nothing left to withhold -- this stays fail-OPEN, unlike + * the success branch. But per C1 SS2 this failure still poisons the + * connection's WAL (a torn line from THIS append is just as real a hazard as + * one from the success branch), and it must be ATTESTED, not just logged: a + * plain log() line here would leave the sealed bundle silent about a WAL + * write this gateway genuinely attempted and failed. Reuses the same + * GATEWAY_RECOVERY_WAL_APPEND_FAILED kind as the success branch and the + * tools/call path (proxy.js:344/795/932) -- one kind for this whole failure + * class, not a new one per call site. */ + poisonWalOnFailure(s, walError, proxy); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: "sampling/createMessage", detail: walError.message }); + log(`Failed to durably record a downstream-initiated sampling call's error result: ${walError.message}`); + } + } logCompletion(true); } else if (!s) { log(`downstream sampling/createMessage forward to agent failed: ${error.message}`); @@ -431,27 +650,109 @@ function buildHealthStatus(ctx) { * real operational path an operator or monitor can act on -- a non-"verified" result * here is a fail-closed, visible signal (tampering, a sequence gap, or an incomplete * append), not silently invisible until someone thinks to run --selftest by hand. */ + /* Round-1 fix-plan item 1, round-2 fix pass (Paul's 2026-09-16 decision): computed + * once, here, rather than inline inside the returned object below (its own previous + * location) so it can ALSO gate the session-chain walk's own maybeRenew() wiring just + * below, without a second disk read. `held_by_this_instance` is false both for a + * caller that never wires a real writer-claim into ctx.writerClaim (e.g. the bare + * `{ status: () => ({}) }` fakes several existing tests already pass) and for the one + * real, expected case where this claim was already released before this ran -- doStop + * calls writeStatusFile ONE LAST TIME after writerClaim.release() (see writeStatusFile's + * own doc comment) specifically so `gateway.js status` still reports an accurate + * "not claimed" snapshot; that call must keep succeeding, not start attempting (and + * failing) a renewal against a claim this instance deliberately gave up. */ + const writerClaimStatus = ctx.writerClaim.status(); + let sessionChainIntegrity; if (headError) { // HEAD.json is already known corrupt/unreadable -- don't bother attempting the walk // (which needs `head`) just to rediscover the same failure less clearly. sessionChainIntegrity = { status: "failed", evidence: [], assumptions: [], failure_domain: "trusted-core", reason: `HEAD.json is corrupt or unreadable: ${headError}` }; } else { + /* Round-1 fix-plan item 1, round-2 fix pass: this walk is exactly the "long + * synchronous phase held under a writer-claim" case commit 1's own plan named + * alongside reconcileHead's startup walk and the WAL-replay loops -- but unlike + * those two (each run exactly once, before any release() can have happened yet), + * this walk also runs on every ~STATUS_WRITE_INTERVAL_MS tick of an already-RUNNING + * gateway, so it must only attempt renewal while a claim is genuinely still held by + * THIS instance (writerClaimStatus, above). createLeaseGuard is the SAME shared + * helper recoverCrashedSessions/reconcileHead's own call sites already use -- no new + * renewal mechanism. */ + const leaseGuard = writerClaimStatus.held_by_this_instance ? createLeaseGuard(ctx.writerClaim) : null; try { + /* Round-N fix (frontier-panel finding #8, second exposure): chain.readChain now + * streams the chain file in bounded chunks instead of reading it whole into a V8 + * string (which has a hard ~512MB string-length ceiling -- see chain.js's own header + * for the full writeup), but a chain large enough to take a while to stream through + * still needs the SAME per-progress lease renewal the per-entry walk below already + * gets, since this call runs on every ~STATUS_WRITE_INTERVAL_MS tick of an + * already-RUNNING, healthy gateway -- not just once at startup like reconcileHead. + * Passing leaseGuard.maybeRenew() as onProgress closes that gap the same way + * reconcileHead's own onProgress wiring does; a detected lease loss here propagates + * through this readChain call exactly like it already does through the per-entry + * maybeRenew below, into this try block's own catch. */ sessionChainIntegrity = registerGatewaySessions.run({ - chain: chain.readChain(stateDir), + chain: chain.readChain(stateDir, leaseGuard ? { onProgress: () => leaseGuard.maybeRenew() } : undefined), head, computeEntrySha256: chain.computeEntrySha256, bundleExists: (bundleId) => fs.existsSync(chain.bundlePath(stateDir, bundleId)), + ...(leaseGuard + ? { maybeRenew: () => leaseGuard.maybeRenew(), isLeaseError: (error) => leaseGuard.isLeaseError(error) } + : {}), }); } catch (error) { + /* A lease loss DETECTED by this walk's own maybeRenew() (wired in just above) + * must not be downgraded to this catch's ordinary "report a failed + * session_chain_integrity, evidence only" handling -- createLeaseGuard's own doc + * comment (above) is explicit that a detected lease loss must never be relabeled + * as "flag for operator review, continue". Rethrown here, it propagates out of + * buildHealthStatus entirely -- this function's every OTHER internal step already + * has its own local try/catch that never lets an exception escape (headError, + * lastPersistedAt, recoveryStatus below), so this is buildHealthStatus's only + * throw path. See writeStatusFile's own doc comment for what happens next. */ + if (leaseGuard && leaseGuard.isLeaseError(error)) throw error; sessionChainIntegrity = { status: "failed", evidence: [], assumptions: [], failure_domain: "trusted-core", reason: `session-chain verification threw: ${error.message}` }; } } + /* Option C: surface every intent an operator still needs to act on (or that is merely + * "dispatched" and not yet stale enough to worry about) so this is discoverable without + * an operator having to already know a crash happened -- the whole point of a durable, + * un-auto-expired fence (recovery.js's own header) is defeated if nothing ever points an + * operator at it. Read directly from disk rather than cached from the one-time startup + * recoverCrashedSessions() pass, since a NEW ambiguous intent can appear at any time + * during normal operation (proxy.js's closeConnection fences a "dispatched" intent + * ambiguous the moment its owning connection closes mid-call, not just at startup). */ + let recoveryStatus; + try { + const allIntents = recovery.listAllIntents(stateDir); + /* Codex PR #33 review "report each unresolved intent key in recovery output": a + * "dispatched" intent is normal (a live call genuinely in flight on a healthy + * connection) on most gateways most of the time, BUT it is also exactly the state a + * crashed connection's own unresolved call is left in when startup recovery could not + * prove its outcome (recoverCrashedSessions' RECOVERY_AMBIGUOUS_INTENT case) -- the + * aggregate `in_flight` count alone cannot tell those two apart, and an operator + * cannot act on a plain number. Itemize the same way `pending_operator_review` + * already does for `ambiguous` intents, so a stuck one is directly actionable + * (recovery-resolve/recovery-abandon both require the exact intent key) without first + * having to know a crash happened and go hunting through raw recovery files. */ + const dispatched = allIntents + .filter((i) => i.state === "dispatched") + .map((i) => ({ connection_id: i.connection_id, intent_key: i.intent_key, tool: i.tool, dispatched_since: i.dispatched_at || null })); + recoveryStatus = { + pending_operator_review: allIntents + .filter((i) => i.state === "ambiguous") + .map((i) => ({ connection_id: i.connection_id, intent_key: i.intent_key, tool: i.tool, ambiguous_since: i.ambiguous_at || null, reason: i.ambiguous_reason || null })), + in_flight: dispatched.length, + dispatched, + }; + } catch (error) { + recoveryStatus = { pending_operator_review: [], in_flight: 0, dispatched: [], error: error.message }; + } + return { schema_version: "1.0", - writer_claim: ctx.writerClaim.status(), + writer_claim: writerClaimStatus, downstream_servers: Array.from(ctx.connections.keys()).map((name) => { const conn = ctx.connections.get(name); return { name, reachable: typeof conn.isReachable === "function" ? conn.isReachable() : !conn.isClosed() }; @@ -461,10 +762,801 @@ function buildHealthStatus(ctx) { ? { seq: head.seq, entry_sha256: head.entry_sha256, last_persisted_at: lastPersistedAt } : { seq: 0, entry_sha256: null, last_persisted_at: lastPersistedAt, ...(headError ? { error: headError } : {}) }, session_chain_integrity: sessionChainIntegrity, + recovery: recoveryStatus, remote_anchor: { implemented: false, reason: "SG-FR-6 not implemented in this build -- see chain.js#pushChainTailToRemoteAnchor" }, }; } +/** Shared by recoverCrashedSessions and abandonConnection's own GATEWAY_BUNDLE_ID_ + * COLLISION handling (Codex PR #33 review "verify bundle collisions before discarding + * recovery state", and its follow-up "verify collisions in recovery-abandon before + * cleanup"): verified directly against gsa-mcp-shim.js: bundle_id = sha256({init, + * grantedTools, n: calls.length}) -- a coarse fingerprint with no timestamp, nonce, or + * actual call content. Re-running recovery/abandon for the SAME crash deterministically + * reproduces the same bundle_id, which is "already durably appended," not a real + * conflict -- but a DIFFERENT crashed session that merely happens to share {init, + * grantedTools, call count} would ALSO collide here, and blindly trusting the id match + * would then discard that other session's real WAL and completed intents, permanently. + * Reads back the bundle actually on disk and compares its real content (the execution + * trace, itself a hash of every call's real arguments/result) before treating this as + * the expected repeated-attempt case. Returns true only when content is read back and + * verified identical; any read/parse failure or mismatch returns false (an unverified + * collision must never be treated as "safe to clean up"). */ +function bundleCollisionIsSameContent(stateDir, sealed) { + try { + const existingRaw = fs.readFileSync(chain.bundlePath(stateDir, sealed.bundle.manifest.bundle_id), "utf8"); + const existingBundle = JSON.parse(existingRaw); + // gsa-produce.js#produceBundle never stores artifact bodies on the bundle itself -- + // manifest.artifacts..sha256 is the real per-artifact content fingerprint (the + // raw bodies live in bundle.contents, keyed by file path, but the hash already IS the + // exact equality check needed here). execution_trace alone (per-call input/result + // hashes, tool, granted, error/model flags) is sufficient: it is a hash of every + // call's real arguments and result, so two sessions cannot share it without sharing + // their actual call content. + const existingHash = existingBundle.manifest && existingBundle.manifest.artifacts && existingBundle.manifest.artifacts.execution_trace && existingBundle.manifest.artifacts.execution_trace.sha256; + const newHash = sealed.bundle.manifest && sealed.bundle.manifest.artifacts && sealed.bundle.manifest.artifacts.execution_trace && sealed.bundle.manifest.artifacts.execution_trace.sha256; + return Boolean(existingHash) && existingHash === newHash; + } catch (readError) { + return false; // could not verify -- treat as a genuine, unverified conflict + } +} + +/** Codex PR #33 review "verify the chain entry before cleaning a colliding WAL": content + * verification alone (bundleCollisionIsSameContent above) proves the bundle FILE on disk + * (chain.appendSession's own step 1) really is this connection's own durable record -- it + * does NOT prove step 2 (the chain.jsonl append) ever happened. If chain.appendSession + * crashed after writing the bundle file but before appending chain.jsonl, a recovery pass + * that stops at content verification would still delete this connection's WAL/intents as + * "already durably sealed," permanently omitting it from the attestation chain even though + * it was never actually chain-appended. Shared by recoverCrashedSessions and + * abandonConnection (same two callers bundleCollisionIsSameContent already serves) so both + * paths get the identical repair, not just content verification. Returns true (safe to + * clean up this connection's WAL/intents now -- durably sealed, chain entry confirmed + * present, repairing it first if it was merely missing) or false (unverified conflict with + * a genuinely different session; caller must not clean up). May throw if content is + * verified but the repair write itself fails -- callers must not clean up in that case + * either, since the chain still does not durably reference this bundle. */ +/* Round-1 fix-plan item 1 (lease keepalive, generalized): a small per-writerClaim wrapper + * shared by recoverCrashedSessions/abandonConnection (and verifyAndRepairBundleCollision, + * which both of those call into) so every call site renews the SAME claim through the SAME + * two methods -- writer-claim.js's own time-gated maybeRenew() for the WAL-replay per-event + * loop, and its unconditional renew() immediately before any write that mutates shared + * chain state (chain.appendSession, chain.repairMissingChainEntry). `writerClaim` may be + * null (existing direct callers/tests that construct no real claim) -- both wrapped methods + * then become no-ops, exactly like the bare `if (writerClaim) writerClaim.renew()` guard + * they replace. + * + * isLeaseError(error) lets a catch block further up this same call chain recognize a + * failure that ALREADY passed through this guard's own renew()/maybeRenew() and distinguish + * it from an unrelated failure (a chain-append error, a repair-write error) caught at the + * same try/catch -- by reference identity, not by error.code string matching, since + * writer-claim.js's renew() can propagate a raw fs error (e.g. EIO) that carries no + * WRITER_CLAIM_LOST code at all. A detected lease loss must never be downgraded to + * per-connection "flag for operator review, continue" handling -- that per-connection + * recovery of DIFFERENT connections' state under a claim this process may no longer + * exclusively hold is exactly the split-brain condition this fix exists to prevent, so + * every catch block that checks isLeaseError rethrows immediately instead, letting it + * propagate all the way out (recoverCrashedSessions' caller, startGateway, already releases + * the claim and rethrows -- see its own comment above -- making a detected lease loss fatal + * to the whole process). + * + * DISCLOSED, NOT FIXED HERE: this narrows the exposure window (a claim renewed just before + * each mutating write is checked far more often than one renewed only by the heartbeat + * interval) but does NOT provide atomic fencing -- a competitor could still acquire between + * this guard's own renew() call returning and the write it guards actually landing. Never + * describe this as "preventing corruption"; see writer-claim.js's own maybeRenew() doc + * comment for the same disclosure at the primitive's own level. */ +function createLeaseGuard(writerClaim) { + let leaseError = null; + return { + renew() { + if (!writerClaim) return; + try { + writerClaim.renew(); + } catch (error) { + leaseError = error; + throw error; + } + }, + maybeRenew() { + if (!writerClaim || typeof writerClaim.maybeRenew !== "function") return; + try { + writerClaim.maybeRenew(); + } catch (error) { + leaseError = error; + throw error; + } + }, + // Reference-identity check, deliberately not `error.code === "WRITER_CLAIM_LOST"` -- + // see this function's own doc comment above. + isLeaseError(error) { + return error === leaseError; + }, + }; +} + +/* Round-1 fix-plan commit 4 rewrite (docs/contracts/chain-validity.md's C2): the + * original shape of this check ("does chain.jsonl contain SOME entry for this + * bundle_id") never actually compared HEAD to anything -- it is strengthened here to + * also assert that HEAD genuinely names the chain's own CURRENT TAIL, not merely that + * this bundle's own entry exists somewhere in the chain. Critically, this new check + * does NOT compare HEAD to THIS bundle's own entry (a draft of this fix did, and that + * would have been wrong): a bundle's chain entry can legitimately be an ANCESTOR of a + * verified, HEAD-consistent tail -- e.g. this exact connection's own WAL survived a + * failed post-seal cleanup, a genuinely newer bundle was committed afterward, and HEAD + * correctly points at that newer bundle. Comparing HEAD to "this entry" would refuse + * that valid, ordinary case; comparing HEAD to the chain's own tail does not. */ +function verifyAndRepairBundleCollision(stateDir, sealed, log, leaseGuard = null) { + if (!bundleCollisionIsSameContent(stateDir, sealed)) return false; + const bundleId = sealed.bundle.manifest.bundle_id; + + /* Round-N fix (frontier-panel finding #8): same streaming reader + onProgress renewal + * as buildHealthStatus and reconcileHead above -- this call is reached from inside a + * per-connection leaseGuard-held recovery/repair path, so a chain large enough to take + * a while to stream through must keep renewing the claim while doing so, not just via + * the caller's own once-per-connection renew() before this function was entered. */ + const chainEntries = chain.readChain(stateDir, leaseGuard ? { onProgress: () => leaseGuard.maybeRenew() } : undefined); + const ownEntry = chainEntries.find((e) => e.bundle_id === bundleId); + + if (!ownEntry) { + if (leaseGuard) leaseGuard.renew(); + chain.repairMissingChainEntry(stateDir, bundleId, log); + log( + `recovery: bundle "${bundleId}" was durably written but never chain-appended (crash between ` + + `chain.appendSession's own bundle-write and chain.jsonl-append steps) -- completed the missing ` + + `chain.jsonl/HEAD.json append now.` + ); + return true; + } + + // This bundle's own chain entry already exists (possibly not as the tail -- see this + // function's own doc comment above). What still needs verifying is that HEAD itself + // has not diverged from the chain as a whole: reuse the exact same append-time + // classifier appendSession/repairMissingChainEntry use (single-step auto-catch-up, + // anything else refuses and latches) rather than hand-rolling a third comparison. + const head = (() => { + try { + return chain.readHead(stateDir); + } catch (error) { + return null; // an unreadable HEAD is handled the same as "missing" by the classifier below. + } + })(); + chain.checkHeadAgainstTailOrRepair(stateDir, head, log); + return true; +} + +/** Startup crash-recovery pass (Codex PR #29 Finding 1, Option C -- external-panel- + * reviewed design, see option-c-hardened-design.md). Runs once, after the writer-claim + * is acquired (so this process is the sole owner of state_dir) and BEFORE any downstream + * connection or agent-facing listener starts, replaying any WAL a prior crashed instance + * left behind under recovery.js's `gateway-recovery/active/` directory. + * + * Replay reconstructs a session using session.js's real, UNMODIFIED recorder functions + * (recordInitialize/recordToolsList/recordCallStart/recordCallResult/ + * markPendingAsDisconnected) -- there is no second, parallel session shape to keep in + * sync with the live one. A call with no matching CALL_RESULT event is exactly Finding + * 1's "crash after a completed call but before disconnect" gap turned inside out: it was + * IN FLIGHT at crash time, so its real intent record (recovery.js's idempotency store, + * Finding 2) is consulted for a proven outcome; only a session with every call proven + * terminal is auto-sealed. Anything else is left alone and reported -- never guessed. */ +function recoverCrashedSessions(stateDir, keys, log, writerClaim = null) { + const pendingOperatorReview = []; + const leaseGuard = createLeaseGuard(writerClaim); + /* Codex + CodeRabbit PR #33 review "sweep expired completed-signature records": the + * 24-hour retention on the completed-signature store was only ever enforced per-key, on + * read, so signatures for calls with unique arguments were never revisited and never + * deleted. Startup recovery is the bounded, once-per-process pass both reviewers named + * as the right place to reclaim them: it already owns the writer claim, already walks + * this state directory, and runs before any traffic is accepted. Wrapped so a sweep + * failure can never keep a gateway from starting -- reclaiming disk space is + * housekeeping, not a correctness gate, and the dispatch path's own per-key expiry check + * (recovery.readCompletedSignature) remains the authority on whether a record is + * replayable. */ + try { + const swept = recovery.sweepExpiredSignatures(stateDir); + if (swept.removed > 0) { + log(`recovery: swept ${swept.removed} expired completed-call signature record(s) of ${swept.scanned} scanned (retention ${recovery.DEFAULT_RETAINED_SIGNATURE_TTL_MS}ms).`); + } + } catch (error) { + log(`recovery: expired-signature sweep skipped (${error.message}) -- continuing startup; this is space reclamation only.`); + } + for (const connectionId of recovery.listActiveConnections(stateDir)) { + /* Round-1 fix-plan commit 6: a genuine chain-integrity failure latched by ANY + * earlier connection's own append-time check in THIS SAME loop (chain.appendSession + * / verifyAndRepairBundleCollision below, both routed through chain.js's + * checkHeadAgainstTailOrRepair) means every remaining connection in this loop is + * guaranteed to hit the identical genuine failure the instant its own + * chain.appendSession runs -- the corruption is a property of the shared chain, not + * of any one connection's WAL. Checking chain.getChainIntegrityFailure() here, before + * spending another connection's worth of fsyncs parsing and replaying its WAL only to + * rediscover the same already-known failure, is what "abort immediately" (round-1 + * fix-round plan, commit 6) means in practice -- this loop has no way to unlatch + * mid-run (only a restart re-runs reconcileHead), so continuing is strictly wasted + * work, never a second chance at success. Left-untouched connections' WAL/intent + * state is exactly as safe to leave in place as any other connection this function + * already flags for operator review -- a future restart (after the corruption is + * fixed per the runbook) replays them normally. */ + const alreadyLatched = chain.getChainIntegrityFailure(); + if (alreadyLatched) { + log( + `RECOVERY ABORTED: a chain-integrity failure is already latched (${alreadyLatched.reason}` + + `${alreadyLatched.class ? `, class=${alreadyLatched.class}` : ""}, at=${alreadyLatched.at}) -- refusing ` + + `to replay any further crash-left WAL, since every remaining connection would be quarantined by the ` + + `identical failure. Leaving their WAL/intent state untouched for investigation. See ` + + `${chain.CHAIN_CORRUPTION_RUNBOOK} for the recovery procedure.` + ); + break; + } + /* Codex PR #33 review "renew ownership during synchronous recovery": this whole + * per-connection body below is synchronous fs work (WAL replay, session.finalizeSession, + * chain.appendSession) -- with enough or large enough crash-left WALs it can run past + * writer-claim.js's own staleAfterMs before ever yielding to the event loop, which is + * the only place the heartbeat startHeartbeat() already started can actually fire. + * writer-claim.js's own startHeartbeat() doc comment discloses exactly this gap and + * prescribes a synchronous renew() call at the boundary of any long synchronous phase + * this claim is held across (mirroring state-store.js's own _assertStillOwned fix) -- + * done here once PER CONNECTION, not once for the whole loop, so many small WALs are + * covered the same as one huge one. `writerClaim` is optional (existing direct callers/ + * tests that construct no real claim keep working, renewal simply skipped) but + * startGateway's own real call site below always passes its live claim. A failed renew + * means another process may already hold this state_dir -- propagate immediately + * (uncaught by this loop's own per-connection try/catch further down) rather than keep + * mutating shared chain/WAL state under a claim that might no longer be exclusive. + * Round-1 fix-plan item 1: routed through createLeaseGuard's own leaseGuard.renew() + * (rather than a bare writerClaim.renew() call) so a failure here is later + * recognizable by isLeaseError -- see this loop's own outer catch below. */ + leaseGuard.renew(); + /* CodeRabbit PR #33 review "continue recovery after an unreadable connection state": + * readWalEvents/readIntent below can throw (GATEWAY_RECOVERY_WAL_UNREADABLE, + * GATEWAY_RECOVERY_INTENT_UNREADABLE, GATEWAY_RECOVERY_INTENT_CORRUPT) on a genuine + * fs-level read failure for THIS connection's own state -- a permissions or I/O + * problem, not a content/JSON one (which readWalEvents already handles by stopping + * at the first bad line, not throwing). Previously unguarded, so one such connection + * aborted recovery for every other connection still waiting in this same loop. Wrap + * the whole per-connection body so a failure here is isolated to this connection -- + * flagged for the operator (whose recovery-abandon can now quarantine an unreadable + * WAL without needing to re-read it -- see abandonConnection below) and otherwise + * left alone, exactly like this function's other own failure branches already do. */ + try { + const events = recovery.readWalEvents(stateDir, connectionId); + if (events.length === 0) { + // Every line was torn (crash mid-write of the very first event) or the file was + // empty -- nothing durable was ever recorded for this connection. + recovery.deleteWal(stateDir, connectionId); + continue; + } + + const startEvent = events.find((e) => e.type === "SESSION_START"); + // Cluster B: reuse the exact session_id this session's own SESSION_START event + // persisted (proxy.js#openConnection) rather than letting session.createSession + // mint a fresh one -- replaying the SAME crash-left WAL twice (e.g. this recovery + // pass itself crashing before cleanup, see recoverIsIdempotentAcrossACrashDuring + // RecoveryItself in tests/gateway/recovery/run-tests.js) must produce the SAME + // bundle_id both times. An older WAL written before this field existed has no + // session_id on its SESSION_START event; passed through as `null` explicitly + // (never generated here) for the same reason -- see session.js#createSession's own + // doc comment on the three-way distinction this argument makes. + const sessionId = startEvent && typeof startEvent.session_id === "string" && startEvent.session_id.length > 0 ? startEvent.session_id : null; + const s = session.createSession(connectionId, { now: () => Date.now(), goal: startEvent ? startEvent.goal : undefined, sessionId }); + // Maps this replay's own stable Symbol.for() keys back to the CALL_START event that + // produced them, needed only to look up that call's real intent record below. + const keyToStartEvent = new Map(); + + for (const event of events) { + /* Round-1 fix-plan item 1: this per-event loop is exactly the "long synchronous + * phase this claim is held across" the per-connection leaseGuard.renew() above + * already anticipates for the whole-connection case -- a single sufficiently large + * WAL can itself run long enough between yields to the event loop that even a + * once-per-connection renew is not enough. maybeRenew() is time-gated by the + * claim's own clock (never Date.now()), so this costs nothing beyond an in-memory + * comparison on every iteration that doesn't actually need to renew. */ + leaseGuard.maybeRenew(); + if (event.type === "SESSION_START") { + session.recordToolsList(s, event.tools || []); + } else if (event.type === "INITIALIZE") { + session.recordInitialize(s, { clientInfo: event.clientInfo, serverInfo: event.serverInfo, model: event.model }); + } else if (event.type === "CALL_START") { + // Symbol.for (the global registry), not Symbol(): this replay's own CALL_START + // and CALL_RESULT events for the same call_seq must resolve to the SAME symbol + // reference for session.js's Map-keyed pendingCalls to correlate them -- a live + // correlationKey may have been a non-reproducible Symbol() or a real JSON-RPC id, + // but replay only needs internal consistency within itself, not to match what + // the crashed process originally used. + const key = Symbol.for(`wal-replay:${connectionId}:${event.call_seq}`); + keyToStartEvent.set(key, event); + session.recordCallStart(s, key, { tool: event.tool, server: event.server, arguments: event.arguments, isModelCall: Boolean(event.isModelCall), ts: event.ts }); + } else if (event.type === "CALL_RESULT") { + const key = Symbol.for(`wal-replay:${connectionId}:${event.call_seq}`); + /* Codex PR #33 review "persist cached replays in the session trace": carry the + * replay marker through crash recovery too, so a WAL replayed after a crash + * reconstructs the SAME sealed shape a clean close would have produced for that + * step -- an idempotency cache hit must not silently become an ordinary call + * just because the gateway restarted before sealing. Absent on every ordinary + * CALL_RESULT event (including every WAL written before this field existed). */ + if (s.pendingCalls.has(key)) session.recordCallResult(s, key, { result: event.result, isError: event.isError, ts: event.ts, replayed: Boolean(event.replayed), replayedFromIntentKey: event.intent_key }); + } else if (event.type === "ANOMALY") { + // Codex PR #33 review "append blocked-retry anomalies to the WAL": replays a + // dispatch-guard block (GATEWAY_AMBIGUOUS_RETRY/GATEWAY_DOWNSTREAM_OUTCOME_UNKNOWN) + // that was recorded in-memory only via session.recordAnomaly in the live proxy -- + // without this, a crash before the connection's own close would silently drop + // that attestation from the recovered/sealed bundle. + session.recordAnomaly(s, { kind: event.kind, tool: event.tool, intent_key: event.intent_key, detail: event.detail, ts: event.ts }); + } + // CLOSING is informational only for replay -- if present, the crash happened + // between "closeConnection began" and "chain.appendSession completed", which the + // GATEWAY_BUNDLE_ID_COLLISION handling below already covers. + } + + // Any call still pending after replaying every event was in flight at crash time. + // Consult its real intent record for a proven outcome before deciding anything. + let needsOperator = false; + // Codex PR #33 review "include the intent key in the advertised resolution + // command": collected as we go so the RECOVERY_AMBIGUOUS_INTENT log below can name + // every actual unresolved key, rather than the literal "" placeholder it used + // to print regardless of how many calls (or which ones) were actually unresolved -- + // recovery-resolve requires the exact intent key, which an operator otherwise had to + // discover by hand-parsing raw recovery files. + const unresolvedIntentKeys = []; + for (const [key] of Array.from(s.pendingCalls.entries())) { + const startedFrom = keyToStartEvent.get(key); + if (!startedFrom) continue; // defensive; should not happen + if (startedFrom.isModelCall) { + /* A downstream-initiated sampling/createMessage forward (see + * forwardDownstreamRequestToAgent's own WAL comment above) has no + * idempotency-intent fence at all -- Finding 2's fence is deliberately scoped to + * real tools/call dispatch only, so there is no intent record to consult here and + * never will be. Unlike a fenced tool call, there is also no external side effect + * this gateway must avoid duplicating: the only real risk of an unresolved + * sampling call is an incomplete attestation of what the agent was asked, which + * this WAL replay has already captured (the CALL_START event, recorded above). + * Record it as a real, disconnected/unproven result and move on rather than + * blocking this whole connection's auto-seal on operator review for a call that + * has nothing an operator could actually resolve (there is no --result-file or + * --confirmed answer that applies to "did the agent's model happen to finish + * replying before the crash"). */ + session.recordCallResult(s, key, { result: { error: "gateway restarted after a crash while this downstream-initiated sampling/createMessage forward was still awaiting the agent's response" }, isError: true, ts: Date.now() }); + continue; + } + const intentKey = recovery.computeIntentKey(connectionId, startedFrom.tool, startedFrom.arguments); + const intent = recovery.readIntent(stateDir, intentKey); + /* Cluster A (generation-aware crash recovery): the intent file's own `generation` + * field is overwritten IN PLACE on every supersede (proxy.js's dispatch guard), + * so by the time recovery runs it always reflects the MOST RECENT generation -- + * which is not necessarily the generation THIS specific crashed CALL_START event + * belongs to. Concretely: generation 1 completes, but its own CALL_RESULT WAL + * write fails (best-effort, logged, not fatal -- see proxy.js's own comment on + * that append); the SAME connection later dispatches generation 2 of the exact + * same (tool, arguments), which also completes and DOES get its CALL_RESULT + * durably written. A later crash then leaves generation 1's CALL_START forever + * unresolved in the WAL while the intent file now sits at generation 2. Without + * this check, the loop below would bind generation 2's cached_result (a + * DIFFERENT call's real outcome) to generation 1's own pending call -- a false + * attestation. Comparing against the generation THIS CALL_START event itself + * recorded (defaulting to 1 for a WAL line written before this field existed) + * ensures a proven outcome is only ever applied to the generation it actually + * belongs to; any other generation's crashed call is left for operator review, + * exactly like a call with no intent record at all. */ + const expectedGeneration = typeof startedFrom.generation === "number" ? startedFrom.generation : 1; + const intentGeneration = intent && typeof intent.generation === "number" ? intent.generation : 1; + if (intent && intent.state === "completed" && intentGeneration === expectedGeneration) { + session.recordCallResult(s, key, { result: intent.cached_result, isError: false, ts: Date.now() }); + } else if (intent && intent.state === "not_executed" && intentGeneration === expectedGeneration) { + /* Codex PR #33 review "persist a terminal not-executed resolution": an operator + * already answered "did it execute?" (no) via recovery-resolve while this + * connection was crashed -- record that as a real (failed) terminal result now, + * rather than re-flagging the same connection for operator review forever + * because no "completed" intent will ever appear for a call that never ran. */ + session.recordCallResult(s, key, { result: { error: "operator confirmed via recovery-resolve that this call did not execute downstream" }, isError: true, ts: Date.now() }); + } else { + needsOperator = true; + unresolvedIntentKeys.push(intentKey); + /* Round-N fix (frontier-panel finding #5, Paul's explicit choice of Fable's + * "make the gap observable" proposal): a pending call with startedFrom.replayed + * true is not an ordinary "in flight at crash time" case -- proxy.js's own + * deliberate fail-OPEN cached-replay branch means the CALL_START for this + * completed-signature cache hit WAS durably written, yet its matching + * CALL_RESULT never made it into the WAL before the crash, so the agent may + * already have received a real result for this exact step. That distinction is + * deliberately NOT recorded here: this function never calls + * session.finalizeSession/chain.appendSession for a needsOperator connection -- + * see the `continue` a few lines below -- so any anomaly attached to `s` at this + * point would be discarded along with the rest of this in-memory session, + * never reaching a sealed bundle (session.recordAnomaly has no side effect + * beyond mutating that in-memory object). The actual "INCOMPLETE: step N" + * marker Paul asked for is recorded in abandonConnection instead -- + * recovery-abandon is the only path that unconditionally seals a connection + * like this one -- which performs the identical startedFrom.replayed check on + * its own WAL replay right before it seals. See that function's own doc + * comment for the full rationale. */ + } + } + + if (needsOperator) { + pendingOperatorReview.push(connectionId); + const resolveCommands = unresolvedIntentKeys + .map((key) => `"node scripts/gateway/gateway.js recovery-resolve --connection ${connectionId} --intent ${key} --confirmed executed|not-executed"`) + .join(", "); + log( + `RECOVERY_AMBIGUOUS_INTENT: connection "${connectionId}" crashed with ${unresolvedIntentKeys.length} call(s) in flight whose outcome is not proven (intent key(s): ${unresolvedIntentKeys.join(", ")}) -- ` + + `leaving its WAL and intent record(s) in place rather than guessing. Resolve each with ${resolveCommands}, ` + + `or "node scripts/gateway/gateway.js recovery-abandon --connection ${connectionId}" to seal only the calls that did reach a proven outcome.` + ); + continue; // do not touch this connection's WAL/intents any further + } + + if (s.pendingCalls.size > 0) { + session.markPendingAsDisconnected(s, "gateway restarted after a crash; connection never received a clean close", () => Date.now()); + } + let sealed; + try { + sealed = session.finalizeSession(s, keys); + } catch (error) { + log(`RECOVERY SEAL FAILURE for connection "${connectionId}": ${error.message} -- leaving its WAL in place for investigation.`); + pendingOperatorReview.push(connectionId); + continue; + } + // Round-1 fix-plan item 1: unconditional renew() (not maybeRenew()) immediately + // before every chain.appendSession call, in both this normal path and the repair + // path below -- this is the one write per connection that actually mutates the + // shared chain, so it gets its own fresh, unconditional renewal regardless of how + // recently the per-event maybeRenew() above last renewed. + leaseGuard.renew(); + try { + chain.appendSession(stateDir, sealed, { log }); + } catch (error) { + if (error.code === "GATEWAY_BUNDLE_ID_COLLISION") { + let verified; + try { + verified = verifyAndRepairBundleCollision(stateDir, sealed, log, leaseGuard); + } catch (repairError) { + // A lease loss detected by verifyAndRepairBundleCollision's own leaseGuard.renew() + // must not be relabeled as an ordinary chain-repair failure -- rethrow so it + // reaches this function's own outer catch below and propagates as fatal. + if (leaseGuard.isLeaseError(repairError)) throw repairError; + log(`RECOVERY CHAIN-REPAIR FAILURE for connection "${connectionId}": ${repairError.message} -- leaving its WAL in place for investigation.`); + pendingOperatorReview.push(connectionId); + continue; + } + if (verified) { + log(`recovery: connection "${connectionId}" was already durably sealed (bundle_id collision, content-verified match) -- cleaning up.`); + } else { + log( + `RECOVERY BUNDLE COLLISION for connection "${connectionId}": bundle_id "${sealed.bundle.manifest.bundle_id}" ` + + `already exists on disk but its recorded content does not match this connection's recovered session -- ` + + `a genuine conflict between two different sessions, not a repeated recovery attempt of the same one. ` + + `Leaving this connection's WAL and intents in place for investigation rather than discarding them.` + ); + pendingOperatorReview.push(connectionId); + continue; + } + } else { + log(`RECOVERY CHAIN-APPEND FAILURE for connection "${connectionId}": ${error.message} -- leaving its WAL in place for investigation.`); + pendingOperatorReview.push(connectionId); + continue; + } + } + recovery.deleteWal(stateDir, connectionId); + for (const intent of recovery.listIntentsForConnection(stateDir, connectionId)) { + if (intent.state === "completed" || intent.state === "not_executed") recovery.deleteIntent(stateDir, intent.intent_key); + } + log(`recovery: connection "${connectionId}" recovered and sealed from a crash-left WAL (${s.calls.length} call(s)).`); + } catch (error) { + // Round-1 fix-plan item 1: a lease loss detected anywhere in this connection's own + // body above (the per-connection renew(), the per-event maybeRenew(), the pre-append + // renew(), or a repair-path renew() rethrown from the inner catch above) must never + // be downgraded to this function's own per-connection "flag for operator review, + // continue with the next connection" handling -- that would keep mutating OTHER + // connections' chain/WAL state under a claim this process may no longer exclusively + // hold, which is exactly the split-brain condition this fix exists to prevent. + // Rethrow so it propagates out of this whole loop/function -- startGateway's own + // call site above already releases the claim and rethrows again, making a detected + // lease loss fatal to the whole process, not just this one connection. + if (leaseGuard.isLeaseError(error)) throw error; + log(`RECOVERY FAILURE for connection "${connectionId}": ${error.message} (${error.code || "no code"}) -- leaving its state in place for investigation. Use "recovery-abandon --connection ${connectionId}" to quarantine it without needing to re-read that state.`); + pendingOperatorReview.push(connectionId); + } + } + return { pendingOperatorReview }; +} + +/** Operator override for a connection stuck in `pendingOperatorReview` (recovery-abandon + * CLI, see main() below): unlike recoverCrashedSessions' own auto-seal path, this treats + * EVERY call still pending after WAL replay as terminal regardless of whether its intent + * ever reached a proven "completed" outcome -- an explicit "I am giving up on finding out + * whether this executed, seal what I have" decision, not a guess made on the operator's + * behalf. Any intent still `dispatched`/`ambiguous` for this connection is deleted + * afterward: the durable fence Finding 2 provides is deliberately given up on for THIS + * specific connection, by explicit operator action, not silently -- if the downstream + * operation actually executed, a future identical call is no longer fenced against + * re-running it. This tradeoff (and that it is scoped per-connection, never global) is + * documented in KNOWN-LIMITATIONS.md. */ +function abandonConnection(stateDir, keys, connectionId, log, writerClaim = null) { + /* Codex PR #33 review "renew ownership during synchronous recovery": same rationale as + * recoverCrashedSessions' own per-connection renew() above, applied to this CLI's own + * synchronous WAL replay/finalize/chain-append for its one connection -- "sufficiently + * large" applies just as well to a single big WAL as to many small ones. Round-1 + * fix-plan item 1: routed through createLeaseGuard so a lease loss anywhere in this + * function is recognizable by isLeaseError below and never relabeled as an ordinary + * chain-repair/abandon failure; a lease-loss failure here is simply left uncaught + * (this function has no per-connection try/catch of its own to swallow it), propagating + * out to runRecoveryAbandonCli's caller exactly like any other uncaught error already + * would -- fatal to this CLI invocation, consistent with recoverCrashedSessions' own + * process-fatal handling. */ + const leaseGuard = createLeaseGuard(writerClaim); + leaseGuard.renew(); + /* CodeRabbit PR #33 review "continue recovery after an unreadable connection state": this + * command is recoverCrashedSessions' own documented remediation path for a connection + * it could not process -- including one whose WAL could not even be READ (permissions/ + * I/O error, not a content problem). Before this, that same unreadable-WAL error would + * simply be thrown again here, leaving the operator with no way to actually resolve + * it. Quarantine (rename, not delete -- see recovery.quarantineWal's own doc comment) + * instead: an operator who explicitly asked to abandon this connection does not need + * this command to succeed at reading state it already knows is unreadable, only to stop + * this connection from blocking every future startup. */ + let events; + try { + events = recovery.readWalEvents(stateDir, connectionId); + } catch (error) { + const quarantined = recovery.quarantineWal(stateDir, connectionId); + for (const intent of recovery.listIntentsForConnection(stateDir, connectionId)) recovery.deleteIntent(stateDir, intent.intent_key); + log( + `recovery-abandon: connection "${connectionId}"'s WAL could not be read (${error.message}) -- ` + + (quarantined ? `moved it to "${quarantined}" for manual inspection` : "no WAL file was found to quarantine") + + `, and released any of its idempotency fence(s). This connection was NOT sealed into the chain ` + + `(there is nothing readable to seal from) -- if it made any real downstream calls, that record is now unrecoverable through this gateway.` + ); + return; + } + if (events.length === 0) { + recovery.deleteWal(stateDir, connectionId); + for (const intent of recovery.listIntentsForConnection(stateDir, connectionId)) recovery.deleteIntent(stateDir, intent.intent_key); + log(`recovery-abandon: connection "${connectionId}" had no WAL to replay -- nothing to seal, cleaned up any leftover intents.`); + return; + } + const startEvent = events.find((e) => e.type === "SESSION_START"); + // Cluster B: same reuse-not-regenerate rule as recoverCrashedSessions' own replay + // above -- see that call site's doc comment and session.js#createSession's. + const sessionId = startEvent && typeof startEvent.session_id === "string" && startEvent.session_id.length > 0 ? startEvent.session_id : null; + const s = session.createSession(connectionId, { now: () => Date.now(), goal: startEvent ? startEvent.goal : undefined, sessionId }); + // Round-N fix (frontier-panel finding #5): mirrors recoverCrashedSessions' own + // keyToStartEvent map (see that function's replay loop above) -- needed here for the + // identical reason: to look back from a still-pending call to the CALL_START event + // that produced it, so a replayed-but-incomplete cache hit can be distinguished from + // an ordinary interrupted call once this replay finishes. + const keyToStartEvent = new Map(); + for (const event of events) { + // Round-1 fix-plan item 1: same time-gated renewal as recoverCrashedSessions' own + // per-event loop above, for this CLI's own single-connection WAL replay. + leaseGuard.maybeRenew(); + if (event.type === "SESSION_START") { + session.recordToolsList(s, event.tools || []); + } else if (event.type === "INITIALIZE") { + session.recordInitialize(s, { clientInfo: event.clientInfo, serverInfo: event.serverInfo, model: event.model }); + } else if (event.type === "CALL_START") { + const key = Symbol.for(`wal-replay:${connectionId}:${event.call_seq}`); + keyToStartEvent.set(key, event); + session.recordCallStart(s, key, { tool: event.tool, server: event.server, arguments: event.arguments, isModelCall: Boolean(event.isModelCall), ts: event.ts }); + } else if (event.type === "CALL_RESULT") { + const key = Symbol.for(`wal-replay:${connectionId}:${event.call_seq}`); + /* Merge fix (PR #29 x PR #33 composition, 2026-09-20): event.malformedResult is + * PR #29's own new field on the CALL_RESULT WAL event (see proxy.js's own doc + * comment on this same fix) -- absent on a WAL line written before this fix (or by + * a non-tools/call CALL_RESULT such as gateway.js's own sampling-forward path, + * which never sets it at all), where `undefined` is exactly right: session.js's + * recordCallResult Boolean-coerces it to `false`, the correct value for a call this + * flag never applied to. */ + if (s.pendingCalls.has(key)) session.recordCallResult(s, key, { result: event.result, isError: event.isError, malformedResult: event.malformedResult, ts: event.ts }); + } else if (event.type === "ANOMALY") { + session.recordAnomaly(s, { kind: event.kind, tool: event.tool, intent_key: event.intent_key, detail: event.detail, ts: event.ts }); + } + } + if (s.pendingCalls.size > 0) { + /* Round-N fix (frontier-panel finding #5, Paul's explicit choice of Fable's "make the + * gap observable" proposal): recoverCrashedSessions' own per-call loop (above in this + * file) performs this identical startedFrom.replayed check, but never reaches + * session.finalizeSession/chain.appendSession for any connection it flags + * needsOperator -- that in-memory session, and any anomaly attached to it, is + * discarded entirely once pushed to pendingOperatorReview. recovery-abandon is the + * ONLY code path that unconditionally seals a connection like this one (see + * chain.appendSession below), so this is the one place a replay-specific INCOMPLETE + * marker can actually land in a produced bundle. Deliberately scoped to only the + * calls still pending after this replay -- a call that already resolved (proven or + * not) needs no such marker. */ + for (const [key] of Array.from(s.pendingCalls.entries())) { + const startedFrom = keyToStartEvent.get(key); + if (startedFrom && startedFrom.replayed) { + const intentKey = recovery.computeIntentKey(connectionId, startedFrom.tool, startedFrom.arguments); + session.recordAnomaly(s, { + kind: "GATEWAY_RECOVERY_INCOMPLETE_REPLAYED_STEP", + tool: startedFrom.tool, + intent_key: intentKey, + detail: + `INCOMPLETE: step ${startedFrom.call_seq} was a cached replay (see proxy.js's ` + + "replay branch) whose CALL_START was durably recorded but whose CALL_RESULT was " + + "never written before this connection crashed -- the agent may already have " + + "received a result for this step under this gateway's deliberate fail-open replay " + + "design, but that outcome was never confirmed durable in this WAL, and the operator " + + "ran recovery-abandon without further confirmation of what actually happened.", + }); + } + } + session.recordAnomaly(s, { kind: "OPERATOR_ABANDONED_RECOVERY", detail: `operator explicitly abandoned recovery for connection "${connectionId}" with ${s.pendingCalls.size} call(s) of unproven outcome` }); + session.markPendingAsDisconnected(s, "operator ran recovery-abandon: outcome could not be confirmed and was not waited on further", () => Date.now()); + } + const sealed = session.finalizeSession(s, keys); + // Round-1 fix-plan item 1: unconditional renew() immediately before chain.appendSession, + // same as recoverCrashedSessions' own normal path above. + leaseGuard.renew(); + try { + chain.appendSession(stateDir, sealed, { log }); + } catch (error) { + if (error.code !== "GATEWAY_BUNDLE_ID_COLLISION") throw error; + /* Codex PR #33 review "verify collisions in recovery-abandon before cleanup": this + * used to treat EVERY collision here as the expected repeated-attempt case and fall + * straight through to deleting this connection's WAL/intents below -- unlike + * recoverCrashedSessions' own already-hardened handling of the identical error code, + * which reads back the bundle on disk and verifies its real content first. An + * operator running recovery-abandon on a connection that happens to collide with a + * genuinely DIFFERENT session's bundle_id would silently discard this connection's + * only remaining unrecoverable record. Apply the same content verification here. */ + let verified; + try { + verified = verifyAndRepairBundleCollision(stateDir, sealed, log, leaseGuard); + } catch (repairError) { + // Round-1 fix-plan item 1: a lease loss detected by verifyAndRepairBundleCollision's + // own leaseGuard.renew() must not be relabeled as an ordinary chain-repair failure -- + // rethrow the original error unwrapped (by reference) rather than wrapping it in + // fail()'s own GATEWAY_RECOVERY_CHAIN_REPAIR_FAILED, so isLeaseError-style identity + // checks further up this same process's call stack still recognize it. + if (leaseGuard.isLeaseError(repairError)) throw repairError; + throw fail( + `recovery-abandon: connection "${connectionId}" bundle "${sealed.bundle.manifest.bundle_id}" was content-verified as this connection's own record, but completing its missing chain.jsonl append failed: ${repairError.message}. Refusing to abandon: this connection's WAL/intents are NOT cleaned up (the chain still does not durably reference this bundle) -- investigate and retry.`, + "GATEWAY_RECOVERY_CHAIN_REPAIR_FAILED" + ); + } + if (!verified) { + throw fail( + `recovery-abandon: connection "${connectionId}" produced bundle_id "${sealed.bundle.manifest.bundle_id}", which already exists on disk, but its recorded content does NOT match this connection's recovered session -- a genuine conflict with a different session, not a repeated abandon attempt. Refusing to abandon: deleting this connection's WAL/intents now would discard its only remaining, unrecoverable record. Investigate the existing bundle before retrying.`, + "GATEWAY_RECOVERY_UNVERIFIED_BUNDLE_COLLISION" + ); + } + log(`recovery-abandon: connection "${connectionId}" was already durably sealed (bundle_id collision, content-verified match) -- cleaning up.`); + } + /* Codex PR #33 review "keep the WAL until corrupt-intent cleanup succeeds": + * listIntentsForConnection reads EVERY intent record on disk before filtering, and + * readIntent throws GATEWAY_RECOVERY_INTENT_UNREADABLE/_CORRUPT on any damaged one -- + * including an unrelated one. Running it AFTER deleteWal meant recovery-abandon could + * irreversibly destroy this connection's last remaining record and then abort on that + * scan, leaving the corrupt fence in place with nothing left to quarantine or replay, + * and every retry failing at the same point. Enumerate first: a failing scan now leaves + * the WAL intact and the whole command safely re-runnable (the bundle is already sealed, + * so a re-run takes the content-verified bundle_id-collision path above). */ + const intentsToClear = recovery.listIntentsForConnection(stateDir, connectionId); + recovery.deleteWal(stateDir, connectionId); + for (const intent of intentsToClear) recovery.deleteIntent(stateDir, intent.intent_key); + log(`recovery-abandon: connection "${connectionId}" sealed (${s.calls.length} call(s)) and its idempotency fence(s) released.`); +} + +/** Minimal `--flag value` / `--boolean-flag` parser for the recovery-* CLI subcommands + * below. Not a general-purpose arg parser (no short flags, no `=` form) -- these two + * subcommands are the only CLI surface this build needs beyond `node gateway.js + * [configPath]` itself. */ +function parseFlags(argv) { + const flags = {}; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a.startsWith("--")) { + const key = a.slice(2); + const next = argv[i + 1]; + if (next !== undefined && !next.startsWith("--")) { + flags[key] = next; + i++; + } else { + flags[key] = true; + } + } + } + return flags; +} + +/** `node scripts/gateway/gateway.js recovery-resolve --connection --intent + * --confirmed executed|not-executed [--result-file ] [--config ]`: answers + * recoverCrashedSessions' own RECOVERY_AMBIGUOUS_INTENT log line. Does not itself seal the + * connection -- only updates the one intent record; restart the gateway (or run + * recovery-abandon) to actually finish sealing once every ambiguous intent on a connection + * has been resolved this way. */ +function runRecoveryResolveCli(argv, log) { + const flags = parseFlags(argv); + const configPath = flags.config || path.join(process.cwd(), "gateway-config.json"); + const config = gatewayConfig.loadConfig(configPath); + const connectionId = flags.connection; + const intentKey = flags.intent; + const confirmed = flags.confirmed; + if (!connectionId || !intentKey || !confirmed) { + throw fail("recovery-resolve requires --connection --intent --confirmed executed|not-executed", "GATEWAY_RECOVERY_CLI_USAGE"); + } + if (confirmed !== "executed" && confirmed !== "not-executed") { + throw fail(`--confirmed must be "executed" or "not-executed", got ${JSON.stringify(confirmed)}.`, "GATEWAY_RECOVERY_CLI_USAGE"); + } + /* Codex PR #33 review "acquire the writer claim before resolving intents": this + * mutates the same intent files the live proxy dispatch guard consults and updates + * (recovery.js's readIntent/updateIntent/deleteIntent), unlike recovery-abandon (which + * already takes the claim below). Without this, the documented operator workflow -- + * run this command, then restart the gateway -- could instead run WHILE a gateway + * still holds the claim: a live dispatch resolving the same intent to "completed"/ + * "ambiguous" and this command deleting/overwriting it concurrently can each observe + * the other's half-applied state (updateIntent throwing GATEWAY_RECOVERY_INTENT_NOT_FOUND + * on a competing delete, or a plain last-write-wins clobber of either resolution). */ + const writerClaim = new WriterClaim(config.state_dir, { hostId: config.host_id }); + writerClaim.acquire(); + try { + const existing = recovery.readIntent(config.state_dir, intentKey); + if (!existing) throw fail(`No such intent "${intentKey}" under this gateway's state_dir.`, "GATEWAY_RECOVERY_INTENT_NOT_FOUND"); + if (existing.connection_id !== connectionId) { + throw fail(`Intent "${intentKey}" belongs to connection "${existing.connection_id}", not "${connectionId}" -- refusing (likely a copy/paste mismatch).`, "GATEWAY_RECOVERY_CLI_MISMATCH"); + } + if (confirmed === "executed") { + /* CodeRabbit PR #33 review "require --result-file for --confirmed executed": a + * missing --result-file used to silently resolve with cached_result: null, which + * recoverCrashedSessions then replays as an ordinary CLEAN SUCCESS (isError: + * false) -- the sealed bundle would attest a confident, is_error:false null + * result for a call the gateway itself never actually observed completing; the + * operator confirmed only that the external effect occurred, not what it + * returned. Reject instead of guessing a shape for evidence that doesn't exist. */ + if (!flags["result-file"]) { + /* Codex PR #33 review "stop advising a false not-executed resolution": this + * message used to suggest "use --confirmed not-executed instead" as the fallback + * when the operator lacks a result file -- but an operator who chose --confirmed + * executed in the first place is telling this CLI they KNOW the side effect ran; + * following that suggestion would record the opposite of what they know happened, + * seal it as a failed/non-executed call, and release the fence so a retry repeats + * the side effect. The only truthful options when the exact result is unrecoverable + * are to leave the intent unresolved for now, or to explicitly give up the fence + * via recovery-abandon -- never to assert non-execution that isn't true. */ + throw fail( + "recovery-resolve --confirmed executed requires --result-file : a null result would be " + + "replayed as an unverified clean success in the sealed bundle. If you know this call executed " + + "but cannot recover its exact result, do NOT use --confirmed not-executed -- that records the " + + "opposite of what you know happened and lets a future retry repeat the side effect. Instead, " + + "either supply the real result via --result-file once you can recover it, leave this intent " + + "unresolved for now (this connection will keep being flagged for operator review on every " + + "restart, harmlessly), or use recovery-abandon to explicitly give up the fence on this connection.", + "GATEWAY_RECOVERY_CLI_USAGE" + ); + } + const result = JSON.parse(fs.readFileSync(flags["result-file"], "utf8")); + recovery.resolveIntentExecuted(config.state_dir, intentKey, result); + log(`recovery-resolve: intent "${intentKey}" (connection "${connectionId}") marked EXECUTED. Restart the gateway to finish sealing this connection.`); + } else { + recovery.resolveIntentNotExecuted(config.state_dir, intentKey); + log(`recovery-resolve: intent "${intentKey}" (connection "${connectionId}") marked NOT EXECUTED; fence released. Restart the gateway to finish sealing this connection.`); + } + } finally { + writerClaim.release(); + } +} + +/** `node scripts/gateway/gateway.js recovery-abandon --connection [--config ]`: + * the "give up on the fence, seal what we have" override -- see abandonConnection's own + * doc comment above for exactly what this does and does not guarantee. Requires the + * writer-claim (this mutates the same trusted chain a live gateway instance would), so it + * cannot run concurrently with an actual gateway process against the same state_dir. */ +function runRecoveryAbandonCli(argv, log) { + const flags = parseFlags(argv); + const configPath = flags.config || path.join(process.cwd(), "gateway-config.json"); + const config = gatewayConfig.loadConfig(configPath); + const connectionId = flags.connection; + if (!connectionId) throw fail("recovery-abandon requires --connection ", "GATEWAY_RECOVERY_CLI_USAGE"); + const keys = loadSigningKeys(config); + const writerClaim = new WriterClaim(config.state_dir, { hostId: config.host_id }); + writerClaim.acquire(); + try { + abandonConnection(config.state_dir, keys, connectionId, log, writerClaim); + } finally { + writerClaim.release(); + } +} + /* Cluster D: operational exposure for buildHealthStatus(ctx) above. This is a * single-tenant, locally-run process (not a hosted service with a real ops network * surface) -- SG-NFR-3's health/status report existed only as `.status()` on the @@ -494,18 +1586,133 @@ function gatewayStatusPath(stateDir) { * this is purely an observability side channel, not part of SG-FR-5's persisted-session * write path. Reuses state-store.js's own atomic-write primitive (temp file + fsync + * rename), same as chain.js#appendSession's HEAD.json write, so a reader (the `status` - * subcommand, or an operator's own tool) can never observe a half-written file. */ + * subcommand, or an operator's own tool) can never observe a half-written file. + * + * `{ mode: null }`: the ONE named exception to the group-readable access model (Paul's + * decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model"). Every other + * caller of atomicOverwriteFile gets that primitive's new 0640-by-default mode; this + * file is the operator-facing health surface commit 6 (SG-NFR-3) depends on being + * readable by whatever ops tooling polls it, so it stays at its current, pre-existing, + * umask-governed mode -- not narrowed, and not pinned to a guessed literal that might + * not match every deployment's own umask either. startup-permissions.js's own startup + * chmod pass likewise never lists this file among the ones it tightens. */ function writeStatusFile(ctx, log) { + const stateDir = ctx.config.state_dir; try { - const stateDir = ctx.config.state_dir; fs.mkdirSync(stateDir, { recursive: true }); - const status = { ...buildHealthStatus(ctx), written_at: new Date().toISOString() }; - stateStore.atomicOverwriteFile(gatewayStatusPath(stateDir), JSON.stringify(status, null, 2), stateDir); + } catch (error) { + log(`failed to write status file (non-fatal): ${error.message}`); + return; + } + + /* Round-1 fix-plan item 1, round-2 fix pass (Paul's 2026-09-16 decision): unlike the + * writes below, buildHealthStatus(ctx) is deliberately called OUTSIDE any try/catch + * here. After this round's fix, its only possible throw is a lease loss DETECTED by + * checks/register-gateway-sessions.js's own maybeRenew() call during the session-chain + * walk (see buildHealthStatus's own doc comment on that call site) -- every one of its + * OTHER internal steps already absorbs its own failures locally and never lets an + * exception escape. That one throw must NOT be caught here and downgraded to this + * function's own "non-fatal" logging below -- createLeaseGuard's own doc comment + * (scripts/gateway/gateway.js, above) is explicit that a detected lease loss must + * never be relabeled as "flag for operator review, continue", and this status-file + * write is no exception to that just because its OTHER failure modes (a momentarily + * full disk, an unwritable state_dir) genuinely are non-fatal by design. Left + * unguarded, this throw surfaces exactly like the fatal-on-renew-failure policy + * already established for reconcileHead/the WAL-replay loops: on the very first call + * (synchronous, at startup, before this function returns to startGateway) it rejects + * startGateway's own promise, reaching main()'s top-level .catch (process.exitCode = + * 1); on a later setInterval tick it is an uncaught exception, halting the process + * loudly -- fatal to the whole process either way, never merely logged. */ + const status = { + ...buildHealthStatus(ctx), + written_at: new Date().toISOString(), + /* Round-1 fix-plan commit 6: documented explicitly, per this commit's own + * requirement, both here (the WRITTEN status output an operator actually reads) + * and in checks/register-gateway-sessions.js's own header comment -- this walk + * reports `session_chain_integrity` as EVIDENCE ONLY (docs/contracts/ + * chain-validity.md SS5: "the status walk only reports evidence and never writes + * anything") and never engages getChainIntegrityFailure()'s admission latch, + * regardless of what it finds. That means the ~10s STATUS_WRITE_INTERVAL_MS + * cadence is NOT what stops new-session admission on a genuine finding -- only + * commit 4's own detectors (the O(1) append-time checkHeadAgainstTailOrRepair, and + * reconcileHead's full walk at startup) ever latch. A HEAD/tail POINTER divergence + * mid-run is caught immediately (not on any cadence) by that append-time check the + * next time this gateway tries to append. Mid-chain INTERIOR corruption -- an + * entry whose own hash no longer recomputes, a sequence gap, or a broken link, + * where HEAD and the chain's own tail already agree -- is structurally invisible + * to that O(1) tail-only check; this walk's own evidence field is currently the + * only place such a finding surfaces at all, and it does not stop anything by + * itself. Detecting AND acting on that class depends on reconcileHead's full walk + * running again at the next startup (which hard-fails per Paul's 2026-09-15 + * decision), or on an explicit, separately-scheduled deep walk that does not exist + * in this build -- see checks/register-gateway-sessions.js's own header for why it + * is deliberately not wired to write/latch anything itself. */ + session_chain_integrity_cadence_note: + `session_chain_integrity above is refreshed roughly every ${STATUS_WRITE_INTERVAL_MS}ms and is EVIDENCE ONLY -- ` + + "it never stops new-session admission by itself. A HEAD/tail POINTER divergence is instead caught " + + "immediately by the append-time check (or by reconcileHead at startup), not by this cadence. Mid-chain " + + "INTERIOR corruption (HEAD and the chain's own tail already agreeing, but an interior record tampered, a " + + "sequence gap, or a broken link) is invisible to that O(1) check; this field may show it as evidence, but " + + "detecting AND acting on it (stopping admission) depends on the full validator running again at the next " + + "startup, or on a separately-scheduled deep walk this build does not implement.", + }; + + try { + stateStore.atomicOverwriteFile(gatewayStatusPath(stateDir), JSON.stringify(status, null, 2), stateDir, { mode: null }); } catch (error) { log(`failed to write status file (non-fatal): ${error.message}`); } } +/* Round-1 fix-plan commit 4: filename for the startup-only diagnostic artifact written + * when chain.reconcileHead finds a genuine structural failure (see startGateway's own + * call site below and Paul's 2026-09-15 startup-posture decision, docs/contracts/ + * chain-validity.md SS4). Distinct from STATUS_FILE_NAME above: that one is refreshed + * every STATUS_WRITE_INTERVAL_MS by a RUNNING gateway; this one is written exactly once, + * by a gateway that is refusing to start at all, and is never cleaned up automatically + * (an operator's own recovery action, per the referenced runbook, is what resolves it -- + * a future successful startup does not delete it, so its presence and mtime remain a + * durable record of the last startup failure even after the underlying corruption is + * fixed and the gateway starts normally again). */ +const CHAIN_CORRUPTION_DIAGNOSTIC_FILE_NAME = "gateway-startup-failure.json"; + +function chainCorruptionDiagnosticPath(stateDir) { + return path.join(stateDir, CHAIN_CORRUPTION_DIAGNOSTIC_FILE_NAME); +} + +/** Round-1 fix-plan commit 4, Paul's 2026-09-15 startup-posture decision (docs/ + * contracts/chain-validity.md SS4): when chain.reconcileHead (or, in principle, any + * other startup structural-failure verdict) forces a hard-fail, this must be LOUD, not + * a silent status field -- a structured top-level diagnostic (logged so it appears even + * if the artifact write below itself fails), a written failure artifact (a status file + * does not require a running gateway), and an explicit pointer to the manual recovery + * runbook this commit ships alongside it, so the diagnostic always ends in a concrete + * next step. Writing the artifact is best-effort (mirrors writeStatusFile's own + * contract just above) -- a failure to write it must never mask or replace the + * already-logged, already-propagating original startup failure. */ +function writeChainCorruptionDiagnostic(stateDir, error, log) { + const diagnostic = { + schema_version: "1.0", + event: "gateway_startup_chain_integrity_failure", + code: error && error.code ? error.code : "GATEWAY_CHAIN_STRUCTURAL_FAILURE", + message: error && error.message ? error.message : String(error), + at: new Date().toISOString(), + runbook: chain.CHAIN_CORRUPTION_RUNBOOK, + action_required: + "This gateway will not start while its gateway-session chain fails structural verification -- " + + `no new sessions are being accepted. Follow ${chain.CHAIN_CORRUPTION_RUNBOOK} (restore from backup ` + + "/ truncate-at-last-good / rebuild HEAD from the validated chain tail) before restarting.", + }; + log(`FATAL STARTUP CHAIN INTEGRITY FAILURE: ${JSON.stringify(diagnostic)}`); + try { + fs.mkdirSync(stateDir, { recursive: true }); + stateStore.atomicOverwriteFile(chainCorruptionDiagnosticPath(stateDir), JSON.stringify(diagnostic, null, 2), stateDir); + } catch (writeError) { + log(`failed to write the startup chain-integrity diagnostic artifact (non-fatal, already logged above): ${writeError.message}`); + } +} + /** * Starts the standalone gateway process. Returns { dormant: true } if attach mode is * active (caller should exit 0). Otherwise returns a running gateway handle with @@ -552,6 +1759,27 @@ async function startGateway(options) { writerClaim.acquire(); // FR-1: throws and this process must exit non-zero on refusal log(`writer-claim acquired: instance ${writerClaim.instanceId} on host ${writerClaim.hostId}`); + /* Access model (Paul's decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model"): re-chmod + * every known-sensitive file/directory already on disk (from a prior run, or an older + * build) to this codebase's group-readable model (0750 dirs / 0640 files) BEFORE + * recovery reads any of them and before this process accepts any new session. Runs + * once we hold the writer-claim (so the state directory and the claim file itself are + * guaranteed to exist) but before recoverCrashedSessions. A real permission fault here + * (see startup-permissions.js's own doc comment on what is and is not swallowed) must + * stop startup outright rather than be discovered later against a file that turned out + * to still be more permissive than this model requires -- released the same way, and + * for the same reason, as the CodeRabbit PR #33 fix just below release the claim on a + * failed recoverCrashedSessions: without this, the claim and its heartbeat are already + * live, and an uncaught throw here would reject startGateway's promise while leaving + * an orphaned claim blocking the next restart. */ + try { + tightenStateDirPermissions(config.state_dir); + } catch (error) { + writerClaim.release(); + throw error; + } + let proxy; writerClaim.onClaimLost = (error) => { log(`writer-claim lost: ${error.message} -- halting: no new sessions will be accepted.`); @@ -569,6 +1797,74 @@ async function startGateway(options) { }; writerClaim.startHeartbeat(); + /* Round-1 fix-plan commit 4 (docs/contracts/chain-validity.md's C2 + SS4): fixes a + * stale HEAD pointer ONCE, under the writer-claim and after the heartbeat has + * started (commit 1's own generalized keepalive covers reconcileHead's own O(n) walk + * via its per-entry maybeRenew()) -- BEFORE recoverCrashedSessions or any chain + * append can ever read HEAD stale. Per Paul's 2026-09-15 startup-posture decision: a + * genuine structural-failure verdict here must hard-fail startup, loudly (never a + * silent status field) -- log a structured top-level diagnostic, write a failure + * artifact naming the recovery runbook, release the writer-claim, and rethrow so + * this process exits non-zero (main()'s own top-level .catch already turns any + * rejected startGateway() into `process.exitCode = 1` with a FATAL log line). */ + try { + const reconciliation = chain.reconcileHead(config.state_dir, writerClaim); + if (reconciliation.action === "advanced") { + log( + JSON.stringify({ + event: "gateway_startup_chain_head_reconciled", + detail: + `startup chain reconciliation advanced a lagging HEAD.json to the chain's own verified ` + + `tail (seq=${reconciliation.to.seq}, bundle_id=${reconciliation.to.bundle_id}) -- this is ` + + "the expected signature of a crash between chain.jsonl's append and HEAD.json's update; " + + "no data was lost.", + }) + ); + } + } catch (error) { + writeChainCorruptionDiagnostic(config.state_dir, error, log); + writerClaim.release(); + throw error; + } + + /* Option C (crash-recovery/idempotency hardening, external-panel-reviewed design -- + * see option-c-hardened-design.md): replay any WAL left behind by a crashed prior + * instance now, while this process holds exclusive writer ownership of state_dir and + * BEFORE any new downstream/agent work starts. Recovery must not race a fresh + * session's own WAL writes, and no new session should be admitted while a prior + * session's calls are still ambiguous. */ + /* CodeRabbit PR #33 review "release the writer claim when startup recovery fails": + * the writer-claim and its heartbeat are already live by this point. Without this, + * an exception from recoverCrashedSessions (an unreadable WAL/intent this function's + * own per-connection try/catch does not fully absorb, or any other unexpected + * failure) would reject startGateway's promise while leaving the claim held -- an + * immediate restart would then be refused until the lease goes stale, and any other + * live handle to this WriterClaim instance would keep renewing an orphaned claim + * indefinitely (writerClaim.release() already stops the heartbeat itself). */ + /* Round-N fix (frontier-panel finding #7): this log() call was previously OUTSIDE the + * try/catch above, even though it runs immediately after recoverCrashedSessions and + * before anything else -- a throwing log() (the same failure mode CodeRabbit's own + * review comment right above this block was written to guard against, just one + * statement too early) would escape startGateway with the writer-claim still held and + * its heartbeat still running, exactly the orphaned-claim outcome this whole guard + * exists to prevent. Moved inside the try so any exception here also releases the + * claim before rethrowing, consistent with every other statement in this guarded + * region. */ + let pendingOperatorReview; + try { + ({ pendingOperatorReview } = recoverCrashedSessions(config.state_dir, keys, log, writerClaim)); + if (pendingOperatorReview.length > 0) { + log( + `startup recovery: ${pendingOperatorReview.length} connection(s) left pending operator ` + + `review (see RECOVERY_AMBIGUOUS_INTENT log lines above) -- resolve via the ` + + "recovery-resolve/recovery-abandon CLI before their state is cleaned up." + ); + } + } catch (error) { + writerClaim.release(); + throw error; + } + /* Mutable box, not a plain variable: connectAllDownstreams() below runs (and each * downstream stdio connection's onRequest closure over it is created) BEFORE the * agent transport is started further down, so at closure-creation time there is @@ -656,6 +1952,10 @@ async function startGateway(options) { keys, stateDir: config.state_dir, log, + // Frontier-panel decision (Paul, 2026-09-10, cluster E2): see proxy.js's own + // pendingOperatorReviewConnections constructor comment for why a startup snapshot + // is sufficient here. + pendingOperatorReviewConnections: pendingOperatorReview, /* Codex PR #29 review "treat sampling as a negotiated client capability" (comment * 4000335132): sampling/createMessage is a distinct MCP capability negotiated at * initialize, not a tools/call -- agentTransportSupportsSampling (computed above @@ -674,6 +1974,13 @@ async function startGateway(options) { // caught here even if this instance's own heartbeat hasn't yet noticed and fired // onClaimLost. isWriterClaimValid: () => writerClaim.status().held_by_this_instance, + /* Round-1 fix-plan commit 6: wires this process's own chain-integrity latch + * (chain.js's process-local getChainIntegrityFailure(), set by commit 4's + * checkHeadAgainstTailOrRepair/reconcileHead) into the ONE place a genuine + * failure must actually stop new work -- see GatewayProxy#openConnection's own + * doc comment for why this is the primary enforcement point and how a runtime + * transition into failure gets the same effect as detecting it at startup. */ + getChainIntegrityFailure: () => chain.getChainIntegrityFailure(), // Re-triage fix: renew the writer-claim synchronously immediately before // chain.appendSession in closeConnection (see GatewayProxy#renewWriterClaim's own // doc comment) -- closes the gap where a long synchronous append blocks the @@ -694,10 +2001,10 @@ async function startGateway(options) { try { if (listenConfig.transport === "http") { const token = gatewayConfig.resolveSecretRef(listenConfig.token_ref, "agent_listen.token_ref"); - httpHandle = await runHttpAgentTransport({ proxy }, listenConfig, token); + httpHandle = await runHttpAgentTransport({ proxy, log }, listenConfig, token); log(`agent-facing HTTP listener on port ${httpHandle.port}`); } else { - stdioHandle = runStdioAgentTransport({ proxy }); + stdioHandle = runStdioAgentTransport({ proxy, log }); // See forwardDownstreamRequestToAgent's doc above: only the stdio transport can // push a request to the agent, so this is the one branch that ever populates it. agentPusher.current = stdioHandle.pushRequest; @@ -764,51 +2071,88 @@ async function startGateway(options) { return stopPromise; } async function doStop(reason) { - log(`shutting down (${reason || "requested"}): draining ${proxy.openSessionCount()} open session(s)`); - clearInterval(statusWriteTimer); // Cluster D: stop refreshing the status file once shutdown begins - proxy.stopAcceptingNewSessions(); - /* SS3.7: "finish in-flight sessions" means actually WAIT (bounded) for calls already - * in flight to complete and be recorded with their real result -- not immediately - * truncate the connection and let closeConnection's own pending-call handling mark - * a call that was about to succeed as an artificial "disconnected" error. Only a - * call that is STILL pending once the drain timeout elapses gets that treatment. */ - await drainOpenSessions(proxy, drainTimeoutMs); - agentPusher.current = null; // no agent left to push a forwarded sampling request to - agentPusher.connectionId = null; - if (stdioHandle) stdioHandle.stop(); - if (httpHandle) { - await new Promise((resolve) => { - let settled = false; - const finish = () => { if (!settled) { settled = true; resolve(); } }; - httpHandle.server.close(finish); - const forceCloseTimer = setTimeout(() => { - // Stop accepting new connections is already implied by close() above; this - // forcibly ends any still-open sockets/responses so the callback above (or - // this fallback) fires within the bounded deadline rather than whenever the - // last active response happens to finish. - if (typeof httpHandle.server.closeAllConnections === "function") httpHandle.server.closeAllConnections(); - finish(); - }, HTTP_LISTENER_CLOSE_TIMEOUT_MS); - if (typeof forceCloseTimer.unref === "function") forceCloseTimer.unref(); - }); - } - // Finalize any still-open sessions (drained above, or forced closed after the timeout). - for (const connectionId of Array.from(proxy.sessions.keys())) { - await proxy.closeConnection(connectionId, `gateway shutdown (${reason || "requested"})`); - } - /* Codex PR #29 review round 8 "bound termination of stdio downstream children": same - * reasoning as the startup-failure cleanup above -- await close() so a downstream that - * traps/ignores SIGTERM is actually confirmed gone (or forcibly SIGKILLed) before this - * releases the writer-claim, rather than left running past shutdown. */ - for (const conn of downstreamHandles.connections.values()) { - try { await conn.close(); } catch (error) { /* best effort */ } + /* Codex PR #33 review "prevent shutdown diagnostics from aborting claim release": + * `log` is CALLER-SUPPLIED plumbing (options.log) and is permitted to throw. An + * earlier round of this PR wrapped each connection's closeConnection in a try/catch so + * one failing session could not abort shutdown -- but the handler (and this function's + * other diagnostics, starting with the very first line below) still called `log` + * directly, so a throwing logger reintroduced the identical failure one level up: + * doStop rejects before the remaining sessions, the downstream closes, and + * writerClaim.release() ever run, and an embedding process keeps renewing an orphaned + * claim indefinitely. Every diagnostic on this shutdown path now goes through this + * non-throwing wrapper (same contract as proxy.js#safeLog), AND the release itself sits + * in a finally covering the whole body -- so neither a close failure nor a logging + * failure can strand the claim. */ + const safeShutdownLog = (message) => { + try { + log(message); + } catch (loggerError) { + // Deliberately swallowed: a failing logger must never keep shutdown from + // releasing the writer claim. See this block's own doc comment above. + } + }; + try { + safeShutdownLog(`shutting down (${reason || "requested"}): draining ${proxy.openSessionCount()} open session(s)`); + clearInterval(statusWriteTimer); // Cluster D: stop refreshing the status file once shutdown begins + proxy.stopAcceptingNewSessions(); + /* SS3.7: "finish in-flight sessions" means actually WAIT (bounded) for calls already + * in flight to complete and be recorded with their real result -- not immediately + * truncate the connection and let closeConnection's own pending-call handling mark + * a call that was about to succeed as an artificial "disconnected" error. Only a + * call that is STILL pending once the drain timeout elapses gets that treatment. */ + await drainOpenSessions(proxy, drainTimeoutMs); + agentPusher.current = null; // no agent left to push a forwarded sampling request to + agentPusher.connectionId = null; + if (stdioHandle) stdioHandle.stop(); + if (httpHandle) { + await new Promise((resolve) => { + let settled = false; + const finish = () => { if (!settled) { settled = true; resolve(); } }; + httpHandle.server.close(finish); + const forceCloseTimer = setTimeout(() => { + // Stop accepting new connections is already implied by close() above; this + // forcibly ends any still-open sockets/responses so the callback above (or + // this fallback) fires within the bounded deadline rather than whenever the + // last active response happens to finish. + if (typeof httpHandle.server.closeAllConnections === "function") httpHandle.server.closeAllConnections(); + finish(); + }, HTTP_LISTENER_CLOSE_TIMEOUT_MS); + if (typeof forceCloseTimer.unref === "function") forceCloseTimer.unref(); + }); + } + // Finalize any still-open sessions (drained above, or forced closed after the timeout). + /* Codex PR #33 review "release the writer claim when intent cleanup aborts shutdown": + * closeConnection can throw before reaching its own guarded sealing block (e.g. + * recovery.listIntentsForConnection hitting a corrupt/unreadable intent FILE for this + * connection -- a genuine fs-level problem, not the kind of failure this codebase + * treats as recoverable). Previously unguarded here, so one such connection's failure + * escaped this loop entirely, skipping every remaining connection's own close, every + * downstream conn.close() below, and writerClaim.release() -- leaking a stale claim + * that blocks the next restart. Isolate per-connection failures (mirrors + * recoverCrashedSessions' own per-connection try/catch) so shutdown always reaches + * the downstream-close and writer-claim-release steps regardless. */ + for (const connectionId of Array.from(proxy.sessions.keys())) { + try { + await proxy.closeConnection(connectionId, `gateway shutdown (${reason || "requested"})`); + } catch (error) { + safeShutdownLog(`SHUTDOWN CLOSE FAILURE for connection "${connectionId}": ${error.message} (${error.code || "no code"}) -- continuing shutdown for other sessions and releasing the writer claim regardless.`); + } + } + /* Codex PR #29 review round 8 "bound termination of stdio downstream children": same + * reasoning as the startup-failure cleanup above -- await close() so a downstream that + * traps/ignores SIGTERM is actually confirmed gone (or forcibly SIGKILLed) before this + * releases the writer-claim, rather than left running past shutdown. */ + for (const conn of downstreamHandles.connections.values()) { + try { await conn.close(); } catch (error) { /* best effort */ } + } + } finally { + writerClaim.release(); } - writerClaim.release(); // Cluster D: one last write so `gateway.js status` run after this process has // exited reports an accurate "not claimed by this instance" / drained snapshot // instead of silently going stale mid-run-looking data. - writeStatusFile(ctx, log); - log("shutdown complete: writer-claim released."); + writeStatusFile(ctx, safeShutdownLog); + safeShutdownLog("shutdown complete: writer-claim released."); } return { @@ -888,7 +2232,18 @@ function runStatusCommand(configPathArg) { } function main() { + const log = (...args) => console.error("[graphsmith-gateway]", ...args); const argv = process.argv.slice(2); + if (argv[0] === "recovery-resolve" || argv[0] === "recovery-abandon") { + try { + if (argv[0] === "recovery-resolve") runRecoveryResolveCli(argv.slice(1), log); + else runRecoveryAbandonCli(argv.slice(1), log); + } catch (error) { + console.error(`[graphsmith-gateway] FATAL: ${error.message}`); + process.exitCode = 1; + } + return; + } if (argv[0] === "status") { runStatusCommand(argv[1]); return; @@ -914,8 +2269,14 @@ module.exports = { startGateway, checkModeGate, buildHealthStatus, + writeStatusFile, loadSigningKeys, drainOpenSessions, + createLeaseGuard, + recoverCrashedSessions, + abandonConnection, + runRecoveryResolveCli, + runRecoveryAbandonCli, gatewayStatusPath, STATUS_WRITE_INTERVAL_MS, runStatusCommand, diff --git a/scripts/gateway/proxy.js b/scripts/gateway/proxy.js index ec25f22d..83670b1c 100644 --- a/scripts/gateway/proxy.js +++ b/scripts/gateway/proxy.js @@ -15,8 +15,11 @@ const fs = require("fs"); const path = require("path"); +const crypto = require("crypto"); const session = require("./session.js"); const chain = require("./chain.js"); +const recovery = require("./recovery.js"); +const stateStore = require("../state-store.js"); function fail(message, code = "GATEWAY_PROXY_ERROR") { const error = new Error(message); @@ -36,12 +39,68 @@ function fail(message, code = "GATEWAY_PROXY_ERROR") { * was about to be lost forever: a best-effort, non-throwing write of the sealed bundle to * a quarantine directory an operator can inspect and manually re-append once the * underlying failure (e.g. disk full) is resolved. */ +/* Round-N fix (frontier-panel finding #10): connectionId here can originate from the + * remote agent side of the transport (it is not generated by this gateway), so building + * a filesystem path directly from it is unsafe the same way recovery.js's own header + * (see its SAFE_ID/assertSafeId) already documents for its own path-construction points + * -- an unvalidated "../../etc/whatever" (or similar) connectionId could otherwise address + * a file outside this quarantine directory. Kept local to this file rather than importing + * recovery.js's private assertSafeId (not exported, and this call site's failure mode is + * different: quarantining is already best-effort, so an unsafe id degrades to a fixed + * placeholder filename rather than aborting the whole quarantine attempt). The raw, + * unsanitized connectionId is still recorded in the JSON body below for operator + * traceability even when the filename itself had to be substituted. */ +const QUARANTINE_SAFE_ID = /^[A-Za-z0-9._-]+$/; +function quarantineFilenameId(connectionId) { + if (typeof connectionId === "string" && connectionId !== "." && connectionId !== ".." && QUARANTINE_SAFE_ID.test(connectionId)) { + return connectionId; + } + return "unsafe-connection-id"; +} + function quarantineSealedBundle(stateDir, connectionId, sealed, cause) { try { const dir = path.join(chain.sessionsDir(stateDir), "quarantine"); - fs.mkdirSync(dir, { recursive: true }); - const target = path.join(dir, `${connectionId}-${Date.now()}.json`); - fs.writeFileSync(target, JSON.stringify({ sealed, quarantined_at: new Date().toISOString(), reason: cause && cause.message }, null, 2), { encoding: "utf8", flag: "wx" }); + /* Codex PR #33 review "enforce modes when creating post-startup session files": this + * directory and file are created at the moment an append FAILS -- long after + * startup-permissions.js's one-time pass skipped them for not existing yet -- and the + * file is the full raw sealed bundle (the most sensitive artifact this gateway + * writes). Without explicit modes it lands 0755/0644 (world-readable) under a 022 + * umask and stays that way until the next restart re-runs that pass. Use the same + * DEFAULT_DIR_MODE/DEFAULT_FILE_MODE constants every other state write here uses. */ + fs.mkdirSync(dir, { recursive: true, mode: stateStore.DEFAULT_DIR_MODE }); + /* Round-N fix (frontier-panel finding #6): mkdirSync's own `mode` option is IGNORED + * for a directory that already existed before this call (recursive:true does not + * revisit or retighten an existing directory's mode) -- see chain.js#appendSession's + * identical fix and doc comment for the same underlying Node behavior. Best effort: + * a failure here (e.g. this process does not own the directory) must not block the + * quarantine write itself. */ + try { + fs.chmodSync(dir, stateStore.DEFAULT_DIR_MODE); + } catch (error) { + /* best effort -- see doc comment above */ + } + /* Round-N fix (frontier-panel finding #10): Date.now() gives millisecond resolution, + * so two quarantine writes for the same connectionId within the same millisecond + * (a realistic case: a poisoned connection can be closed and immediately reopened/ + * re-quarantined by a retrying agent) collide on the same filename -- flag:"wx" then + * makes the SECOND write throw and silently discard that sealed bundle via this + * function's own best-effort catch below, exactly the kind of silent data loss + * quarantining exists to prevent. crypto.randomBytes is cheap and collision-safe + * enough here (this is a diagnostic artifact, not a security token). */ + const target = path.join(dir, `${quarantineFilenameId(connectionId)}-${crypto.randomBytes(8).toString("hex")}.json`); + /* Round-N fix (frontier-panel finding #10): the raw writeFileSync/flag:"wx" here never + * fsynced the file or its parent directory, so a quarantine write that appeared to + * succeed could still be lost on a crash shortly after -- for a directory (`dir`) + * already guaranteed to exist by the mkdirSync above, stateStore.atomicOverwriteFile + * is the same already-reviewed temp-file+fsync+rename+directory-fsync primitive every + * other durable state write in this codebase uses (see state-store.js's own header); + * reused here rather than hand-rolling new fsync logic. connection_id is included in + * the body (not just derived from the filename) so an operator can recover the real, + * original connectionId even when quarantineFilenameId above had to substitute a + * placeholder for an unsafe one. */ + const payload = JSON.stringify({ sealed, connection_id: connectionId, quarantined_at: new Date().toISOString(), reason: cause && cause.message }, null, 2); + stateStore.atomicOverwriteFile(target, payload, dir, { mode: stateStore.DEFAULT_FILE_MODE }); return target; } catch (quarantineError) { // Best effort only: a quarantine-write failure (e.g. the same ENOSPC that caused the @@ -59,6 +118,53 @@ function isModelCallMethod(method) { return method === "sampling/createMessage"; } +/* Round-2 fix pass (2026-09-16), closing commit 5's WAL-poison enforcement gap: commit 5 + * (b14bf35c) made gateway.js#forwardDownstreamRequestToAgent (the downstream-initiated + * sampling-forward path) mark `s.walPoisoned` on its own first appendWalEvent failure and + * refuse every later append on that connection -- see gateway.js#poisonWalOnFailure and + * its call site's own doc comments. session.js#createSession's own walPoisoned doc + * comment states that contract as connection-wide ("every later append attempt on the + * same connection must be refused"), but until now THIS file's own appendWalEvent call + * sites (SESSION_START/INITIALIZE/ANOMALY/CALL_START/CALL_RESULT/CLOSING) never set or + * checked it at all -- an agent-initiated call could still append to (and poison, or + * concatenate onto an already-torn) the very same connection's WAL that the sampling path + * had already given up on, and CLOSING's own append ran completely unguarded on every + * connection close, including one already poisoned by an earlier failure on this file's + * own dispatch path. These two helpers give every appendWalEvent call site below in THIS + * file the identical set/check contract gateway.js's sampling path already has, per + * docs/contracts/wal-append-failure-semantics.md (C1 SS2-3). Defined locally rather than + * imported from gateway.js to avoid a proxy.js/gateway.js circular require (gateway.js + * already requires this file) -- kept identical in shape/semantics to + * gateway.js#poisonWalOnFailure by design, not a new mechanism. */ +function poisonWalOnFailure(s, walError, now) { + if (s && !s.walPoisoned) { + s.walPoisoned = { reason: walError.message, at: typeof now === "function" ? now() : Date.now() }; + } +} + +/* Companion to poisonWalOnFailure above: the shared refusal every gated call site below + * returns once `s.walPoisoned` is already set, matching gateway.js# + * forwardDownstreamRequestToAgent's own wording/JSON-RPC shape for the identical + * situation on the sampling-forward path (commit 5) verbatim, so both paths present one + * consistent contract to whatever is on the other end of a connection. `verb` names the + * specific durable write being refused (e.g. "initializing this connection") so the + * message stays accurate per call site while everything else -- including the contract + * cited -- stays identical everywhere it is used. */ +function walPoisonedError(id, s, verb) { + return { + jsonrpc: "2.0", + id, + error: { + code: -32000, + message: + `This connection's WAL was poisoned by an earlier append failure (${s.walPoisoned.reason}) ` + + "and is refusing every further durable write on this connection rather than risk " + + "concatenating onto a possibly-torn line -- per docs/contracts/wal-append-failure-" + + `semantics.md (C1). Not ${verb}; close and reopen the connection to get a fresh WAL.`, + }, + }; +} + /* PR #29 review thread 4000335147, frontier-panel review (4/4 independent models * converged, two rounds of critique) "flag structurally malformed tools/call results * distinctly from tool-level errors": before this, a `tools/call` result was only ever @@ -150,6 +256,8 @@ class GatewayProxy { * @param {(session: object, error: Error) => void} [opts.onSealFailure] SS7: sealBoundaryBundle throws * @param {string} opts.stateDir passed straight to chain.appendSession (SG-FR-5) * @param {() => number} [opts.now] + * @param {string[]} [opts.pendingOperatorReviewConnections] connectionIds startup + * recovery flagged for operator review (E2 tool-level quarantine, see constructor body) * @param {() => boolean} [opts.isWriterClaimValid] Cluster C narrow fix: a synchronous, * read-fresh-from-disk liveness check (e.g. `() => writerClaim.status().held_by_this_instance`) * consulted immediately before chain.appendSession in closeConnection -- see that method's @@ -157,6 +265,23 @@ class GatewayProxy { * the moment onClaimLost first fires. Defaults to always-valid so existing callers/tests that * construct a GatewayProxy directly (no writer-claim of their own) are unaffected, mirroring * onSessionFinalized/onSealFailure's own default-no-op contract above. + * @param {() => ({reason: string, at: string}|null)} [opts.getChainIntegrityFailure] + * Round-1 fix-plan commit 6: the PRIMARY enforcement point for a genuine chain- + * integrity failure (chain.js's own process-local latch, set by commit 4's + * checkHeadAgainstTailOrRepair/reconcileHead -- see chain.js#getChainIntegrityFailure's + * own doc comment). Consulted at the top of openConnection, before the + * acceptingNewSessions drain/writer-claim-lost gate below, so the error a caller sees + * names the failure that actually happened rather than a generic "not accepting + * sessions" message. Deliberately a DETAIL-OBJECT accessor, never a bare + * `(): boolean` -- a boolean cannot carry the diagnostic reason a human needs to act + * on (this is the exact naming trap this commit's own contract calls out). Checking + * this here, not only at startup, is what makes a runtime transition into failure + * have the SAME effect as detecting it at startup: the moment ANY append-time check + * anywhere in this process latches a genuine failure, the very next openConnection + * call refuses, with no separate "stop accepting sessions" step required. Defaults + * to `() => null` (never latched) so existing callers/tests that construct a + * GatewayProxy directly (no chain-integrity concern of their own) are unaffected, + * mirroring isWriterClaimValid's own always-valid default just above. * @param {() => void} [opts.renewWriterClaim] Re-triage fix: a synchronous * `() => writerClaim.renew()` consulted immediately before chain.appendSession in * closeConnection, alongside isWriterClaimValid above -- writer-claim.js's own @@ -180,6 +305,7 @@ class GatewayProxy { this.stateDir = opts.stateDir; this.now = opts.now || (() => Date.now()); this.isWriterClaimValid = typeof opts.isWriterClaimValid === "function" ? opts.isWriterClaimValid : () => true; + this.getChainIntegrityFailure = typeof opts.getChainIntegrityFailure === "function" ? opts.getChainIntegrityFailure : () => null; this.renewWriterClaim = typeof opts.renewWriterClaim === "function" ? opts.renewWriterClaim : () => {}; /* Codex PR #29 review "treat sampling as a negotiated client capability" (comment * 4000335132): whether THIS gateway instance negotiated the sampling capability at @@ -207,6 +333,15 @@ class GatewayProxy { * after computing that response could unlock tools/list and tools/call before the * client has actually completed the MCP-required handshake. */ this.agentInitialized = new Map(); // connectionId -> "none" | "awaiting_notification" | "ready" + /* Frontier-panel decision (Paul, 2026-09-10, cluster E2): connectionIds this + * gateway's own startup recovery (gateway.js#recoverCrashedSessions) flagged as + * pendingOperatorReview -- consulted at dispatch time to quarantine a TOOL (not just + * an exact intent key) until an operator resolves the connection that left it + * ambiguous. A plain snapshot from startup, not live-updated: recovery-resolve/ + * recovery-abandon require the writer-claim this live gateway process already holds, + * so nothing else can change these connections' outcome while this process is up -- + * see recovery.js#resolveIntentNotExecuted's own header comment. */ + this.pendingOperatorReviewConnections = new Set(opts.pendingOperatorReviewConnections || []); /* Optional structured per-call log sink (board decision 2026-09-04, PR #29 review * "emit the required structured log for each call") -- defaults to a no-op so unit * tests that construct a GatewayProxy directly (no logging concern of their own) @@ -233,11 +368,49 @@ class GatewayProxy { } openConnection(connectionId, options = {}) { + /* Round-1 fix-plan commit 6: PRIMARY enforcement point (see this class's own + * getChainIntegrityFailure doc above) -- a genuine chain-integrity failure, + * however and whenever it was detected (startup reconcileHead, or an append-time + * checkHeadAgainstTailOrRepair failure from this same process's own prior + * closeConnection/recoverCrashedSessions), must immediately stop admitting NEW + * sessions. This is belt-and-braces on top of the append-time check (commit 4), + * which remains the PRIMARY correctness guarantee: even if a caller somehow got a + * session opened and its downstream calls dispatched, chain.appendSession's own + * checkHeadAgainstTailOrRepair would still refuse the eventual append. Refusing + * HERE, at admission, is what makes a runtime transition into failure behave the + * same as detecting it at startup rather than merely being recorded for the next + * restart. Checked before the acceptingNewSessions drain/writer-claim-lost gate + * below so the error a caller sees names the failure that actually happened, not a + * generic "not accepting sessions" message. */ + const integrityFailure = this.getChainIntegrityFailure(); + if (integrityFailure) { + throw fail( + `Gateway is not accepting new sessions: a chain-integrity failure is latched (${integrityFailure.reason}` + + `${integrityFailure.class ? `, class=${integrityFailure.class}` : ""}, latched at ${integrityFailure.at}). ` + + `This is a structural-corruption finding, not a transient condition a retry will clear -- see ` + + `${chain.CHAIN_CORRUPTION_RUNBOOK} for the recovery procedure.`, + "GATEWAY_CHAIN_INTEGRITY_FAILED" + ); + } if (!this.acceptingNewSessions) { throw fail("Gateway is no longer accepting new sessions (writer-claim lost or shutting down)", "GATEWAY_NOT_ACCEPTING"); } if (this.sessions.has(connectionId)) throw fail(`connectionId "${connectionId}" is already open`, "GATEWAY_DUPLICATE_CONNECTION"); - const s = session.createSession(connectionId, { now: this.now, goal: options.goal, samplingNegotiated: this.agentSupportsSampling }); + /* `options.sessionId` (Cluster B): normally omitted so session.createSession mints a + * fresh UUID for this live session -- see that function's own doc comment. Exposed + * here only so tests can deliberately force two DIFFERENT connections to share a + * session_id (and therefore a bundle_id) to exercise bundle-id-collision handling, + * exactly as they could before session_id existed by giving two connections + * identical {init, tools, calls} content; no real caller of openConnection has a + * reason to ever pass this. Combined with samplingNegotiated (PR #29 "treat sampling + * as a negotiated client capability") -- both fields are independent, additive + * session-creation inputs and there is no interaction between them. */ + const s = session.createSession(connectionId, { + now: this.now, + goal: options.goal, + sessionId: options.sessionId, + samplingNegotiated: this.agentSupportsSampling, + }); /* SS3.3: the granted tool surface must be recorded regardless of whether the agent * ever bothers to issue tools/list on this connection -- otherwise a cached tool * invoked without a prior tools/list would be sealed with an empty granted surface, @@ -245,6 +418,33 @@ class GatewayProxy { * call the gateway legitimately authorized. The (idempotent) tools/list handler below * simply re-records the same surface if the agent does ask. */ session.recordToolsList(s, this.mergedTools); + /* Codex PR #29 Finding 1 (Option C, external-panel-reviewed design -- see + * option-c-hardened-design.md): crash-recovery WAL, recorded from the very start of + * the connection so a crash before any tool call still leaves a durable record of + * the granted tool surface and goal. Separate, unsigned, outside the hash chain -- + * see recovery.js's own header for why this makes zero changes to session.js's own + * call-recording functions or to sealBoundaryBundle. + * + * Codex PR #33 review "roll back session publication when its first WAL write + * fails": this append used to run AFTER `this.sessions.set(...)` below, and + * unguarded -- if it threw (e.g. a full state volume), openConnection still threw + * out to its caller (who never gets a usable connectionId back), but the session had + * ALREADY been published into this.sessions/agentInitialized: an unreachable "ghost" + * session that stays counted by openSessionCount() and is later sealed at shutdown. + * Persist first; only publish the session into this proxy's live maps once the WAL + * durably has its start event. */ + recovery.appendWalEvent(this.stateDir, connectionId, { + type: "SESSION_START", + started_at: s.startedAt, + goal: s.goal || null, + // Cluster B: persists the session_id session.createSession minted for this live + // session so a later replay (gateway.js#recoverCrashedSessions/abandonConnection) + // reads the SAME id back instead of minting a new one -- see session.js# + // createSession's own doc comment on why that distinction matters for + // replay-idempotency. + session_id: s.sessionId || null, + tools: this.mergedTools.map((t) => ({ name: t.name, server: t.server, schema: t.schema })), + }); this.sessions.set(connectionId, s); this.agentInitialized.set(connectionId, INIT_NONE); return s; @@ -343,6 +543,41 @@ class GatewayProxy { const mergedServerInfo = serverNames.length === 1 ? this.serverInfos[serverNames[0]] : { name: "graphsmith-standalone-gateway(" + serverNames.join("+") + ")", version: "1.0", fronted: this.serverInfos }; + /* Codex PR #33 review "commit initialization only after its WAL event succeeds": + * this used to record into the in-memory session and flip agentInitialized=true + * BEFORE this WAL append, with the append itself unguarded. If the append then + * failed (e.g. a full state volume), the exception escaped handleMessage entirely + * (breaking its own "never throws" contract) while leaving agentInitialized already + * true -- a retried initialize would be rejected as a repeat, yet crash replay would + * never see an INITIALIZE event for this connection, sealing an inconsistent + * recovered session. Persist first; only commit the in-memory mutations once the WAL + * durably has this event, matching the existing CALL_START-failure pattern below. */ + /* Round-2 fix pass: this connection's WAL cannot legitimately already be poisoned + * this early (SESSION_START, the only prior append for this connectionId, either + * durably succeeds or -- per its own doc comment in openConnection -- throws + * straight out before this session is ever published, leaving no session here to + * poison) -- gated anyway, uniformly with every other call site below, so this + * file's own enforcement is a single provable invariant ("every appendWalEvent call + * site in this file checks s.walPoisoned first") rather than one that happens to + * hold only because of each site's own reachability analysis. */ + if (s.walPoisoned) { + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED", tool: "initialize", detail: s.walPoisoned.reason }); + if (isNotification) return null; + return walPoisonedError(id, s, "initializing this connection"); + } + try { + recovery.appendWalEvent(this.stateDir, connectionId, { + type: "INITIALIZE", + clientInfo: params && params.clientInfo, + serverInfo: mergedServerInfo, + model: params && params.model, + }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: "initialize", detail: walError.message }); + if (isNotification) return null; + return { jsonrpc: "2.0", id, error: { code: -32000, message: `Failed to durably record this connection's initialize before completing it: ${walError.message}. Not initialized -- safe to retry.` } }; + } session.recordInitialize(s, { clientInfo: params && params.clientInfo, serverInfo: mergedServerInfo, @@ -454,28 +689,497 @@ class GatewayProxy { } const conn = method === "tools/call" ? this.connections.get(serverName) : this.connections.values().next().value; const callArgs = method === "tools/call" ? (params && params.arguments) : params; - const ts = this.now(); + /* Correlate by an internally-generated marker even for notification-shaped calls - * (SS3.3's Map-keyed-by-id requirement is about the DOWNSTREAM leg's own id, which - * downstream.js already manages; here we key the SESSION record by the AGENT's own - * JSON-RPC id when present, or a synthetic one for a fire-and-forget call). */ - /* Codex PR #29 review "handle JSON-RPC null IDs before recording calls": JSON-RPC - * 2.0 permits an explicit `id: null` on a REQUEST (distinct from a notification, - * which omits the id key entirely) -- `isNotification` above only catches the - * latter. Using `id` (null) directly as this call's correlation key made it into - * session.recordCallStart's Map key, which explicitly throws INVALID_ARGUMENT on a - * null/undefined key -- turning a legal-if-unusual request into an uncaught - * internal error instead of a normal response. A null id still gets a real - * response below (the outer `id` variable, unchanged, is echoed back as JSON-RPC - * requires) -- only the internal bookkeeping key needs to never be null. */ + * (SS3.3's Map-keyed-by-id requirement is about the DOWNSTREAM leg's own id, + * which downstream.js already manages; here we key the SESSION record by the + * AGENT's own JSON-RPC id when present, or a synthetic one for a fire-and-forget + * call). Codex PR #29 review "handle JSON-RPC null IDs before recording calls": + * JSON-RPC 2.0 permits an explicit `id: null` on a REQUEST (distinct from a + * notification, which omits the id key entirely) -- `isNotification` above only + * catches the latter. A null id still gets a real response below (the outer `id` + * variable, unchanged, is echoed back as JSON-RPC requires) -- only the internal + * bookkeeping key needs to never be null. + * + * Computed HERE, before the idempotency-intent dispatch guard below (Codex PR #33 + * review "reject duplicate request IDs before creating intents"): this used to be + * computed only right before session.recordCallStart, well after a durable + * "dispatched" intent could already have been created for a brand-new intentKey. + * An agent reusing an in-flight JSON-RPC id for a genuinely different operation + * would then get that intent created, dispatch never actually reach + * recordCallStart (it throws DUPLICATE_JSONRPC_ID) -- an exception that, + * unguarded, escaped handleMessage entirely -- and be left with an orphaned + * "dispatched" intent permanently blocking a call that never happened. Checking + * id availability first means a genuine duplicate id is rejected as an ordinary + * JSON-RPC error before anything durable is touched. */ const correlationKey = isNotification || id === null ? Symbol(`${isNotification ? "notify" : "null-id"}:${toolName}`) : id; + if (!isNotification && id !== null && s.pendingCalls.has(correlationKey)) { + return { jsonrpc: "2.0", id, error: { code: -32600, message: `Duplicate in-flight JSON-RPC id ${JSON.stringify(id)} on this connection -- a response for it is still pending.` } }; + } + + /* Codex PR #29 Finding 2 (Option C, external-panel-reviewed design -- see + * option-c-hardened-design.md): a durable idempotency fence for real tools/call + * dispatch, keyed by (connectionId, tool, canonicalized arguments) so a retry + * under a NEW JSON-RPC id is still recognized as the same logical operation. + * Deliberately scoped to real tools/call only -- the isModelCallMethod branch + * (an agent sending "sampling/createMessage" into this dispatcher, a different + * case from gateway.js's own downstream-initiated sampling forward) is out of + * scope for this fix; see the design doc's disclosed limitations. */ + let intentKey = null; + let dispatchGeneration = 1; + // Hoisted above the tools/call-only block below (unlike its previous purely-local + // declaration) so the post-dispatch completion handling further down -- which + // records this call's outcome into the Cluster A retained-signature store -- can + // still see which idempotency key (if any) the caller actually supplied. + let callerIdempotencyKey = null; + /* Codex PR #33 review "restore the prior completed intent when CALL_START WAL + * persistence fails" / CodeRabbit PR #33 review (same finding): when a completed + * intent is superseded below, this snapshot is the complete pre-supersede record -- + * consulted by the CALL_START WAL-failure rollback further down so a transient WAL + * failure restores the earlier generation's real cached result instead of deleting + * it outright (which would make a later request carrying the ORIGINAL idempotency + * key find no record and dispatch again). Left null for the "brand new intent" + * dispatch path, where deleteIntent on rollback is already correct (nothing to + * restore). */ + let supersededIntentSnapshot = null; + if (method === "tools/call") { + intentKey = recovery.computeIntentKey(connectionId, toolName, callArgs); + /* Frontier-panel decision (Paul, 2026-09-10, PR #33 review clusters E1/E2 -- see + * graphsmith-pr33-panel-verdict-pr29-round4-halt-status-2026-09-10.md): a + * caller-supplied idempotency key, carried in `params._meta.idempotencyKey`, + * disambiguates "replay a call that already completed" from "dispatch a new, + * independent call that happens to share this tool+arguments" -- argument + * equality alone cannot tell those apart, so two deliberate identical calls used + * to silently collapse into one (both reviewers' finding). Only the COMPLETED + * branch below is key-gated; the dispatched/ambiguous blocking branches stay + * unconditional, since an unproven-outcome retry is a real double-dispatch risk + * independent of caller intent. */ + callerIdempotencyKey = + params && params._meta && typeof params._meta.idempotencyKey === "string" && params._meta.idempotencyKey.length > 0 + ? params._meta.idempotencyKey + : null; + let intentDecision = null; + for (let attempt = 0; attempt < 2 && !intentDecision; attempt++) { + const existing = recovery.readIntent(this.stateDir, intentKey); + if (existing && existing.state === "completed") { + if (callerIdempotencyKey && existing.idempotency_key && callerIdempotencyKey === existing.idempotency_key) { + intentDecision = { kind: "replay", cachedResult: existing.cached_result }; + } else { + /* No caller-signaled retry: this is a new, independent dispatch that + * happens to share this connection+tool+arguments with a call that already + * completed. Supersede the record in place with a fresh "dispatched" + * generation under the SAME intentKey -- still fenced against an + * in-flight/crashed retry of THIS new attempt -- rather than creating a + * second file, keeping exactly one record per (connection, tool, args). */ + try { + // Snapshot BEFORE overwriting -- see supersededIntentSnapshot's own doc + // comment above for why this must be the complete pre-supersede record. + supersededIntentSnapshot = existing; + recovery.updateIntent(this.stateDir, intentKey, { + state: "dispatched", + dispatched_at: this.now(), + idempotency_key: callerIdempotencyKey, + generation: (existing.generation || 1) + 1, + cached_result: undefined, + completed_at: undefined, + resolved_at: undefined, + resolution: undefined, + }); + dispatchGeneration = (existing.generation || 1) + 1; + intentDecision = { kind: "dispatch" }; + /* Codex PR #33 review "make the post-fence diagnostic non-throwing": the + * durable intent was already flipped to "dispatched" by updateIntent + * directly above, so a caller-supplied logger that throws here is caught + * by this block's own catch and rethrown (its code is not + * GATEWAY_RECOVERY_INTENT_NOT_FOUND), aborting before recordCallStart, + * the CALL_START WAL append and the downstream dispatch -- leaving a + * false in-flight fence that blocks every future retry of an effect that + * never happened. safeLog is this file's existing non-throwing wrapper + * for exactly this "log after a durable state change" position. */ + this.safeLog(JSON.stringify({ event: "gateway_intent_signature_reused_without_key", connection_id: connectionId, tool: toolName, intent_key: intentKey, generation: dispatchGeneration, detail: "a completed call's signature was reused without a matching idempotency key -- dispatched as a new, independent call rather than replayed" })); + } catch (updateError) { + supersededIntentSnapshot = null; + if (updateError.code !== "GATEWAY_RECOVERY_INTENT_NOT_FOUND") throw updateError; + // else: intent was concurrently removed -- loop again, this attempt's + // re-read will see no existing intent and take the fresh-dispatch path. + } + } + } else if (existing && existing.state === "dispatched") { + intentDecision = { + kind: "block", + code: -32080, + gatewayCode: "GATEWAY_AMBIGUOUS_RETRY", + message: "A prior attempt of this exact operation is still in flight -- dispatch halted to avoid a duplicate side effect.", + }; + } else if (existing && existing.state === "ambiguous") { + intentDecision = { + kind: "block", + code: -32081, + gatewayCode: "GATEWAY_DOWNSTREAM_OUTCOME_UNKNOWN", + message: "A prior attempt of this exact operation did not reach a confirmed successful outcome -- dispatch halted pending operator resolution (see recovery-resolve).", + }; + } else if (existing && existing.state === "not_executed") { + /* CodeRabbit PR #33 review "add a not_executed branch to the live dispatch + * guard": this state is only ever set by an operator's recovery-resolve + * --confirmed not-executed on a CRASHED connection (recovery.js# + * resolveIntentNotExecuted). Without this branch, this fell through to the + * final `else` below, whose createIntentIfAbsent returns null for the + * already-existing file -- after two such attempts `intentDecision` was never + * set, and the generic "could not establish a durable dispatch intent" + * AMBIGUOUS_RETRY fallback fired instead of a response that actually reflects + * what happened (an operator already proved this exact operation never ran). + * Report that fact with its own code/message rather than a generic one. */ + intentDecision = { + kind: "block", + code: -32084, + gatewayCode: "GATEWAY_OPERATOR_CONFIRMED_NOT_EXECUTED", + message: "An operator confirmed via recovery-resolve that a prior attempt of this exact operation did NOT execute downstream -- this intent record has not yet been cleaned up; retry once the owning crashed connection finishes being sealed.", + }; + } else { + /* Frontier-panel decision (Paul, 2026-09-10, cluster E2): a reconnecting + * agent gets a brand-new connectionId, so its retry of "the same" logical + * operation computes a DIFFERENT intentKey than a still-unresolved intent its + * own crashed predecessor connection left behind -- the exact-key lookup + * above would never see it. Before dispatching a genuinely new key, also + * check whether this TOOL (any arguments) has an unresolved intent + * (dispatched or ambiguous) belonging to a connection this gateway's own + * startup recovery already flagged as pendingOperatorReview -- if so, this + * tool stays quarantined until an operator resolves that connection, + * regardless of which connection or arguments are now trying to use it. + * Deliberately scoped to pendingOperatorReview connections only -- concurrent + * use of the same tool by unrelated, healthy live connections is normal and + * unaffected (same O(all intents on disk) disclosed cost as + * recovery.listAllIntents' other callers; only reached on a brand-new key). */ + /* CodeRabbit PR #33 review "fail closed when the quarantine scan cannot read + * an intent": listAllIntents propagates GATEWAY_RECOVERY_INTENT_UNREADABLE/ + * GATEWAY_RECOVERY_INTENT_CORRUPT for ANY intent file on disk, including one + * totally unrelated to this dispatch -- this scan ran unguarded, ahead of the + * downstream recovery try/catch block, so a single unreadable/corrupt intent + * anywhere rejected handleMessage with a generic -32603 instead of this + * quarantine check's own documented -32082 JSON-RPC error, and the tool was + * NOT actually blocked from dispatching. Fail closed instead: a scan that + * cannot be proven safe blocks this dispatch via the existing quarantine + * response, same as a real quarantine hit. */ + let quarantine = null; + if (this.pendingOperatorReviewConnections.size > 0) { + try { + quarantine = recovery.listAllIntents(this.stateDir).find( + (i) => + i.tool === toolName && + i.intent_key !== intentKey && + this.pendingOperatorReviewConnections.has(i.connection_id) && + (i.state === "dispatched" || i.state === "ambiguous") + ) || null; + } catch (scanError) { + quarantine = { connection_id: `` }; + } + } + if (quarantine) { + intentDecision = { + kind: "block", + code: -32082, + gatewayCode: "GATEWAY_TOOL_QUARANTINED_PENDING_OPERATOR_REVIEW", + message: `Tool "${toolName}" is quarantined: connection "${quarantine.connection_id}" left an unresolved call to this tool from a crash, pending operator review -- resolve it via recovery-resolve/recovery-abandon before dispatching this tool again.`, + }; + } else { + /* Cluster A (cross-connection replay): this connectionId has never itself + * dispatched this exact (tool, arguments) before (that is what "brand new + * intentKey" means, since intentKey is scoped to THIS connectionId -- see + * computeIntentKey's own header). A reconnecting agent's retry after its + * old connection crashed looks EXACTLY like this from the gateway's point + * of view: new connectionId, same tool+arguments. Consult the + * connection-independent retained-signature store -- but ONLY when the + * caller supplies an idempotencyKey that matches the one recorded at the + * time that prior call completed, mirroring the live, same-connection + * replay gate above exactly (an unkeyed retry is always treated as a new, + * independent dispatch, never silently collapsed onto an old result). + * Deliberately placed AFTER the quarantine check above, not before: a tool + * quarantined because SOME crashed connection left it with an unresolved + * outcome must stay blocked even for a caller that also happens to supply + * a matching idempotency key for a DIFFERENT, already-resolved signature -- + * "extend the existing quarantine mechanism to also cover this case" per + * this fix's own design note, rather than give the retained-signature path + * a way to bypass it. */ + let replayed = null; + let signatureReadFailure = null; + if (callerIdempotencyKey) { + let retained = null; + try { + retained = recovery.readCompletedSignature(this.stateDir, recovery.computeSignatureKey(toolName, callArgs, callerIdempotencyKey)); + } catch (signatureError) { + /* Codex PR #33 review "block dispatch when a retained signature is + * unreadable": this comment already claimed to fail closed, but setting + * `retained = null` is a CACHE MISS -- it fell straight through to + * createIntentIfAbsent and a real downstream dispatch. An EIO/permission + * failure or a corrupt signature file therefore repeated an + * already-completed side effect at precisely the moment its deduplication + * evidence could not be verified. Fail closed for real, the same way the + * quarantine scan above does: turn the unprovable read into a block. + * readCompletedSignature returns null (not a throw) for an absent or + * expired record, so this path is only ever reached by a genuinely + * unreadable/corrupt file, and the blast radius is exactly the one + * (tool, arguments, idempotency key) whose evidence is damaged -- any + * other key hashes to a different signature file. */ + signatureReadFailure = signatureError; + } + if (retained && retained.idempotency_key && retained.idempotency_key === callerIdempotencyKey) { + replayed = retained; + } + } + if (signatureReadFailure) { + intentDecision = { + kind: "block", + code: -32083, + gatewayCode: "GATEWAY_RETAINED_SIGNATURE_UNREADABLE", + message: `The retained completed-call record that would prove whether this idempotent retry already executed could not be read (${signatureReadFailure.message}) -- dispatch halted rather than risk repeating a side effect that may already have happened. Inspect/remove the damaged record under gateway-recovery/signatures/ before retrying.`, + }; + } else if (replayed) { + intentDecision = { kind: "replay", cachedResult: replayed.cached_result }; + } else { + const created = recovery.createIntentIfAbsent(this.stateDir, intentKey, { + connection_id: connectionId, + tool: toolName, + arguments: callArgs, + state: "dispatched", + dispatched_at: this.now(), + idempotency_key: callerIdempotencyKey, + generation: 1, + }); + if (created) intentDecision = { kind: "dispatch" }; + // else: lost a race to a concurrent identical call on this connection -- + // loop once to re-read its freshly-created state and respond consistently. + } + } + } + } + if (!intentDecision) { + // Two races back to back is vanishingly unlikely; fail closed rather than loop + // forever or risk a double dispatch on an assumption. + intentDecision = { + kind: "block", + code: -32080, + gatewayCode: "GATEWAY_AMBIGUOUS_RETRY", + message: "Could not establish a durable dispatch intent for this operation -- halted rather than risk a duplicate side effect.", + }; + } + if (intentDecision.kind === "replay") { + if (isNotification) return null; + /* Codex PR #33 review "persist cached replays in the session trace": this branch + * returned the cached tool result WITHOUT recordCallStart/recordCallResult and + * without either WAL append, so the agent observed a successful tools/call that + * appeared in neither the clean-close nor the crash-recovered signed trace -- + * defeating this PR's own premise that the sealed bundle records every MCP + * boundary input/output the session handled. Record it as a real, ordered step, + * explicitly MARKED as a replay (`replayed: true`, carried through session.js's + * existing optional-marker mechanism -- the same one `disconnected` already + * uses -- and through the WAL so crash recovery reconstructs it identically) so + * the trace is complete without misrepresenting a cache hit as a fresh + * downstream dispatch. The per-session admission caps above (MAX_PENDING_CALLS_ + * PER_SESSION / MAX_COMPLETED_CALLS_PER_SESSION) already gate this path, so the + * extra recorded step stays bounded exactly like any other call. + * + * Durability posture, deliberately fail-OPEN: unlike a fresh dispatch there is + * nothing here this gateway could refuse to do -- the side effect already + * happened, on an earlier call that WAS durably recorded, and withholding the + * cached result would only push the caller into another retry that lands right + * back on this same branch. So an append failure poisons the WAL (C1 SS2), + * attests itself as an anomaly, and still returns the replayed result -- the + * same choice proxy.js's own CALL_RESULT append already makes. */ + const replayTs = this.now(); + const replayCallSeq = s.nextCallSeq; + session.recordCallStart(s, correlationKey, { tool: toolName, server: serverName, arguments: callArgs, isModelCall: false, ts: replayTs }); + /* Round-N fix (frontier-panel finding #5, Paul's explicit choice of Fable's + * "make the gap observable" proposal over the majority's fail-open-and-document + * position): both anomalies below now carry call_seq (this replayed step's own + * identity) so a normally-sealed bundle's anomaly list says exactly WHICH step's + * WAL record is in question, not just that some unspecified append failed -- + * previously indistinguishable from any other WAL failure in the same session. + * This does not change the deliberate fail-open durability posture documented + * above (append failure still poisons and still returns the cached result) -- + * only what gets recorded about it. See gateway.js#recoverCrashedSessions for + * the companion half: an explicit INCOMPLETE marker when a crash leaves this + * exact replayed step's CALL_START durably written but its CALL_RESULT missing. */ + if (s.walPoisoned) { + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED", tool: toolName, call_seq: replayCallSeq, intent_key: intentKey, detail: s.walPoisoned.reason }); + } else { + try { + recovery.appendWalEvent(this.stateDir, connectionId, { type: "CALL_START", call_seq: replayCallSeq, tool: toolName, server: serverName, arguments: callArgs, ts: replayTs, generation: dispatchGeneration, replayed: true }); + recovery.appendWalEvent(this.stateDir, connectionId, { type: "CALL_RESULT", call_seq: replayCallSeq, result: intentDecision.cachedResult, isError: false, ts: replayTs, generation: dispatchGeneration, replayed: true, intent_key: intentKey }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: toolName, call_seq: replayCallSeq, intent_key: intentKey, detail: `replayed step: ${walError.message}` }); + } + } + session.recordCallResult(s, correlationKey, { result: intentDecision.cachedResult, isError: false, ts: replayTs, replayed: true, replayedFromIntentKey: intentKey }); + /* Codex PR #33 review "emit one log line for each replayed step": an earlier + * round of this same PR added a gateway_call_completed line ALONGSIDE the + * pre-existing gateway_intent_replayed one, so every cached replay emitted two + * records for a single handled step and any consumer counting steps double- + * counted it -- breaking AGENTS.md's one-line-per-step invariant that the same + * earlier round was trying to satisfy. Consolidated: ONE gateway_call_completed + * line, in the same shape every other handled call emits, carrying the replay + * metadata (status "replayed" plus the intent key the separate + * gateway_intent_replayed event used to supply) as fields rather than as a + * second line. */ + this.safeLog(JSON.stringify({ event: "gateway_call_completed", connection_id: connectionId, step: replayCallSeq, tool: toolName, server: serverName, status: "replayed", duration_ms: 0, replayed: true, intent_key: intentKey })); + return { jsonrpc: "2.0", id, result: intentDecision.cachedResult }; + } + if (intentDecision.kind === "block") { + session.recordAnomaly(s, { kind: intentDecision.gatewayCode, tool: toolName, intent_key: intentKey, detail: intentDecision.message }); + /* Codex PR #33 review "append blocked-retry anomalies to the WAL": the + * anomaly above is recorded in-memory only -- a crash before this connection's + * own close would silently drop it from the recovered/sealed bundle, even + * though the gateway genuinely prevented a duplicate dispatch. Durable WAL + * event + the matching structured log line (see the replay branch above). + * + * CodeRabbit PR #33 review "guard this WAL append the same way the CALL_START + * append is guarded": handleMessage documents that it never throws for a + * well-formed envelope, but this append was unguarded -- if it failed (e.g. + * ENOSPC), the exception would escape handleMessage entirely instead of still + * returning the block response the gateway already decided on. Degraded + * durability for this one anomaly (already recorded in-memory above) is + * preferable to an escaping exception on the fence path. */ + /* Round-2 fix pass: gated on s.walPoisoned like every other appendWalEvent call + * site in this file (see poisonWalOnFailure/walPoisonedError's own doc comments + * above) -- this append is already best-effort (a failure here does not change + * the block response returned below), so a skip-due-to-poisoning is likewise + * best-effort: attest it via the same structured log line a real append failure + * gets, rather than silently doing nothing. */ + if (s.walPoisoned) { + this.safeLog(JSON.stringify({ event: "gateway_wal_append_skipped_poisoned", connection_id: connectionId, tool: toolName, type: "ANOMALY", detail: s.walPoisoned.reason })); + } else { + try { + recovery.appendWalEvent(this.stateDir, connectionId, { type: "ANOMALY", kind: intentDecision.gatewayCode, tool: toolName, intent_key: intentKey, detail: intentDecision.message, ts: this.now() }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + this.safeLog(JSON.stringify({ event: "gateway_wal_append_failed", connection_id: connectionId, tool: toolName, type: "ANOMALY", detail: walError.message })); + } + } + this.safeLog(JSON.stringify({ event: "gateway_call_completed", connection_id: connectionId, step: null, tool: toolName, server: serverName, status: "blocked", duration_ms: 0 })); + if (isNotification) return null; + return { + jsonrpc: "2.0", + id, + error: { code: intentDecision.code, message: intentDecision.message, data: { gateway_code: intentDecision.gatewayCode, intent_key: intentKey, retryable: false } }, + }; + } + // intentDecision.kind === "dispatch": fall through to the existing dispatch path. + } + + const ts = this.now(); + // Captured BEFORE recordCallStart consumes it -- see recovery.js's header on why + // this already-monotonic per-session counter is exactly the stable, JSON-safe + // replay identity Option C's WAL needs (a live correlationKey can be a Symbol, + // which cannot round-trip through the WAL's JSON lines). + const walCallSeq = s.nextCallSeq; session.recordCallStart(s, correlationKey, { tool: toolName, server: method === "tools/call" ? serverName : "sampling", arguments: callArgs, isModelCall: isModelCallMethod(method), ts }); + if (method === "tools/call") { + /* Codex PR #33 review "undo the fence when CALL_START persistence fails": the + * intent was already durably created as "dispatched" above (createIntentIfAbsent), + * before downstream dispatch, before this WAL append -- if THIS fails (e.g. the + * state volume is full), the call never actually reaches conn.call() below, but + * without this catch that exception would escape handleMessage entirely (breaking + * its own documented "never throws for a well-formed envelope" contract) while + * leaving the intent permanently "dispatched": every identical retry would then be + * blocked forever as "still in flight" for an operation that in fact never + * dispatched. Roll back both the intent and the just-added pendingCalls entry, and + * return a normal, retryable JSON-RPC error instead. */ + /* Round-2 fix pass, closing commit 5's WAL-poison enforcement gap: this is the + * one call site on this path where a poisoned connection was previously able to + * dispatch a brand-new tools/call and append its CALL_START straight onto a + * possibly-torn WAL line -- the exact violation the independent verify pass on + * commit 5 flagged. Treated exactly like a caught append failure just below (per + * docs/contracts/wal-append-failure-semantics.md C1 SS2: "the caller should treat + * the refusal exactly like a second append failure"): roll back the pendingCalls + * entry and the just-created/superseded intent the same way, attest it as an + * anomaly, and return the same "not dispatched -- safe to retry" shape (via + * walPoisonedError) a real append failure would. */ + if (s.walPoisoned) { + s.pendingCalls.delete(correlationKey); + try { + if (supersededIntentSnapshot) { + recovery.updateIntent(this.stateDir, intentKey, { + state: supersededIntentSnapshot.state, + dispatched_at: undefined, + idempotency_key: supersededIntentSnapshot.idempotency_key, + generation: supersededIntentSnapshot.generation, + cached_result: supersededIntentSnapshot.cached_result, + completed_at: supersededIntentSnapshot.completed_at, + resolved_at: supersededIntentSnapshot.resolved_at, + resolution: supersededIntentSnapshot.resolution, + }); + } else { + recovery.deleteIntent(this.stateDir, intentKey); + } + } catch (cleanupError) { /* best effort */ } + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED", tool: toolName, intent_key: intentKey, detail: s.walPoisoned.reason }); + if (isNotification) return null; + return walPoisonedError(id, s, "dispatching this call"); + } + try { + /* Cluster A (generation-aware crash recovery): tags this CALL_START event with + * the dispatch generation active at THIS dispatch, rather than leaving + * recoverCrashedSessions to infer it later from whatever generation the intent + * file happens to be sitting at by the time recovery runs -- the intent's own + * `generation` field is overwritten in place on every supersede, so without an + * explicit per-event tag a crashed EARLIER generation's own unresolved WAL + * event could be (and, before this fix, silently was) matched against a LATER + * generation's completed cached_result. See gateway.js#recoverCrashedSessions' + * own generation-match check for the other half of this fix. */ + recovery.appendWalEvent(this.stateDir, connectionId, { type: "CALL_START", call_seq: walCallSeq, tool: toolName, server: serverName, arguments: callArgs, ts, generation: dispatchGeneration }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + s.pendingCalls.delete(correlationKey); + try { + /* CodeRabbit/Codex PR #33 review "restore the prior completed intent when + * CALL_START WAL persistence fails": if this dispatch SUPERSEDED an earlier + * completed intent (supersededIntentSnapshot set above), deleting it outright + * would discard that earlier generation's real cached result -- a later + * request carrying the ORIGINAL idempotency key would then find no record and + * dispatch again instead of replaying it, defeating the whole point of E1's + * idempotency fence for the call that was already safely completed. Restore + * the exact pre-supersede fields instead. Only a brand-new intent (nothing + * superseded) still gets a plain delete -- there is no earlier record to lose. */ + if (supersededIntentSnapshot) { + recovery.updateIntent(this.stateDir, intentKey, { + state: supersededIntentSnapshot.state, + dispatched_at: undefined, + idempotency_key: supersededIntentSnapshot.idempotency_key, + generation: supersededIntentSnapshot.generation, + cached_result: supersededIntentSnapshot.cached_result, + completed_at: supersededIntentSnapshot.completed_at, + resolved_at: supersededIntentSnapshot.resolved_at, + resolution: supersededIntentSnapshot.resolution, + }); + } else { + recovery.deleteIntent(this.stateDir, intentKey); + } + } catch (cleanupError) { /* best effort */ } + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: toolName, intent_key: intentKey, detail: walError.message }); + if (isNotification) return null; + return { jsonrpc: "2.0", id, error: { code: -32000, message: `Failed to durably record this call before dispatch: ${walError.message}. Not dispatched -- safe to retry.` } }; + } + } const cancelKey = !isNotification ? `${connectionId}:${JSON.stringify(id)}` : null; - let result, transportFailed = false; + let result, transportFailed = false, transportErrorCode = null; try { + /* Option C: propagate this call's durable intent key to downstream.js's _meta + * merge (both transports) so a downstream that itself understands a conventional + * idempotency key gets genuine at-most-once execution too -- see downstream.js's + * META_IDEMPOTENCY_KEY header comment. Only ever set for real tools/call dispatch + * (intentKey is null for the isModelCallMethod branch, matching the dispatch + * guard above's own "deliberately scoped to tools/call only" decision). Suffixed + * with the generation number (E1 above) whenever this dispatch superseded a prior + * completed intent under the same key: without this, a downstream implementing + * its own idempotency-key dedup would see the SAME key as the earlier, unrelated + * completed call and could wrongly treat this new, independent dispatch as a + * duplicate of it -- undermining the very "treat as independent" outcome E1 + * exists to guarantee. The first generation keeps the plain, unsuffixed key + * (unchanged wire behavior for the common single-generation case). */ + const downstreamIdempotencyKey = intentKey ? (dispatchGeneration > 1 ? `${intentKey}.g${dispatchGeneration}` : intentKey) : undefined; result = await conn.call(method, params, undefined, cancelKey ? (downstreamId) => this.downstreamCallIds.set(cancelKey, { server: serverName, downstreamId }) - : undefined); + : undefined, downstreamIdempotencyKey); } catch (error) { /* Board decision 2026-09-04, PR #29 review "preserve downstream JSON-RPC error * envelopes": downstream.js's connectStdio/connectHttp both already attach the @@ -484,6 +1188,7 @@ class GatewayProxy { * flattening every downstream failure into a generic -32000. */ result = { error: error.message, code: error.code, rpcError: error.rpcError || null }; transportFailed = true; + transportErrorCode = error.code; } finally { if (cancelKey) this.downstreamCallIds.delete(cancelKey); } @@ -517,6 +1222,72 @@ class GatewayProxy { * out of scope). */ const malformedResult = !transportFailed && method === "tools/call" && isMalformedToolCallResult(result); const completedAt = this.now(); + + /* Codex PR #29 Finding 2 (Option C): resolve the intent based on the outcome. Only + * a structurally CLEAN success (no transport failure, no tool-level isError) + * proves the downstream reached a known-good terminal state -- deliberately + * stricter than treating an explicit downstream error as "safe to retry" (design + * doc point 7a: a tool can commit a side effect and still report failure). + * Everything else becomes `ambiguous`, a durable fence only an operator can clear. + * Runs regardless of `correlatedNow` below: the intent tracks the downstream's + * real outcome, independent of whether this session is still around to record it + * (if closeConnection already fenced this same intent as ambiguous while this call + * was in flight, a later proven outcome correctly resolves it here). */ + if (method === "tools/call") { + /* Codex PR #33 review "guard the post-dispatch intent update": closeConnection + * can fence this same intent to "ambiguous" and remove the session WHILE conn.call + * above is still being awaited; recovery-resolve --confirmed not-executed (now + * writer-claim-gated, but a live gateway can still be mid-shutdown when its own + * claim is lost) can also delete it. Either way updateIntent then throws + * GATEWAY_RECOVERY_INTENT_NOT_FOUND -- previously uncaught, escaping handleMessage + * entirely and preventing the JSON-RPC response for a call that DID complete. + * Catch only that specific error and log it; anything else is a real bug and + * should still surface. */ + try { + if (!isError) { + recovery.updateIntent(this.stateDir, intentKey, { state: "completed", completed_at: completedAt, cached_result: result }); + /* Cluster A (cross-connection replay of a proven-completed call): retained + * independently of this per-connection intent record (which a future + * SUPERSEDE on this SAME connection will overwrite) so a DIFFERENT, + * reconnecting connectionId presenting a matching caller idempotency key can + * still find this exact outcome -- see recovery.js's own header on this + * store and why it is deliberately narrower (idempotency-key-gated, TTL- + * bounded) than a plain argument match. Best-effort: a failure to retain this + * secondary record must not undo or fail the call that already genuinely + * completed -- the primary intent record above is already durable, and this + * is only ever an ADDITIONAL convenience for a future reconnect. */ + try { + recovery.recordCompletedSignature(this.stateDir, recovery.computeSignatureKey(toolName, callArgs, callerIdempotencyKey), { + tool: toolName, + arguments: callArgs, + intent_key: intentKey, + connection_id: connectionId, + generation: dispatchGeneration, + idempotency_key: callerIdempotencyKey, + cached_result: result, + completed_at: completedAt, + }); + } catch (signatureError) { + /* CodeRabbit PR #33 review "use safeLog, not log, once the downstream + * effect already landed": by this point the real call already completed + * and only this best-effort secondary cache write failed -- this.log can + * itself throw (see safeLog's own doc comment), which would incorrectly + * escape handleMessage and deny the agent the response it is already + * owed for a failure in a purely-optional convenience write. */ + this.safeLog(JSON.stringify({ event: "gateway_retained_signature_write_failed", connection_id: connectionId, tool: toolName, intent_key: intentKey, detail: signatureError.message })); + } + } else { + recovery.updateIntent(this.stateDir, intentKey, { + state: "ambiguous", + ambiguous_at: completedAt, + ambiguous_reason: transportFailed ? `transport failure: ${transportErrorCode || "unknown"}` : "downstream reported a tool-level error (isError: true)", + }); + } + } catch (error) { + if (error.code !== "GATEWAY_RECOVERY_INTENT_NOT_FOUND") throw error; + this.safeLog(JSON.stringify({ event: "gateway_intent_update_skipped", connection_id: connectionId, tool: toolName, intent_key: intentKey, reason: "intent no longer exists (concurrently removed by close or an operator resolution)" })); + } + } /* CodeRabbit PR #29 review "recording the result after the session is closed can * throw or write a false anomaly": closeConnection()/handleDownstreamDisconnect() * can run for this same connectionId WHILE conn.call() above is still being @@ -531,6 +1302,50 @@ class GatewayProxy { const correlatedNow = !s.finalized && s.pendingCalls.has(correlationKey); if (correlatedNow) { session.recordCallResult(s, correlationKey, { result, isError, malformedResult, ts: completedAt }); + if (method === "tools/call") { + /* Part of the Codex/CodeRabbit PR #33 review cluster on unguarded WAL appends + * (mirrors the CALL_START/INITIALIZE/ANOMALY appends elsewhere in this file): + * the downstream call has ALREADY completed for real by this point -- there is + * no "don't dispatch" option left, so a WAL write failure here must be + * best-effort (log it, keep going) rather than escape handleMessage and deny the + * agent the real response it is owed. The intent file's own cached_result + * (updated separately above) remains the source of truth recovery-replay falls + * back to if this WAL line never lands; see recoverCrashedSessions' own + * intent-consultation comment. */ + /* Round-2 fix pass: gated on s.walPoisoned like every other appendWalEvent call + * site in this file. Same best-effort shape as the real-failure catch just + * below (the downstream call already completed for real -- there is no + * "don't return the result" option left on this fail-OPEN path, see this + * block's own header comment on why proxy.js stays fail-open here unlike + * gateway.js's sampling CALL_RESULT path): skip the append, attest it as an + * anomaly and a structured log line, and still return the real result. */ + if (s.walPoisoned) { + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED", tool: toolName, intent_key: intentKey, detail: s.walPoisoned.reason }); + this.safeLog(JSON.stringify({ event: "gateway_wal_append_skipped_poisoned", connection_id: connectionId, tool: toolName, call_seq: walCallSeq, type: "CALL_RESULT", detail: s.walPoisoned.reason })); + } else { + try { + // Cluster A: tagged with the same dispatchGeneration as this call's own + // CALL_START event above -- see that event's own doc comment. + /* Merge fix (PR #29 x PR #33 composition, 2026-09-20): malformedResult is + * PR #29's own new audit annotation (isMalformedToolCallResult above, + * "flag structurally malformed tools/call results distinctly from + * tool-level errors") and did not exist when this WAL CALL_RESULT event + * shape was designed for PR #33's crash recovery. Included here so a crash + * between this append and CLOSING does not silently lose the flag on + * replay -- see gateway.js#recoverCrashedSessions' own CALL_RESULT replay + * branch, updated in the same fix to read it back. */ + recovery.appendWalEvent(this.stateDir, connectionId, { type: "CALL_RESULT", call_seq: walCallSeq, result, isError, malformedResult, ts: completedAt, generation: dispatchGeneration }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", tool: toolName, intent_key: intentKey, detail: walError.message }); + /* Codex PR #33 review "use safeLog once the downstream call already + * completed": the real tools/call result is already committed to being + * returned by this point -- a throw from this.log here must not be + * allowed to escape and deny the agent that already-earned response. */ + this.safeLog(JSON.stringify({ event: "gateway_wal_append_failed", connection_id: connectionId, tool: toolName, call_seq: walCallSeq, type: "CALL_RESULT", detail: walError.message })); + } + } + } /* Board decision 2026-09-04, PR #29 review "emit the required structured log for * each call": the only prior gateway log for a call was the session-finalize log * emitted much later (or never, if the process crashes first) -- this gives every @@ -623,8 +1438,89 @@ class GatewayProxy { (call, at) => this.logDisconnectedCall(connectionId, call, at) ); } - this.sessions.delete(connectionId); - this.agentInitialized.delete(connectionId); + /* Codex PR #29 Finding 2 (Option C): a "dispatched" intent whose owning connection + * is closing (agent hung up, or the connection is being force-closed) is exactly + * Finding 2's ambiguous case -- the downstream may still complete the side effect + * after this point even though the session itself is ending. Fence it the same way + * a crash mid-flight would; do not leave it looking permanently "in flight" once + * nothing is left to receive its response, and do not guess it failed. */ + /* Round-N fix (frontier-panel finding #1): listIntentsForConnection/updateIntent are + * disk I/O (recovery.js) and a failure partway through this fencing loop (e.g. an I/O + * error reading/writing an intent file -- exactly the fixture agent-transport.js's own + * stdio-disconnect catch and gateway.js's own shutdown-loop catch already name and + * handle) previously threw out of closeConnection BEFORE the two deletes below ran, + * leaving a ghost entry in this.sessions/this.agentInitialized forever -- + * openConnection's own admission checks and every other lookup key off connectionId, + * so a ghost session could never be closed or reopened again through the normal + * lifecycle, even though every existing caller already treats this exact failure as + * one it catches and continues past. Deliberately try/FINALLY, not try/catch: every + * caller of closeConnection already has its own established catch (and, after this + * same round's agent-transport.js fix, log) convention for a failure here -- this fix + * only guarantees the cleanup always runs, without swallowing the exception itself + * (callers still see and handle it exactly as before). One observable side effect: a + * connection whose intent-fencing failed during an ordinary (non-shutdown) disconnect + * is now actually removed from this.sessions immediately, so it can no longer be + * redundantly re-attempted (and re-logged) by gateway.js's own shutdown loop later -- + * see agent-transport.js's own updated doc comment for the corresponding logging fix + * this made necessary. */ + try { + for (const intent of recovery.listIntentsForConnection(this.stateDir, connectionId)) { + if (intent.state === "dispatched") { + recovery.updateIntent(this.stateDir, intent.intent_key, { + state: "ambiguous", + ambiguous_at: this.now(), + ambiguous_reason: reason || "connection closed with this call still in flight", + }); + } + } + } finally { + this.sessions.delete(connectionId); + this.agentInitialized.delete(connectionId); + } + /* Round-2 fix pass, closing commit 5's WAL-poison enforcement gap: the independent + * verify pass on commit 5 (b14bf35c) named this exact append as a GUARANTEED + * violation path -- every connection eventually closes, so any connection poisoned + * by an earlier append failure on this same file's own dispatch path (CALL_START/ + * CALL_RESULT/ANOMALY/INITIALIZE above) was previously 100% certain to have this + * CLOSING event concatenated onto its possibly-torn WAL line, with no check at all. + * `s` here is the same session object fetched at the top of this method (still + * pre-finalization -- session.recordAnomaly below only rejects a FINALIZED session, + * and finalizeSession has not run yet), so its walPoisoned flag (if any) is the + * live, current one. Skipping this append changes nothing else about close: the + * WAL file is left as-is (not deleted -- recovery.deleteWal below only ever runs + * after a successful chain.appendSession, unaffected by this) for an operator to + * inspect per the poisoned-connection contract; sealing/finalization proceeds + * exactly as it would if this append had been attempted and skipped for any other + * best-effort reason. */ + if (s.walPoisoned) { + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED", detail: s.walPoisoned.reason }); + this.safeLog(JSON.stringify({ event: "gateway_wal_append_skipped_poisoned", connection_id: connectionId, type: "CLOSING", detail: s.walPoisoned.reason })); + } else { + /* Codex PR #33 review "continue sealing when the closing WAL append fails": this was + * the one unguarded appendWalEvent left on this path, and it sits AFTER + * this.sessions.delete(connectionId) above -- so a throw here (ENOSPC, EIO) escaped + * closeConnection with the session already gone from both live maps and + * finalizeSession/chain.appendSession never reached. On the HTTP idle-timeout/DELETE + * path the caller swallows that rejection, so an otherwise completely healthy, + * finished session was permanently absent from the signed chain with nothing left in + * memory to retry from. The CLOSING event is INFORMATIONAL for replay (see + * gateway.js#recoverCrashedSessions: "CLOSING is informational only for replay"), so + * losing it is strictly less harmful than losing the whole sealed session. Treat it + * exactly like every other best-effort append failure in this file -- poison the WAL + * per docs/contracts/wal-append-failure-semantics.md C1 SS2 so no later append + * concatenates onto a possibly-torn line, ATTEST it as an anomaly so the sealed + * bundle is not silent about a durable write this gateway attempted and failed, log + * it through safeLog -- and then fall through to finalization rather than abandoning + * it. The WAL file itself is left in place for recovery either way (deleteWal only + * runs after a successful chain.appendSession). */ + try { + recovery.appendWalEvent(this.stateDir, connectionId, { type: "CLOSING", reason: reason || null }); + } catch (walError) { + poisonWalOnFailure(s, walError, this.now); + session.recordAnomaly(s, { kind: "GATEWAY_RECOVERY_WAL_APPEND_FAILED", detail: `CLOSING event: ${walError.message}` }); + this.safeLog(JSON.stringify({ event: "gateway_wal_append_failed", connection_id: connectionId, type: "CLOSING", detail: walError.message })); + } + } let sealed; try { sealed = session.finalizeSession(s, this.keys); @@ -685,12 +1581,24 @@ class GatewayProxy { * quarantine-and-report handling as an appendSession failure below, exactly as it * should: either way, this instance can no longer be trusted to append safely. */ this.renewWriterClaim(); - entry = chain.appendSession(this.stateDir, sealed); + /* `{ log: this.log }` (Round-1 fix-plan commit 4/6, chain-validity.md C2 SS4): + * chain.appendSession's own HEAD-vs-tail check logs loudly on an auto-repaired + * single-step lag -- preserved here across the writer-claim-renewal fix above, + * which touched this same call site independently for an unrelated reason. */ + entry = chain.appendSession(this.stateDir, sealed, { log: this.log }); } catch (error) { error.quarantinedTo = quarantineSealedBundle(this.stateDir, connectionId, sealed, error); this.onSealFailure(s, error); return null; } + /* Durably sealed: the WAL's job is done, and any intent that reached a clean + * "completed" state before close is no longer needed either. An `ambiguous` intent + * (see above) is NOT cleaned up here -- the fence must survive this session's own + * lifecycle; only an explicit operator resolution (recovery-resolve) removes one. */ + recovery.deleteWal(this.stateDir, connectionId); + for (const intent of recovery.listIntentsForConnection(this.stateDir, connectionId)) { + if (intent.state === "completed") recovery.deleteIntent(this.stateDir, intent.intent_key); + } this.onSessionFinalized(connectionId, entry, sealed); return entry; } diff --git a/scripts/gateway/recovery.js b/scripts/gateway/recovery.js new file mode 100644 index 00000000..2aa50151 --- /dev/null +++ b/scripts/gateway/recovery.js @@ -0,0 +1,842 @@ +#!/usr/bin/env node +/* GraphSmith standalone gateway -- crash-recovery WAL and idempotency-intent store. + * + * Fixes two Codex findings on PR #29, deliberately built as a SEPARATE PR after an + * independent 5-model external architecture panel review (Option C; see + * option-c-hardened-design.md in the project record) rather than fixed unilaterally by + * the team that wrote the original gateway code: + * + * Finding 1 (session.js): a completed tools/call lives only in memory until + * closeConnection() seals the whole session at once. A crash between "downstream + * returned a result" and "the agent disconnects" silently drops that evidence from the + * signed chain -- the gateway would have no way to know, on restart, that it ever + * observed that call at all. + * + * Finding 2 (proxy.js): nothing ties a retried tools/call to its original attempt, so + * a timeout-then-retry on a side-effecting tool (email, payment, ...) can execute the + * downstream side effect twice. + * + * Both close with a SEPARATE, UNSIGNED local recovery store -- this module -- leaving + * session.js, gsa-mcp-shim.js#sealBoundaryBundle, and chain.js's write order and + * collision semantics completely untouched (verified against the actual current code, + * not assumed): this module never signs anything and is never itself chain-appended. + * `session.js` is very nearly untouched by this change (its only diff, added late in PR + * #33 for Codex's "persist cached replays in the session trace" finding, is one OPTIONAL + * `replayed` marker spread conditionally into an already-recorded call -- exactly the + * mechanism the pre-existing `disconnected` marker already used, so no ordinary session's + * sealed shape changes): the WAL replays through session.js's + * existing, unmodified createSession/recordInitialize/recordToolsList/recordCallStart/ + * recordCallResult/markPendingAsDisconnected -- recovery reconstructs an in-memory + * session using the exact same recorder functions the live gateway already calls, so + * there is no second, parallel "session shape" to keep in sync. + * + * Layout under the existing `stateDir`: + * gateway-recovery/ + * active/.jsonl an append-only WAL, one JSON line per session event + * (SESSION_START/INITIALIZE/TOOLS_LIST/CALL_START/ + * CALL_RESULT/CLOSING), deleted once that connection's + * session is durably chain-appended. + * intents/.json one file per logical (connection, tool, args) + * operation -- NOT deleted just because its owning + * session got sealed; only deleted on a clean + * `completed` outcome at session-close cleanup, or by + * an explicit operator `resolveNotExecuted` call. An + * `ambiguous` intent is the durable fence Finding 2 + * needs and is never auto-expired by this module. + * + * Why append-only WAL rather than a rewritten-per-call snapshot (the shape most of the + * five external panel models proposed): every event is one `appendDurableLine` call -- + * the exact primitive chain.js already uses and this codebase already trusts -- with no + * per-call O(session-size) rewrite cost, and recovery replay needs no new + * snapshot/rehydrate shape to keep in sync with session.js: it just calls session.js's + * real functions with the real recorded arguments, in order. + */ +"use strict"; + +const fs = require("fs"); +const path = require("path"); +const crypto = require("crypto"); +const stateStore = require("../state-store.js"); + +const RECOVERY_DIRNAME = "gateway-recovery"; +const ACTIVE_DIRNAME = "active"; +const INTENTS_DIRNAME = "intents"; +const SIGNATURES_DIRNAME = "completed-signatures"; +const INTENT_SCHEMA_VERSION = 1; +/* Cluster A (generation-aware crash recovery / cross-connection replay -- Codex PR #29 + * Finding 2 follow-up): a completed intent is normally keyed by (connectionId, tool, + * args) alone (see computeIntentKey's own header on why that scoping is deliberate for + * the LIVE dispatch fence), which means a reconnecting agent -- a brand-new + * connectionId after its old connection crashed or dropped -- computes a DIFFERENT + * intentKey than its own crashed predecessor's completed call, and would otherwise + * re-dispatch a side effect that already durably succeeded. This second, much narrower + * store retains just enough about the MOST RECENT completed call for a given (tool, + * args) SIGNATURE (no connectionId) to let that reconnect replay the real result instead + * of re-executing it -- but ONLY when the caller proves it is deliberately asking for a + * replay via the same `params._meta.idempotencyKey` mechanism the live fence already + * uses (see proxy.js's own dispatch-guard doc comment); never a bare argument match, + * which would silently collapse two deliberately-repeated independent calls into one, + * exactly the ambiguity the caller-idempotency-key mechanism exists to resolve. Bounded + * by DEFAULT_RETAINED_SIGNATURE_TTL_MS below rather than kept forever. */ +const DEFAULT_RETAINED_SIGNATURE_TTL_MS = 24 * 60 * 60 * 1000; // 24h + +function fail(message, code = "GATEWAY_RECOVERY_ERROR") { + const error = new Error(message); + error.code = code; + return error; +} + +function recoveryDir(stateDir) { + return path.join(stateDir, RECOVERY_DIRNAME); +} +function activeDir(stateDir) { + return path.join(recoveryDir(stateDir), ACTIVE_DIRNAME); +} +function intentsDir(stateDir) { + return path.join(recoveryDir(stateDir), INTENTS_DIRNAME); +} +function quarantineDir(stateDir) { + return path.join(recoveryDir(stateDir), "quarantine"); +} +function signaturesDir(stateDir) { + return path.join(recoveryDir(stateDir), SIGNATURES_DIRNAME); +} + +/* CodeRabbit PR #33 review "reject path-bearing recovery identifiers": connectionId and + * intentKey normally come from this codebase's own generated IDs (safe), but + * recovery-resolve/recovery-abandon's CLI flags and this module's exported API both also + * accept them directly from a caller, unvalidated -- and path.join happily normalizes a + * "../" segment, so an unvalidated value could address a file outside its recovery + * directory. Applied at the two path-construction points below, not at every call site + * individually, so nothing can reach fs.* with an unsafe path regardless of caller. */ +const SAFE_ID = /^[A-Za-z0-9._-]+$/; +function assertSafeId(value, what) { + if (typeof value !== "string" || !SAFE_ID.test(value) || value === "." || value === "..") { + throw fail(`Unsafe ${what} ${JSON.stringify(value)} -- refusing to build a recovery path from it.`, "GATEWAY_RECOVERY_UNSAFE_ID"); + } + return value; +} +function walPath(stateDir, connectionId) { + return path.join(activeDir(stateDir), `${assertSafeId(connectionId, "connectionId")}.jsonl`); +} +function intentPath(stateDir, intentKey) { + return path.join(intentsDir(stateDir), `${assertSafeId(intentKey, "intentKey")}.json`); +} +function signaturePath(stateDir, signatureKey) { + return path.join(signaturesDir(stateDir), `${assertSafeId(signatureKey, "signatureKey")}.json`); +} + +/* Codex PR #33 review "restrict permissions on raw recovery records": these directories + * (and, below, the WAL/intent files inside them) hold raw goals, tool arguments, and + * cached results -- data the signed execution trace otherwise only ever stores as + * hashes. Under the common `022` umask, mkdirSync's default mode is `0755`, letting any + * other local user on the host read them. `0700` scopes the whole recovery tree to this + * process's own owner, matching the "OS process boundary is the trust boundary" + * convention this codebase already uses elsewhere. An explicit chmod (not just the mode + * passed to mkdirSync) also re-tightens a directory that was created by an older build + * of this file before this fix, since `recursive: true` does not revisit dirs that + * already existed. Best-effort: a chmod that fails (e.g. a filesystem that does not + * support POSIX modes) must not mask the real error from the write that follows. + * + * Codex PR #33 review "preserve group access when creating recovery state": the literal + * `0700`/`0600` this originally used predate startup-permissions.js's access model, which + * is explicitly group-readable (stateStore.DEFAULT_DIR_MODE 0750 / DEFAULT_FILE_MODE 0640, + * "NOT single-user 0700/0600") so same-group ops tooling running under another UID can + * read recovery state. That startup pass chmods these very directories to 0750 -- and + * then every subsequent WAL/intent/signature write came back through this helper and + * reset them to 0700, silently undoing it. Use the shared constants so the two agree; the + * original intent (never the umask-dependent world-readable 0755/0644 default) is + * unchanged, only the group bit that the later access model deliberately requires. */ +/* Round-N fix (frontier-panel finding #4): mkdirSync(dir, {recursive:true}) can create + * MULTIPLE new ancestor levels in one call (e.g. both gateway-recovery/ and + * gateway-recovery/active/ when neither existed yet on a fresh state_dir), but every call + * site below only ever fsyncs `dir` itself (the leaf) afterward, via its own later + * fsyncDir(dir) call following the actual WAL/intent/signature write (CodeRabbit PR #33's + * review just above named exactly that convention for the leaf). A newly created + * ancestor's own directory entry -- its name appearing in ITS parent -- never gets + * fsynced by anyone in that case, so a crash shortly after the FIRST-ever recovery write + * on a fresh state_dir could lose the gateway-recovery/ directory entry itself even + * though active/ and the file inside it were both durably fsynced, leaving a WAL file + * that is unreachable (its parent directory's own entry never landed) despite the + * gateway believing it was fully durable. Detect which ancestors are missing BEFORE + * creating anything (existsSync walk-up, stopping at the first that already exists), + * then fsync each of those -- and only those -- newly created ancestors after mkdirSync + * returns. `dir` itself is intentionally excluded here: every call site already fsyncs it + * via its own existing post-write fsyncDir(dir) call, unchanged by this fix. + * + * Post-review correction (found during PR #33/#29 merge-verification, still describing + * finding #4): the loop below originally stopped at the TOPMOST missing ancestor without + * also fsyncing that ancestor's own pre-existing PARENT (`cursor`, once the walk-up loop + * ends). fsyncDir(X) persists X's own directory CONTENTS -- i.e. that X's children are + * durably listed inside X -- it says nothing about whether X's own entry is durably + * listed inside X's parent. So the original code fsynced gateway-recovery/'s listing of + * active/ (via fsyncDir(gateway-recovery)) but never fsynced state_dir's listing of + * gateway-recovery/ itself -- exactly the "lost the gateway-recovery/ directory entry + * itself" scenario this comment already claimed to prevent. Fixed below by also fsyncing + * `cursor` (the first pre-existing ancestor the walk-up found) whenever anything was + * actually created under it. */ +function ensureDir(dir) { + const missingAncestors = []; + let cursor = dir; + while (cursor && !fs.existsSync(cursor)) { + missingAncestors.push(cursor); + const parent = path.dirname(cursor); + if (parent === cursor) break; // filesystem root -- stop rather than loop forever + cursor = parent; + } + + fs.mkdirSync(dir, { recursive: true, mode: stateStore.DEFAULT_DIR_MODE }); + try { + fs.chmodSync(dir, stateStore.DEFAULT_DIR_MODE); + } catch (error) { + /* best effort -- see doc comment above */ + } + + // Fsync every newly created ancestor's directory entry, topmost (closest to the + // pre-existing root) first. Index 0 is `dir` itself -- excluded, see doc comment above. + for (let i = missingAncestors.length - 1; i >= 1; i--) { + fsyncDir(missingAncestors[i]); + } + // The topmost missing ancestor's own directory ENTRY lives in `cursor` (the first + // pre-existing ancestor found by the walk-up above). Nothing else fsyncs cursor's + // listing, so without this, a crash right after the very first recovery write on a + // fresh state_dir could still lose the whole newly-created subtree's entry point. + if (missingAncestors.length > 0) { + fsyncDir(cursor); + } +} + +/* Same rationale as ensureDir's own doc comment above, for the individual WAL/intent + * files themselves: `fs.openSync(path, "a")`/atomicCreateExclusive/atomicOverwriteFile + * all default to `0666 & ~umask` (typically `0644`), which is WORLD-readable under a + * `022` umask. Called after every write (mirroring fsyncDir's own "not just on first + * creation" convention above) so a file that predates this fix is re-tightened too. + * Uses stateStore.DEFAULT_FILE_MODE (0640) rather than a literal 0600 for the same + * reason ensureDir's own doc comment above gives. */ +function restrictFileMode(filePath) { + try { + fs.chmodSync(filePath, stateStore.DEFAULT_FILE_MODE); + } catch (error) { + /* best effort -- see doc comment above */ + } +} + +/* CodeRabbit PR #33 review "fsync the recovery directory after creating WAL files": a + * file's own fsync (appendDurableLine, atomicCreateExclusive) guarantees its CONTENT is + * durable, but not that the new directory entry (its name appearing in its parent + * directory) survives a power loss -- that needs the parent directory's own fd fsynced + * too. Mirrors state-store.js's atomicOverwriteFile, which already fsyncs its target + * directory on every write for the same reason; called after every append/create here + * rather than only on first creation, matching that existing convention. + * + * Codex PR #33 review "propagate real directory fsync failures": this used to swallow + * EVERY error, so a genuine storage fault (EIO, ENOSPC) while making a brand-new WAL or + * intent's DIRECTORY ENTRY durable was reported to the caller as success. The file's own + * fsync does not cover its directory entry, so the gateway could dispatch a side effect + * believing CALL_START and its fence were durable and then restart to find neither. Only + * the cases this was actually written to tolerate are still ignored: the directory being + * gone (ENOENT -- mid-teardown in a test, the case the original comment names) and a + * platform/filesystem that simply does not support fsync on a directory fd. Everything + * else propagates, before dispatch. */ +const DIR_FSYNC_TOLERATED_CODES = new Set([ + "ENOENT", // the directory is already gone -- nothing left to make durable + "EINVAL", // filesystem rejects fsync on a directory fd + "ENOTSUP", + "EOPNOTSUPP", + "ENOSYS", + "EISDIR", // platforms (Windows) that refuse to open a directory as a file at all + "EPERM", + "EACCES", +]); +function fsyncDir(dir) { + let fd; + try { + fd = fs.openSync(dir, "r"); + fs.fsyncSync(fd); + } catch (error) { + if (!DIR_FSYNC_TOLERATED_CODES.has(error.code)) throw error; + /* tolerated platform/teardown case only -- see doc comment above */ + } finally { + if (fd !== undefined) fs.closeSync(fd); + } +} + +/* Codex PR #33 review "complete WAL writes before treating them as durable": fs.writeSync + * is permitted by Node's own docs to write FEWER bytes than requested in one call (short + * write) -- this previously trusted a single call to have written the whole line before + * fsyncSync'ing it. A short write here is silently invisible to the caller (appendWalEvent + * has already returned success to a dispatch decision), but readWalEvents' own "stop at + * the first malformed line" replay logic (see its header) treats the truncated line as a + * torn crash-tail and discards it AND every real, complete event appended after it. Loops + * until every byte of the encoded line has actually been written, on the same fd, before + * the fsync that is supposed to make it durable. */ +function writeFullySync(fd, buffer) { + /* CodeRabbit PR #33 review "truncate the partial record before throwing + * GATEWAY_RECOVERY_SHORT_WRITE": a short write that makes SOME progress before + * stalling (or a later call in this same loop stalling after earlier calls already + * wrote bytes) leaves an incomplete JSON line appended past the file's prior end -- + * readWalEvents' own "stop at the first malformed line" replay logic (see its header) + * then discards that truncated line AND every real, complete event appended after it. + * Capturing the size before any write lets a stall truncate the file back to exactly + * its pre-append state, so a subsequent successful append starts clean rather than + * leaving a torn line for replay to trip over. */ + const originalSize = fs.fstatSync(fd).size; + let offset = 0; + while (offset < buffer.length) { + /* Codex PR #33 review "roll back the WAL when a write throws, not only when it + * returns non-positive progress": fs.writeSync can itself throw (e.g. ENOSPC, + * EIO) partway through a multi-call write, leaving bytes from a prior successful + * writeSync call in this same loop already appended past the file's pre-append + * end. Only checking `written > 0` misses that case entirely, since a throw never + * reaches that check. Wrap the call itself so a throw also truncates back to + * originalSize before propagating, matching the short-write branch below. */ + let written; + try { + written = fs.writeSync(fd, buffer, offset, buffer.length - offset); + } catch (writeError) { + try { + fs.ftruncateSync(fd, originalSize); + } catch (truncateError) { + /* best effort -- see the short-write branch's identical comment below. */ + } + throw writeError; + } + if (!(written > 0)) { + try { + fs.ftruncateSync(fd, originalSize); + } catch (truncateError) { + /* best effort -- if truncation itself fails, the original SHORT_WRITE error + * below is still the one that matters; masking it with a truncate failure + * would hide the more actionable diagnosis. */ + } + throw fail(`fs.writeSync made no progress (wrote ${written} of ${buffer.length - offset} remaining byte(s)) -- refusing to fsync a possibly-incomplete record.`, "GATEWAY_RECOVERY_SHORT_WRITE"); + } + offset += written; + } +} + +/* Mirrors chain.js's own appendDurableLine, which is not exported from that module -- + * this codebase's own established convention (see config.js's header on why a second, + * small hand-rolled helper scoped to its own file is preferred over reaching into + * another module's private internals) is to duplicate a tiny fs primitive like this + * rather than change chain.js's export surface for an unrelated module's benefit. */ +function appendDurableLine(filePath, line) { + /* CodeRabbit PR #33 review "create recovery files with mode 0o600 at creation": + * fs.openSync's default mode (0666 & ~umask) leaves a freshly-created WAL file + * group/world-readable for the window between creation and restrictFileMode's chmod + * below under a permissive (022) umask. Passing an explicit mode here closes that + * window for new files; restrictFileMode is kept unchanged so a file created by an older + * build still gets tightened on its next append. The mode is stateStore.DEFAULT_FILE_MODE + * (0640) to match restrictFileMode -- see ensureDir's doc comment for why. */ + const fd = fs.openSync(filePath, "a", stateStore.DEFAULT_FILE_MODE); + try { + writeFullySync(fd, Buffer.from(`${line}\n`, "utf8")); + fs.fsyncSync(fd); + } finally { + fs.closeSync(fd); + } + restrictFileMode(filePath); +} + +/** Appends one WAL event for `connectionId`. Synchronous end-to-end (open/write/fsync/ + * close), matching chain.js's own durability discipline for chain.jsonl -- safe against + * interleaving from concurrent async call sites on the SAME connection (two tools/call + * messages pipelined without the agent awaiting the first) because Node's single- + * threaded event loop never runs other JS in the middle of a synchronous fs call. */ +function appendWalEvent(stateDir, connectionId, event) { + ensureDir(activeDir(stateDir)); + appendDurableLine(walPath(stateDir, connectionId), JSON.stringify({ ...event, recorded_at: Date.now() })); + fsyncDir(activeDir(stateDir)); +} + +/** Best-effort: moves a connection's WAL out of `active/` into `gateway-recovery/ + * quarantine/` rather than deleting it, for the case where the file could not even be + * read (permission/I/O error, not a JSON/content problem -- see readWalEvents) and an + * operator has explicitly decided to give up on recovering it (gateway.js#abandonConnection). + * A rename does not require read access to the file's own content, only to its directory + * entry, so this can succeed even when the read that motivated it could not. Returns the + * quarantined path, or null if there was no WAL file to move. */ +function quarantineWal(stateDir, connectionId) { + ensureDir(quarantineDir(stateDir)); + const from = walPath(stateDir, connectionId); + const to = path.join(quarantineDir(stateDir), `${connectionId}-${Date.now()}.jsonl`); + try { + fs.renameSync(from, to); + } catch (error) { + if (error.code === "ENOENT") return null; + throw error; + } + /* Codex PR #33 review "fsync both directories after quarantining a WAL": this rename IS + * the durable save point recovery-abandon reports success on -- it must both remove the + * connection from active/ and preserve its evidence under quarantine/. rename(2) only + * mutates two DIRECTORY ENTRIES, and neither is durable until its own parent directory's + * fd is fsynced (exactly the reason appendWalEvent/createIntentIfAbsent already fsync + * theirs). Without this, a power loss right after the command printed "quarantined" can + * resurrect the active/ entry or lose the quarantine/ one -- and by then abandonConnection + * has already deleted this connection's intent fences, so the resurrected WAL would + * re-enter startup recovery stripped of the very guards that were protecting it. Both + * directories, source and destination, through the same tolerated-codes fsyncDir helper + * hardened earlier in this PR. */ + fsyncDir(activeDir(stateDir)); + fsyncDir(quarantineDir(stateDir)); + return to; +} + +function deleteWal(stateDir, connectionId) { + try { + fs.unlinkSync(walPath(stateDir, connectionId)); + } catch (error) { + if (error.code !== "ENOENT") throw error; + } +} + +/** Reads and parses every valid line of a connection's WAL, stopping at the first + * malformed line rather than discarding the whole file -- a crash mid-`fs.writeSync` + * can only ever torn-write the LAST line still in flight (every earlier line already + * completed its own open/write/fsync/close cycle), so everything before the first bad + * line is real, durable, and safe to replay. Returns [] if no WAL exists for this + * connection (nothing was ever in flight, or it was already cleaned up). */ +function readWalEvents(stateDir, connectionId) { + let raw; + try { + raw = fs.readFileSync(walPath(stateDir, connectionId), "utf8"); + } catch (error) { + if (error.code === "ENOENT") return []; + throw fail(`Unreadable WAL for connection "${connectionId}": ${error.message}`, "GATEWAY_RECOVERY_WAL_UNREADABLE"); + } + const events = []; + for (const line of raw.split("\n")) { + if (line.length === 0) continue; + let event; + try { + event = JSON.parse(line); + } catch (error) { + break; // torn tail line from a crash mid-write: stop here, keep everything before it + } + events.push(event); + } + return events; +} + +/** Lists connectionIds with an active (not yet cleaned-up) WAL file -- the startup + * recovery scan's entry point (gateway.js#startGateway, before connectAllDownstreams). */ +function listActiveConnections(stateDir) { + let entries; + try { + entries = fs.readdirSync(activeDir(stateDir)); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } + /* Codex PR #33 review "sort active connections before recovery": readdirSync gives no + * portable ordering guarantee across platforms/filesystems -- gateway.js# + * recoverCrashedSessions consumes this array directly, in order, to decide chain-append + * sequence numbers and tail hashes for each recovered connection. An unsorted, platform- + * dependent order made re-running recovery on the same crash-left files potentially + * produce a different resulting chain across restarts/hosts. Sorting here (once, at the + * source) keeps every consumer's replay/append order deterministic. */ + return entries.filter((f) => f.endsWith(".jsonl")).map((f) => f.slice(0, -".jsonl".length)).sort(); +} + +/** Stable-key JSON stringify -- {a:1,b:2} and {b:2,a:1} must canonicalize identically so + * a retry with the same logical arguments in a different key order still hashes to the + * same intent key. Not a general-purpose canonical-JSON library: sufficient for hashing + * plain JSON-shaped tool arguments (objects/arrays/primitives), which is all a JSON-RPC + * `params.arguments` value can ever be. */ +function canonicalJson(value) { + if (value === null || typeof value !== "object") return JSON.stringify(value); + if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`; + const keys = Object.keys(value).sort(); + return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalJson(value[k])}`).join(",")}}`; +} + +/** The stable per-operation identity Finding 2 needs: same connection + tool + logical + * arguments always produces the same key regardless of the agent's own JSON-RPC id (a + * retry legitimately uses a NEW JSON-RPC id -- that id is a transport correlation + * concern, never an operation identity, per every reviewing model's own analysis). + * Deliberately scoped to the live connectionId, not a cross-session/cross-restart + * identity: an agent resubmitting "the same" call in a brand-new session is a new, + * intentional request from the gateway's point of view, not a retry of an old one -- + * cross-restart continuity for a call that was already in flight is handled by the WAL + * replay path instead (the intent file itself is what survives a restart, keyed the + * same way, so an in-flight call's fence is preserved across a crash even though a NEW + * connectionId after reconnect would not itself match the old key). */ +function computeIntentKey(connectionId, tool, args) { + const material = `${connectionId}\0${tool}\0${canonicalJson(args === undefined ? null : args)}`; + return "gs_" + crypto.createHash("sha256").update(material, "utf8").digest("hex").slice(0, 32); +} + +/** The signature identity Cluster A's cross-connection replay needs: (tool, logical + * arguments) alone, deliberately WITHOUT connectionId (unlike computeIntentKey above) -- + * see this file's own header comment on DEFAULT_RETAINED_SIGNATURE_TTL_MS for why. */ +function computeSignatureKey(tool, args, idempotencyKey) { + /* CodeRabbit PR #33 review "signature key collides across different idempotency + * keys for the same (tool, args)": two distinct caller-supplied idempotencyKeys + * calling the same tool with the same logical arguments previously hashed to the + * exact same signature, so the second caller's genuinely-distinct request would + * read back the first caller's cached result (or overwrite it) instead of being + * treated as its own operation. Folding idempotencyKey into the hashed material + * disambiguates them; omitting it (undefined/null) still hashes deterministically + * to the same key as before for callers that never supplied one. */ + const material = `${tool}\0${canonicalJson(args === undefined ? null : args)}\0${idempotencyKey === undefined || idempotencyKey === null ? "" : idempotencyKey}`; + return "gs_sig_" + crypto.createHash("sha256").update(material, "utf8").digest("hex").slice(0, 32); +} + +/** Retains (overwriting any prior record for this exact signature) the most recent + * proven-successful outcome for a (tool, arguments) signature, independent of which + * connection produced it -- called only from the same completion point that transitions + * an intent to "completed" (proxy.js's handleMessage). Unconditional overwrite is + * correct here: only ONE "most recent completed call for this signature" can ever be + * replayed at a time, and a newer completion is strictly more useful to a future + * reconnect than an older one. */ +function recordCompletedSignature(stateDir, signatureKey, data) { + ensureDir(signaturesDir(stateDir)); + const record = { schema_version: INTENT_SCHEMA_VERSION, signature_key: signatureKey, ...data }; + stateStore.atomicOverwriteFile(signaturePath(stateDir, signatureKey), JSON.stringify(record), signaturesDir(stateDir)); + fsyncDir(signaturesDir(stateDir)); + restrictFileMode(signaturePath(stateDir, signatureKey)); +} + +/** Reads a retained completed-call signature, honoring the bounded retention window -- + * an expired record is treated exactly as if it never existed (returns null) and is + * best-effort deleted so it stops taking up space, rather than kept around forever + * (this file's own header explains why an unbounded retention would be wrong: a stale + * "it once succeeded" fact should eventually stop being replayable at all). `ttlMs` + * defaults to DEFAULT_RETAINED_SIGNATURE_TTL_MS when not given. */ +function readCompletedSignature(stateDir, signatureKey, ttlMs) { + let raw; + try { + raw = fs.readFileSync(signaturePath(stateDir, signatureKey), "utf8"); + } catch (error) { + if (error.code === "ENOENT") return null; + throw fail(`Unreadable retained completed-call signature "${signatureKey}": ${error.message}`, "GATEWAY_RECOVERY_SIGNATURE_UNREADABLE"); + } + let record; + try { + record = JSON.parse(raw); + } catch (error) { + throw fail(`Corrupt retained completed-call signature "${signatureKey}": ${error.message}`, "GATEWAY_RECOVERY_SIGNATURE_CORRUPT"); + } + const effectiveTtl = typeof ttlMs === "number" ? ttlMs : DEFAULT_RETAINED_SIGNATURE_TTL_MS; + if (typeof record.completed_at !== "number" || Date.now() - record.completed_at > effectiveTtl) { + try { + fs.unlinkSync(signaturePath(stateDir, signatureKey)); + } catch (error) { + /* best effort -- see this function's own doc comment */ + } + return null; + } + return record; +} + +/* Codex PR #33 review "sweep expired completed-signature records" + CodeRabbit PR #33 + * review "sweep expired completed signatures" (the same finding, from both reviewers): + * readCompletedSignature above enforces DEFAULT_RETAINED_SIGNATURE_TTL_MS only for the + * one key being looked up, and recordCompletedSignature writes one file per DISTINCT + * (tool, arguments, idempotency key) signature. Calls with unique arguments are therefore + * never looked up again, so their records -- which hold the raw arguments and the raw + * cached result -- are never deleted by anything, and a long-running gateway accumulates + * them (and their inodes) without bound despite the advertised 24-hour retention. This is + * the global counterpart to that per-key check: one readdir of signatures/, unlink of + * every record already past its own TTL. + * + * Deliberately BOUNDED and best-effort, because its only caller is startup recovery, on + * the critical path to accepting traffic: + * - `maxEntries` caps how many directory entries one sweep will examine (default + * DEFAULT_SIGNATURE_SWEEP_MAX_ENTRIES). A directory bigger than that is simply swept + * across successive restarts rather than making one startup arbitrarily long -- + * progress is monotonic, since a swept file is gone and never reconsidered. + * - Expiry is decided by the record's own `completed_at`, the EXACT field + * readCompletedSignature already uses, so the two can never disagree about what + * "expired" means. + * - A record that cannot be read, parsed, or unlinked is SKIPPED, not fatal: this is + * space reclamation, never a correctness gate, and an unreadable signature is already + * handled (fail-closed) by readCompletedSignature on the dispatch path where it + * actually matters. A malformed record with no usable `completed_at` is treated as + * expired, matching readCompletedSignature's own identical rule. + * Returns { scanned, removed } so the caller can log what it reclaimed. */ +const DEFAULT_SIGNATURE_SWEEP_MAX_ENTRIES = 5000; + +function sweepExpiredSignatures(stateDir, options = {}) { + const ttlMs = typeof options.ttlMs === "number" ? options.ttlMs : DEFAULT_RETAINED_SIGNATURE_TTL_MS; + const maxEntries = typeof options.maxEntries === "number" ? options.maxEntries : DEFAULT_SIGNATURE_SWEEP_MAX_ENTRIES; + const now = typeof options.now === "function" ? options.now() : Date.now(); + const dir = signaturesDir(stateDir); + let entries; + try { + entries = fs.readdirSync(dir); + } catch (error) { + if (error.code === "ENOENT") return { scanned: 0, removed: 0 }; + throw error; + } + let scanned = 0; + let removed = 0; + for (const name of entries) { + if (scanned >= maxEntries) break; + if (!name.endsWith(".json")) continue; + scanned += 1; + const filePath = path.join(dir, name); + let expired; + try { + const record = JSON.parse(fs.readFileSync(filePath, "utf8")); + expired = typeof record.completed_at !== "number" || now - record.completed_at > ttlMs; + } catch (error) { + continue; // unreadable/corrupt: leave it for an operator, never delete on a guess + } + if (!expired) continue; + try { + fs.unlinkSync(filePath); + removed += 1; + } catch (error) { + /* best effort -- another process may have removed it, or it may be locked */ + } + } + if (removed > 0) fsyncDir(dir); + return { scanned, removed }; +} + +function readIntent(stateDir, intentKey) { + let raw; + try { + raw = fs.readFileSync(intentPath(stateDir, intentKey), "utf8"); + } catch (error) { + if (error.code === "ENOENT") return null; + throw fail(`Unreadable intent "${intentKey}": ${error.message}`, "GATEWAY_RECOVERY_INTENT_UNREADABLE"); + } + try { + return JSON.parse(raw); + } catch (error) { + throw fail(`Corrupt intent record "${intentKey}": ${error.message}`, "GATEWAY_RECOVERY_INTENT_CORRUPT"); + } +} + +/** Creates a new intent record IFF one does not already exist. Race-safe across + * concurrent async call sites (and, in principle, processes) because it relies on + * atomicCreateExclusive's O_EXCL semantics as the real guarantee -- the caller's own + * prior readIntent() check is an optimization, not the safety mechanism. Returns the + * created record, or null if one already existed (the caller must re-read to see its + * current state and respond consistently, rather than assume what it would have been). */ +function createIntentIfAbsent(stateDir, intentKey, data) { + ensureDir(intentsDir(stateDir)); + const record = { schema_version: INTENT_SCHEMA_VERSION, intent_key: intentKey, ...data }; + try { + stateStore.atomicCreateExclusive(intentPath(stateDir, intentKey), JSON.stringify(record)); + } catch (error) { + if (error.code === "EEXIST") return null; + throw error; + } + // See fsyncDir's own doc comment: atomicCreateExclusive (state-store.js, shared with + // callers outside this module) fsyncs the file's own content but not the directory + // entry that makes this new intent file discoverable after a crash -- done here, + // locally, rather than changing that shared helper's behavior for every other caller. + fsyncDir(intentsDir(stateDir)); + // Same "restrict permissions on raw recovery records" rationale as ensureDir/ + // appendDurableLine above -- state-store.js's shared atomicCreateExclusive is not + // itself changed (other callers outside this module rely on its current mode), so this + // module re-tightens its own files locally after the fact instead. + restrictFileMode(intentPath(stateDir, intentKey)); + return record; +} + +function updateIntent(stateDir, intentKey, patch) { + const current = readIntent(stateDir, intentKey); + if (!current) throw fail(`Cannot update unknown intent "${intentKey}"`, "GATEWAY_RECOVERY_INTENT_NOT_FOUND"); + const updated = { ...current, ...patch }; + stateStore.atomicOverwriteFile(intentPath(stateDir, intentKey), JSON.stringify(updated), intentsDir(stateDir)); + restrictFileMode(intentPath(stateDir, intentKey)); + return updated; +} + +function deleteIntent(stateDir, intentKey) { + try { + fs.unlinkSync(intentPath(stateDir, intentKey)); + } catch (error) { + if (error.code !== "ENOENT") throw error; + } + /* Codex PR #33 review "fsync the intent directory after rollback deletion": proxy.js's + * CALL_START-failure rollback (handleMessage) calls this to remove a just-created + * "dispatched" intent it now knows never actually dispatched, then tells the caller the + * retry is safe -- but the unlink's directory-entry removal was never itself fsynced. + * A crash/power-loss shortly after can leave the OLD directory entry (and therefore the + * stale "dispatched" record) resurrected on the next mount, permanently blocking every + * future retry of that same operation until an operator notices and cleans it up by + * hand. Same fsyncDir convention as createIntentIfAbsent's own directory-entry fsync + * above, applied here for the deletion side too. */ + fsyncDir(intentsDir(stateDir)); +} + +/** Lists every intent belonging to `connectionId` -- used to find ambiguous intents + * blocking a clean auto-seal at startup recovery, and to clean up terminal (`completed`) + * intents once a session is durably sealed. O(all intents on disk) per call: acceptable + * because intents are cleaned up promptly on a normal close (only orphaned `ambiguous` + * ones persist, which should be rare in practice), not indexed by connection -- a real + * scaling limit, disclosed rather than hidden, not worth a secondary index for v1. */ +function listIntentsForConnection(stateDir, connectionId) { + let entries; + try { + entries = fs.readdirSync(intentsDir(stateDir)); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } + const intents = []; + for (const f of entries) { + if (!f.endsWith(".json")) continue; + const record = readIntent(stateDir, f.slice(0, -".json".length)); + if (record && record.connection_id === connectionId) intents.push(record); + } + return intents; +} + +/** Lists every intent on disk regardless of connection -- used by the health surface + * (gateway.js#buildHealthStatus) to report ambiguous/dispatched intents an operator still + * needs to act on, across every connection, not just the one most recently recovered. + * Same O(all intents on disk) characteristic as listIntentsForConnection above, and for + * the same disclosed reason. */ +function listAllIntents(stateDir) { + let entries; + try { + entries = fs.readdirSync(intentsDir(stateDir)); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } + const intents = []; + for (const f of entries) { + if (!f.endsWith(".json")) continue; + const record = readIntent(stateDir, f.slice(0, -".json".length)); + if (record) intents.push(record); + } + return intents; +} + +/** Operator resolution for an `ambiguous` intent (never automatic -- see this module's + * header and KNOWN-LIMITATIONS.md): the operator is answering "did it actually execute?", + * not "should we allow a retry?" -- those are different questions and conflating them is + * exactly the mistake this module's stricter (dispatched -> completed | ambiguous, no + * "failed") state machine is designed to avoid (see option-c-hardened-design.md point 7a). */ +/* Codex PR #33 review "reject resolutions that overwrite terminal intents": both resolve* + * functions used to patch unconditionally regardless of the intent's CURRENT state -- a + * stale or repeated operator command (e.g. a shell history re-run, or two operators + * racing to resolve the same intent) could silently rewrite an already-`completed` + * intent's own observed result, or flip it to `not_executed`, discarding the original + * evidence recovery already signed into a sealed bundle for another connection's replay. + * Only a `dispatched`/`ambiguous` intent may transition; a repeat of the SAME terminal + * resolution is a harmless no-op (returns the existing record unchanged, no write), and + * any other terminal state is refused outright rather than silently overwritten. */ +function resolveIntentExecuted(stateDir, intentKey, result) { + const current = readIntent(stateDir, intentKey); + if (!current) throw fail(`Cannot update unknown intent "${intentKey}"`, "GATEWAY_RECOVERY_INTENT_NOT_FOUND"); + if (current.state === "completed") return current; // identical terminal resolution: no-op + if (current.state !== "dispatched" && current.state !== "ambiguous") { + throw fail( + `Cannot mark intent "${intentKey}" executed: it is already resolved as "${current.state}" -- refusing to overwrite a different terminal resolution with a new one.`, + "GATEWAY_RECOVERY_INTENT_TERMINAL" + ); + } + const resolvedAt = Date.now(); + const updated = updateIntent(stateDir, intentKey, { + state: "completed", + resolved_at: resolvedAt, + resolution: "operator_confirmed_executed", + cached_result: result, + }); + /* Codex PR #33 review "retain operator-confirmed results for reconnect replay": this + * used to update only the CONNECTION-SCOPED intent. Startup recovery then consumes that + * result and deletes the intent -- so, unlike proxy.js's live completion path, nothing + * ever wrote the connection-INDEPENDENT completed-signature record, and a reconnecting + * agent presenting the same caller idempotency key computed a brand-new intentKey, + * found nothing, and dispatched the already-executed side effect a second time. Write + * the same record the live path writes, from the same proven-executed outcome. + * + * Gated on a caller idempotency key actually being present, which is where this + * deliberately differs from proxy.js's unconditional write: proxy.js's own read gate + * (`retained.idempotency_key && retained.idempotency_key === callerIdempotencyKey`) + * can never replay an unkeyed record, so writing one here would only leave a file that + * is unreplayable by construction until its retention window expires. + * + * Best-effort, for the same reason the live path's own write is: the operator's + * decision is already durable in the intent record above, and this secondary + * cross-connection cache must not be able to undo or fail it. */ + if (typeof current.idempotency_key === "string" && current.idempotency_key.length > 0) { + try { + recordCompletedSignature(stateDir, computeSignatureKey(current.tool, current.arguments, current.idempotency_key), { + tool: current.tool, + arguments: current.arguments, + intent_key: intentKey, + connection_id: current.connection_id, + generation: current.generation, + idempotency_key: current.idempotency_key, + cached_result: result, + completed_at: resolvedAt, + }); + } catch (error) { + /* best effort -- see doc comment above */ + } + } + return updated; +} + +/** Confirmed NOT executed. Codex PR #33 review "persist a terminal not-executed + * resolution": this used to delete the intent immediately, on the reasoning that a + * future identical dispatch should be treated as brand new. That is correct for a LIVE + * connection, but recovery-resolve only ever runs while no gateway process holds the + * writer-claim for this state_dir (gateway.js#runRecoveryResolveCli now acquires it + * itself) -- so the connection this intent belongs to is always a crashed one, still + * sitting on a WAL with the same call's CALL_START still pending replay. Deleting the + * intent outright left recoverCrashedSessions with no record of the operator's decision + * on the next restart: it would find the same pending call, find no "completed" intent, + * and re-flag the same connection for operator review forever -- this CLI resolution + * could never actually finish recovery. Persisting a terminal `not_executed` state + * instead lets recoverCrashedSessions replay it as a real (failed) call result and then + * clean up the intent itself once the connection is actually sealed -- see + * gateway.js#recoverCrashedSessions's own handling of this state. */ +function resolveIntentNotExecuted(stateDir, intentKey) { + const current = readIntent(stateDir, intentKey); + if (!current) throw fail(`Cannot update unknown intent "${intentKey}"`, "GATEWAY_RECOVERY_INTENT_NOT_FOUND"); + if (current.state === "not_executed") return current; // identical terminal resolution: no-op + if (current.state !== "dispatched" && current.state !== "ambiguous") { + throw fail( + `Cannot mark intent "${intentKey}" not-executed: it is already resolved as "${current.state}" -- refusing to overwrite a different terminal resolution with a new one.`, + "GATEWAY_RECOVERY_INTENT_TERMINAL" + ); + } + return updateIntent(stateDir, intentKey, { + state: "not_executed", + resolved_at: Date.now(), + resolution: "operator_confirmed_not_executed", + }); +} + +module.exports = { + RECOVERY_DIRNAME, + DEFAULT_RETAINED_SIGNATURE_TTL_MS, + recoveryDir, + activeDir, + intentsDir, + quarantineDir, + signaturesDir, + walPath, + intentPath, + signaturePath, + appendWalEvent, + deleteWal, + quarantineWal, + readWalEvents, + listActiveConnections, + canonicalJson, + computeIntentKey, + computeSignatureKey, + recordCompletedSignature, + readCompletedSignature, + sweepExpiredSignatures, + DEFAULT_SIGNATURE_SWEEP_MAX_ENTRIES, + readIntent, + createIntentIfAbsent, + updateIntent, + deleteIntent, + listIntentsForConnection, + listAllIntents, + resolveIntentExecuted, + resolveIntentNotExecuted, +}; diff --git a/scripts/gateway/session.js b/scripts/gateway/session.js index e37afd1e..39f8759f 100644 --- a/scripts/gateway/session.js +++ b/scripts/gateway/session.js @@ -19,6 +19,7 @@ */ "use strict"; +const crypto = require("crypto"); const { sealBoundaryBundle } = require("../gsa-mcp-shim.js"); function fail(message, code = "GATEWAY_SESSION_ERROR") { @@ -27,13 +28,44 @@ function fail(message, code = "GATEWAY_SESSION_ERROR") { return error; } -/** Creates a fresh, empty in-memory session record (SS5.1's shape). */ +/** Creates a fresh, empty in-memory session record (SS5.1's shape). + * + * `options.sessionId` (Cluster B: session-identity distinctness in the audit trail -- + * see gsa-mcp-shim.js#sealBoundaryBundle's own doc comment on why bundle_id needs this): + * a permanent, unique identifier for this session, generated ONCE, folded into the + * sealed bundle's bundle_id so two sessions that happen to record identical + * {init, tools, calls} content are never treated as "the same session" just because a + * content-only hash cannot tell them apart. + * + * Three ways this argument is used, distinguished so replay never invents a new + * identity for an old session (which would break replay-idempotency -- the same + * session's own WAL replayed twice must produce the same bundle_id) while a genuinely + * new live session always gets a real one: + * - omitted entirely (proxy.js#openConnection, the live-dispatch path): a fresh + * `crypto.randomUUID()` is minted here, once, and the caller is expected to persist + * it (proxy.js writes it into the WAL's own SESSION_START event) so a later replay + * of this exact session reads the SAME id back rather than minting a new one. + * - a non-empty string (gateway.js#recoverCrashedSessions/abandonConnection replaying + * a WAL whose SESSION_START event already carries a `session_id`): reused verbatim, + * so replaying the same WAL twice yields the same session_id and therefore the same + * bundle_id. + * - explicitly `null`/absent-on-the-event (replaying an older WAL written before this + * field existed): stays `null` rather than randomly generated -- an old WAL replayed + * twice must still be idempotent, and inventing a random id here on every replay + * would break that for data that predates this fix. */ function createSession(connectionId, options = {}) { if (typeof connectionId !== "string" || connectionId.length === 0) { throw fail("connectionId must be a non-empty string", "INVALID_ARGUMENT"); } + const sessionId = + typeof options.sessionId === "string" && options.sessionId.length > 0 + ? options.sessionId + : options.sessionId === undefined + ? crypto.randomUUID() + : null; return { connectionId, + sessionId, initialize: null, tools: [], calls: [], @@ -43,6 +75,19 @@ function createSession(connectionId, options = {}) { startedAt: typeof options.now === "function" ? options.now() : Date.now(), finalized: false, nextCallSeq: 1, // monotonic per-session invocation counter; see recordCallStart. + /* Per docs/contracts/wal-append-failure-semantics.md (C1 SS2): the moment an + * appendWalEvent call fails for this connection, this is set to {reason, at} and + * every later append attempt on the same connection must be refused before it + * touches the filesystem -- a torn line from the first failure could otherwise be + * concatenated onto by a second attempt (see the contract doc). In-memory only, + * connection-scoped, never written to the sealed bundle (toSealableSession below + * does not include it) or the WAL itself; a fresh connection/session always starts + * with this null. Not read or written anywhere in this file -- callers (gateway.js's + * forwardDownstreamRequestToAgent, and every appendWalEvent call site in proxy.js as + * of the round-2 fix pass closing commit 5's enforcement gap) set and check it + * directly on the session record, the same way they already read/write + * pendingCalls/calls. */ + walPoisoned: null, /* Re-triage fix: running total of retained call payload bytes (request params + * response result, serialized -- see proxy.js's MAX_SESSION_CALL_BYTES for why), * updated alongside session.calls in recordCallResult/markPendingAsDisconnected. */ @@ -155,6 +200,16 @@ function recordCallResult(session, jsonRpcId, result) { model_call: pending.model_call, ts: pending.ts, seq: pending.seq, + /* Codex PR #33 review "persist cached replays in the session trace": an OPTIONAL, + * additive marker (absent on every ordinary call, exactly like `disconnected` above) + * letting proxy.js record an idempotency-key cache hit as a real, ordered step that + * is nonetheless not claimed to be a fresh downstream dispatch. Only the caller can + * know this -- session.js never decides it -- so it is passed in rather than derived, + * and toSealableSession spreads it conditionally the same way it already spreads the + * disconnect fields, so no ordinary session's sealed shape (or bundle_id) changes. */ + ...(result && result.replayed + ? { replayed: true, replayed_from_intent_key: result.replayedFromIntentKey || null } + : {}), }); /* Re-triage fix: MAX_SESSION_CALL_BYTES (proxy.js) needs a running total of what this * session has actually retained -- see payloadByteSize's own comment above. */ @@ -167,9 +222,61 @@ function recordCallResult(session, jsonRpcId, result) { * discipline as MAX_PENDING_CALLS_PER_SESSION/MAX_COMPLETED_CALLS_PER_SESSION (proxy.js). * Without one, a misbehaving or compromised downstream emitting a stream of responses * with unknown ids could grow this array (and the eventual sealed bundle) without bound, - * independently of both call-history caps -- neither of which this ever went through. */ + * independently of both call-history caps -- neither of which this ever went through. + * + * Cluster I (frontier-panel review, 2026-09-14, 5/5 unanimous): Option C's own + * recordAnomaly (below) originally bypassed this cap entirely on the theory that an + * internally-detected event (an ambiguous-retry-blocked attestation) is trusted where an + * externally-supplied one is not. The panel's unanimous view: internal detection doesn't + * make the FREQUENCY trusted -- an agent that can repeatedly trigger the internal + * condition (e.g. by retrying ambiguously) can still drive unbounded growth through an + * uncapped path, reopening the exact resource-exhaustion class this cap was added to + * close. Both anomaly-recording entry points below now share one capped, internal + * implementation so there is a single bound on session.anomalies regardless of which + * one records the event. */ const MAX_ANOMALIES_PER_SESSION = 1000; +/** Shared internal implementation for every anomaly-recording entry point in this file. + * Appends `anomaly` (stamped with ts unless the caller already supplied one) once + * session.anomalies is below MAX_ANOMALIES_PER_SESSION; once the cap is reached, further + * anomalies are dropped rather than grow session.anomalies without bound, but the cap + * being hit is itself recorded once (a single terminal marker entry) so it leaves a + * trace rather than silently truncating. Returns true if the anomaly was recorded + * (or the cap-reached marker was just appended), false if it was dropped. Not exported -- + * recordAnomaly and recordUnmatchedResponseAnomaly are the only external entry points, + * so no existing call site anywhere in this codebase needs to change. */ +function pushCappedAnomaly(session, anomaly) { + if (session.finalized) throw fail("Cannot record into a finalized session", "SESSION_FINALIZED"); + if (session.anomalies.length >= MAX_ANOMALIES_PER_SESSION) return false; + const entry = { ts: Date.now(), ...anomaly }; + session.anomalies.push(entry); + if (session.anomalies.length >= MAX_ANOMALIES_PER_SESSION) { + // Reuses entry.ts (an explicit historical ts if the caller supplied one, e.g. + // recordUnmatchedResponseAnomaly's ts param) rather than a fresh Date.now(), matching + // recordUnmatchedResponseAnomaly's original pre-merge behavior exactly. + session.anomalies.push({ + kind: "ANOMALY_CAP_REACHED", + detail: `This session reached the ${MAX_ANOMALIES_PER_SESSION}-anomaly cap -- further anomalies are dropped, not recorded.`, + ts: entry.ts, + }); + } + return true; +} + +/** Records a protocol-level irregularity that is not itself a call outcome -- generic + * and additive, mirroring recordCallResult's own UNMATCHED_RESPONSE anomaly shape so + * sealBoundaryBundle's existing anomaly handling needs no changes. Added for Option C + * (Codex PR #29 Finding 2, external-panel-reviewed design -- see + * option-c-hardened-design.md): records an ambiguous-retry-blocked event so the sealed + * bundle attests that a duplicate dispatch was prevented, not just that one happened to + * not occur. Routed through the shared, capped pushCappedAnomaly (Cluster I, + * frontier-panel review 2026-09-14) rather than appending directly -- see that helper's + * comment for why an internally-detected event still needs the same cap as an + * externally-supplied one. */ +function recordAnomaly(session, anomaly) { + return pushCappedAnomaly(session, anomaly); +} + /** Records a downstream response that could not be correlated to any pending call, * WITHOUT going through recordCallResult's agent-facing pendingCalls lookup (CodeRabbit * PR #29 review round 8 "record unmatched downstream responses without agent-call @@ -183,27 +290,16 @@ const MAX_ANOMALIES_PER_SESSION = 1000; * session trace and losing the UNMATCHED_RESPONSE anomaly entirely. This function only * ever appends the anomaly -- it never touches pendingCalls. * - * Capped at MAX_ANOMALIES_PER_SESSION (Codex PR #29 review round 8 "cap unmatched-response - * anomalies per session"): once reached, further anomalies are dropped rather than grow - * session.anomalies without bound, but the cap being hit is itself recorded once (a single - * terminal marker entry) so it leaves a trace rather than silently truncating. */ + * Capped at MAX_ANOMALIES_PER_SESSION via the shared pushCappedAnomaly helper (Codex + * PR #29 review round 8 "cap unmatched-response anomalies per session", generalized by + * Cluster I, frontier-panel review 2026-09-14). */ function recordUnmatchedResponseAnomaly(session, jsonRpcId, detail, ts) { - if (session.finalized) throw fail("Cannot record into a finalized session", "SESSION_FINALIZED"); - if (session.anomalies.length >= MAX_ANOMALIES_PER_SESSION) return false; - session.anomalies.push({ + return pushCappedAnomaly(session, { kind: "UNMATCHED_RESPONSE", jsonRpcId, detail: detail || "response arrived for a JSON-RPC id with no matching pending call", ts: ts !== undefined ? ts : Date.now(), }); - if (session.anomalies.length >= MAX_ANOMALIES_PER_SESSION) { - session.anomalies.push({ - kind: "ANOMALY_CAP_REACHED", - detail: `This session reached the ${MAX_ANOMALIES_PER_SESSION}-anomaly cap -- further anomalies are dropped, not recorded.`, - ts: ts !== undefined ? ts : Date.now(), - }); - } - return true; } /** Called when the downstream side of a connection disconnects (or the whole session is @@ -285,6 +381,7 @@ function markPendingAsDisconnected(session, reason, now, serverFilter, onDisconn function toSealableSession(session) { const orderedCalls = session.calls.slice().sort((a, b) => (a.seq || 0) - (b.seq || 0)); return { + session_id: session.sessionId || null, initialize: session.initialize || {}, tools: session.tools, calls: orderedCalls.map((c) => ({ @@ -303,6 +400,7 @@ function toSealableSession(session) { model_call: c.model_call, ts: c.ts, ...(c.disconnected ? { disconnected: true, disconnect_reason: c.disconnect_reason, jsonRpcId: c.jsonRpcId } : {}), + ...(c.replayed ? { replayed: true, replayed_from_intent_key: c.replayed_from_intent_key } : {}), })), goal: session.goal, anomalies: session.anomalies, @@ -348,6 +446,7 @@ module.exports = { recordToolsList, recordCallStart, recordCallResult, + recordAnomaly, recordUnmatchedResponseAnomaly, MAX_ANOMALIES_PER_SESSION, markPendingAsDisconnected, diff --git a/scripts/gateway/startup-permissions.js b/scripts/gateway/startup-permissions.js new file mode 100644 index 00000000..b50f69b8 --- /dev/null +++ b/scripts/gateway/startup-permissions.js @@ -0,0 +1,172 @@ +#!/usr/bin/env node +/* GraphSmith standalone gateway -- startup permission-tightening pass. + * + * Access model (Paul's decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model"): the gateway's + * state directories are group-readable (0750 dirs / 0640 files), NOT single-user + * (0700/0600) -- other-uid ops tooling must keep working against them. The one named + * exception is gateway.js's operator-facing gateway-status.json health file, which stays + * at its current, operator-readable mode (see that file's own writeStatusFile comment) -- + * it is deliberately never listed below. + * + * WHY A SEPARATE STARTUP PASS, not just a creation-mode fix. + * + * scripts/state-store.js's atomicCreateExclusive/atomicOverwriteFile now default every + * NEW file they create to 0640 (see that file's own doc comments), which protects + * anything written fresh from here on. That alone protects nothing that ALREADY EXISTS + * on disk from a prior run or an older build, and it does not even cover every future + * write: several of the files this pass targets (chain.jsonl, the writer-claim's own + * renew()) are append-only or rewritten-in-place through a raw fs.openSync/writeSync + * pair, never through either primitive, precisely because they are long-lived records + * whose IDENTITY (not just their content) must survive across appends -- a + * creation-mode-only fix would protect the file's very first byte and nothing an + * operator would actually be worried about leaking. Retrofitting the mode explicitly, + * once, at startup, covers both gaps: the file that already existed yesterday, and the + * file whose own write path was never going to call either primitive to begin with. + * + * WHAT IS COVERED, and why each one: + * - the writer-claim file (writer-claim.js's CLAIM_FILE) -- the single-writer + * attestation record for this state directory. + * - chain.jsonl / HEAD.json (chain.js) -- the signed session hash-chain and its tail + * pointer. + * - every sealed session bundle in gateway-sessions/*.json (chain.js's own + * `.json` files) -- raw tool arguments/results the signed chain otherwise + * only ever stores as a hash. + * - every quarantined sealed bundle in gateway-sessions/quarantine/*.json (proxy.js's + * quarantineSealedBundle) -- a full, unsealed copy of the same raw session data, + * written by a RAW fs.writeFileSync that never goes through either shared primitive + * at all, so it is exactly the "creation-mode-only fix protects nothing" case this + * pass exists for. + * - every WAL file in gateway-recovery/active/*.jsonl (recovery.js) -- an in-flight + * session's raw, unsigned event log. + * - every quarantined WAL in gateway-recovery/quarantine/*.jsonl (recovery.js's + * quarantineWal) -- the same raw event log, moved rather than deleted when an + * operator gives up on recovering it. + * - every intent/signature record in gateway-recovery/intents/*.json and + * gateway-recovery/completed-signatures/*.json (recovery.js) -- raw tool arguments + * and cached results kept for crash-recovery idempotency. + * Deliberately NOT covered: scripts/state-store.js's own general state files + * (window.json, run-registry.jsonl, the state.lock, ...) -- a different subsystem, used + * far outside the gateway, with its own existing convention and out of this item's + * scope. mode-selection.js's per-deployment secret key already chmods itself to 0600 + * (stricter than 0640) and is left untouched here too, for the same reason. + * + * WHY DIRECTORIES ARE INCLUDED TOO (0750): a file's own mode is moot if the directory + * that contains it is still world/group-executable-and-readable in a way that lets + * another local user list its contents or open it by name under a permissive umask -- + * mkdirSync's default mode (0755 under a common 022 umask) does exactly that, and + * `{ recursive: true }` never revisits a directory that already existed before this fix. + * + * Best-effort per path, but NOT best-effort on a real fault: a missing file/directory + * (a fresh deployment that has never had a crash, or never had one recovered/sealed yet) + * is skipped silently via stateStore.chmodPathOrFail's own ENOENT handling -- that is an + * entirely ordinary, expected state, not a fault. A genuine EPERM/EIO chmod failure is + * NOT swallowed: several of these files can hold a bearer credential or a full raw + * session bundle, so a failed permission-tightening attempt on one of them is an + * operational error this process should refuse to start past, not a logged nitpick. + */ +"use strict"; + +const fs = require("fs"); +const path = require("path"); + +const stateStore = require("../state-store.js"); +const recovery = require("./recovery.js"); +const chain = require("./chain.js"); +const writerClaimModule = require("../writer-claim.js"); + +const FILE_MODE = stateStore.DEFAULT_FILE_MODE; // 0o640 +const DIR_MODE = stateStore.DEFAULT_DIR_MODE; // 0o750 + +/* proxy.js's quarantineSealedBundle writes into /quarantine/ -- a + * literal here, not an import from proxy.js, so this module (loaded very early at + * startup, before proxy.js's own dependency graph is otherwise needed) stays a one-way + * dependency on chain.js/recovery.js/writer-claim.js rather than adding a reverse edge + * from a permissions module back into the request-dispatch module. Kept in sync with + * proxy.js's own literal by the dedicated cross-check test in this module's own suite. */ +const SESSION_QUARANTINE_DIRNAME = "quarantine"; +function sessionQuarantineDir(stateDir) { + return path.join(chain.sessionsDir(stateDir), SESSION_QUARANTINE_DIRNAME); +} + +/* Lists the full paths of every entry directly inside `dir` whose name ends with + * `suffix`. Returns [] for a directory that does not exist yet (nothing to protect); + * any other readdir failure propagates -- an unreadable-but-present sensitive directory + * is exactly the kind of real fault this pass must not paper over. */ +function listMatchingFiles(dir, suffix) { + let entries; + try { + entries = fs.readdirSync(dir); + } catch (error) { + if (error.code === "ENOENT") return []; + throw error; + } + return entries.filter((name) => name.endsWith(suffix)).map((name) => path.join(dir, name)); +} + +/** Every directory this pass tightens to DIR_MODE (0750), in an order that only matters + * for readability (parents before children) -- chmod does not require a particular + * order, and a missing entry anywhere in this list is simply skipped. */ +function sensitiveDirectories(stateDir) { + return [ + stateDir, + chain.sessionsDir(stateDir), + sessionQuarantineDir(stateDir), + recovery.recoveryDir(stateDir), + recovery.activeDir(stateDir), + recovery.intentsDir(stateDir), + recovery.signaturesDir(stateDir), + ]; +} + +/** Every individual file this pass tightens to FILE_MODE (0640) -- the fixed, + * known-by-name files (the claim, chain.jsonl, HEAD.json) plus every CURRENT entry in + * each of the per-record directories above, discovered fresh each call rather than + * cached, since this only ever runs once at startup before any of those directories' + * contents can have changed under it. */ +function sensitiveFiles(stateDir) { + return [ + writerClaimModule.claimPath(stateDir), + chain.chainPath(stateDir), + chain.headPath(stateDir), + ...listMatchingFiles(chain.sessionsDir(stateDir), ".json").filter( + (file) => path.basename(file) !== chain.HEAD_FILE + ), + ...listMatchingFiles(sessionQuarantineDir(stateDir), ".json"), + ...listMatchingFiles(recovery.activeDir(stateDir), ".jsonl"), + ...listMatchingFiles(recovery.intentsDir(stateDir), ".json"), + ...listMatchingFiles(recovery.signaturesDir(stateDir), ".json"), + ]; +} + +/** Re-chmods every known-sensitive EXISTING file/directory under `stateDir` to this + * codebase's group-readable access model. Called once, synchronously, at gateway + * startup -- see this module's own header for the full rationale and the exact, + * deliberately-bounded list of what is and is not covered. + * + * Returns `{ dirs, files }`, the paths actually changed (a path that did not exist yet + * is simply absent from both arrays, not an error). Throws on the FIRST real + * permission fault (stateStore.chmodPathOrFail's own STATE_STORE_CHMOD_FAILED) -- + * callers (gateway.js#startGateway) let that abort startup rather than proceed with a + * sensitive file left more permissive than this access model requires. */ +function tightenStateDirPermissions(stateDir) { + const changed = { dirs: [], files: [] }; + + for (const dir of sensitiveDirectories(stateDir)) { + if (stateStore.chmodPathOrFail(dir, DIR_MODE)) changed.dirs.push(dir); + } + for (const file of sensitiveFiles(stateDir)) { + if (stateStore.chmodPathOrFail(file, FILE_MODE)) changed.files.push(file); + } + + return changed; +} + +module.exports = { + FILE_MODE, + DIR_MODE, + sessionQuarantineDir, + sensitiveDirectories, + sensitiveFiles, + tightenStateDirPermissions, +}; diff --git a/scripts/gsa-mcp-shim.js b/scripts/gsa-mcp-shim.js index 3cd40c71..ef12792e 100644 --- a/scripts/gsa-mcp-shim.js +++ b/scripts/gsa-mcp-shim.js @@ -68,6 +68,18 @@ function sealBoundaryBundle(session, keys) { is_error: !!c.isError, model_call: !!c.model_call, // sampling/createMessage → non-deterministic ...(c.disconnected ? { disconnected: true, disconnect_reason: c.disconnect_reason || null } : {}), + /* Round-N fix (frontier-panel finding #2): session.js's own per-call recording + * (toSealableSession / the earlier per-call `replayed` bookkeeping) already + * propagates `replayed`/`replayed_from_intent_key` correctly onto each call object + * -- this sealed execution_trace.jsonl line was the one place downstream that + * silently dropped it again, by mirroring the `disconnected` conditional spread + * immediately above without also carrying this one. Without it, a replayed call + * (the gateway detected a duplicate intent_key and returned the previously + * recorded result instead of re-dispatching) is indistinguishable in the signed, + * durable attestation bundle from an ordinary fresh call -- an auditor reviewing + * the bundle alone has no way to know that call's own downstream effect did not + * happen again at that step. */ + ...(c.replayed ? { replayed: true, replayed_from_intent_key: c.replayed_from_intent_key || null } : {}), ...(c.jsonRpcId !== undefined ? { jsonRpcId: c.jsonRpcId } : {}), /* PR #29 review thread 4000335147, frontier-panel review (4/4 converged) "flag * structurally malformed tools/call results distinctly from tool-level errors": @@ -89,10 +101,24 @@ function sealBoundaryBundle(session, keys) { }); const outputs = calls.filter((c) => !c.isError).map((c, i) => ({ call: i + 1, result_sha256: sha256Hex(JSON.stringify(c.result === undefined ? null : c.result)) })); + /* Cluster B (session-identity distinctness in the audit trail): `session.session_id` + * is an optional, additive field (scripts/gateway/session.js#createSession -- a + * permanent id minted once per session and, for the gateway's own recovery WAL, + * persisted and read back on replay rather than regenerated). Folded into bundle_id + * alongside the existing content fields so two sessions that happen to record + * identical {init, grantedTools, call count} are never collapsed into the same + * bundle_id just because this hash cannot otherwise tell them apart -- without this, a + * second, genuinely distinct session with byte-identical recorded content was + * indistinguishable from a replay of the first one. Defaults to `null` when absent + * (every non-gateway caller of sealBoundaryBundle, and any session recorded before + * this field existed) so bundle_id stays exactly as deterministic as before for those + * callers -- only ever the SAME session replayed twice (session_id read back from its + * own persisted record, not regenerated) collides with itself, which is correct. */ + const sessionId = typeof session.session_id === "string" && session.session_id.length > 0 ? session.session_id : null; const BOUNDARY = "boundary-only: the MCP shim observes tool I/O, not the agent's plan. Not emitted by the agent."; const A = (path, body) => ({ path, body }); const run = { - bundle_id: "gsa-" + sha256Hex(JSON.stringify({ init, grantedTools, n: calls.length })).slice(0, 16), + bundle_id: "gsa-" + sha256Hex(JSON.stringify({ init, grantedTools, n: calls.length, session_id: sessionId })).slice(0, 16), mode: "standard", profiles: ["A"], // boundary tier earns A only, never full profiles producer: { name: "gsa-mcp-shim", version: "0.3.0" }, diff --git a/scripts/state-store.js b/scripts/state-store.js index bd323bb5..ed156832 100644 --- a/scripts/state-store.js +++ b/scripts/state-store.js @@ -186,6 +186,89 @@ function clone(value) { * .plans/v0.5.0/GATEWAY-MULTI-INSTANCE-HANDOFF.md) reuses the same two functions rather * than inventing new ones, per that plan's NFR-2 (no new atomic-write primitive). */ +/* Access-model defaults (Paul's decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model"): group-readable + * (0750 dirs / 0640 files), NOT single-user (0700/0600) -- other-uid ops tooling must + * keep working against these directories. Every NEW file either of the two primitives + * below creates defaults to DEFAULT_FILE_MODE from here on; a caller that genuinely + * needs a different mode (or, for the one named exception -- gateway.js's + * operator-facing gateway-status.json health file, which must NOT be narrowed by this + * change -- no mode enforcement at all) passes `{ mode }` explicitly. `DEFAULT_DIR_MODE` + * has no analogous enforcement point in these two functions (neither creates a + * directory), but is exported so the gateway startup permission-tightening pass + * (scripts/gateway/startup-permissions.js) and this same file's own directory creation + * use the identical constant rather than a second, driftable copy of the number. */ +const DEFAULT_FILE_MODE = 0o640; +const DEFAULT_DIR_MODE = 0o750; + +/* Applies `mode` to an already-open file descriptor via fchmodSync, which -- unlike the + * mode argument to fs.openSync/fs.mkdirSync -- is never masked by the process umask. That + * distinction is the entire point: a mode merely PASSED to open() only produces the + * requested bits under a permissive umask (see tests/state-store/atomic-primitives' + * `-umask-0-` case, which proves this call is doing real work rather than the ambient + * umask happening to already produce 0640), and it does nothing at all for + * atomicOverwriteFile's rename-into-place, which inherits whatever mode the file HAD + * BEFORE the overwrite unless something explicitly sets the new inode's mode first (see + * that suite's `-umask-022-` case, over a pre-existing 0644 target). + * + * `mode === null` is a deliberate, explicit opt-out -- not "forgot to pass one". The one + * caller that must NOT be tightened by this change (gateway.js's `writeStatusFile`, + * writing the operator-facing gateway-status.json health file -- Paul's decision: it + * stays at its current, operator-readable mode, "do not tighten it") passes `{ mode: + * null }` to keep the pre-existing, umask-governed default completely untouched rather + * than guessing a replacement literal mode that might not match every deployment's own + * umask. + * + * A real chmod failure (EPERM, EIO, ...) is surfaced, not swallowed: these primitives + * back the writer-claim, the signed session chain, and the crash-recovery WAL/intent + * store, several of which can hold a bearer credential or a full raw session bundle -- a + * failed permission-tightening attempt on one of those is an operational fault, not a + * cosmetic nitpick. The one exception, matching this codebase's own existing convention + * (see mode-selection.js's ensureSecret, which draws exactly this line already) is a + * platform with no POSIX mode bits to set at all (win32): there is nothing to fail at, + * so that case is skipped silently rather than reported as an error. */ +function applyFileMode(fd, mode, targetPath) { + if (mode === null || mode === undefined) return; + try { + fs.fchmodSync(fd, mode); + } catch (error) { + if (process.platform === "win32") return; // no POSIX mode bits here: nothing to do + throw fail( + `Failed to set permissions (mode ${mode.toString(8)}) on ${targetPath}: ${error.message}. ` + + "This file can hold sensitive material (a bearer credential, raw tool arguments, or a " + + "signed session bundle) -- a failed permission-tightening attempt is treated as an " + + "operational error rather than silently ignored.", + "STATE_STORE_CHMOD_FAILED" + ); + } +} + +/* Same real-error-vs-unsupported-platform distinction as applyFileMode above, for a + * caller that already has a PATH on disk rather than an open fd -- the gateway startup + * permission-tightening pass (scripts/gateway/startup-permissions.js), which re-chmods + * files/directories that ALREADY EXIST (created by an older build, or before this fix), + * rather than creating new ones. A missing path (ENOENT) is treated as "nothing to do + * yet" rather than a fault -- a fresh deployment legitimately has no chain/recovery + * directories yet the first time this runs. Returns true if the mode was actually + * applied, false if the path was skipped (missing, or no POSIX mode bits on this + * platform) -- never swallows an EPERM/EIO the caller needs to know about. */ +function chmodPathOrFail(targetPath, mode) { + try { + fs.chmodSync(targetPath, mode); + return true; + } catch (error) { + if (error.code === "ENOENT") return false; // nothing there yet: not a fault + if (process.platform === "win32") return false; // no POSIX mode bits: nothing to do + throw fail( + `Failed to set permissions (mode ${mode.toString(8)}) on ${targetPath}: ${error.message}. ` + + "This path can hold sensitive material (a bearer credential, raw tool arguments, or a " + + "signed session bundle/chain record) -- a failed permission-tightening attempt is " + + "surfaced as an operational error rather than silently ignored.", + "STATE_STORE_CHMOD_FAILED" + ); + } +} + // The file must never be OBSERVABLE in a half-formed state. The historical // `openSync(target, "wx")` + `writeSync` pair made the file exist before its content // did, so a competing reader that hit EEXIST and read it saw "" (or a truncated @@ -194,11 +277,65 @@ function clone(value) { // exactly as "wx" did -- while making the record atomically visible, fully formed. // Filesystems without hard links fall back to the old two-step create; the caller's // bounded retry loop covers that residual window. -function atomicCreateExclusive(targetPath, payload) { +// +// `options.mode` (default DEFAULT_FILE_MODE, 0640) is applied via fchmodSync to the +// TEMPORARY file before it is linked/renamed into place -- a hard link shares the +// source's inode (and therefore its mode) with the target, so setting it on the +// temporary file is sufficient for both the link path and the no-hard-link fallback +// below. Pass `{ mode: null }` to opt out entirely (see applyFileMode's own doc comment). +/* Codex PR #33 review "write the entire intent record before dispatching": fs.writeSync + * is permitted by Node's own docs to write FEWER bytes than requested in a single call. + * Both atomic primitives below previously made exactly ONE writeSync call and then + * fsync'd and linked/renamed the result into place, so a short write PUBLISHED a + * truncated file while still returning success to the caller. For the gateway's + * recovery.js#createIntentIfAbsent that is the worst possible shape: the function returns + * a valid-looking intent record and proxy.js dispatches the downstream side effect, while + * the on-disk fence that is supposed to make that dispatch replay-safe is invalid JSON -- + * a later crash or completion update cannot use it and forces ambiguous operator + * recovery. Loops until every byte has actually been written, mirroring + * gateway/recovery.js#writeFullySync's own established pattern for the WAL. + * + * No truncate-on-failure rollback is needed here (unlike writeFullySync, which appends + * in place to a live WAL): both call sites below write into a freshly-created file that + * is either a temporary never linked into place, or -- on the no-hard-link fallback + * path -- a target the caller must treat as failed anyway, so a throw before fsync can + * never publish a partial record through either primitive's success path. */ +function writeFullySync(fd, payload) { + const buffer = Buffer.isBuffer(payload) ? payload : Buffer.from(String(payload), "utf8"); + let offset = 0; + while (offset < buffer.length) { + const written = fs.writeSync(fd, buffer, offset, buffer.length - offset); + if (!(written > 0)) { + throw fail( + `fs.writeSync made no progress (wrote ${written} of ${buffer.length - offset} remaining byte(s)) -- ` + + "refusing to publish a possibly-incomplete record.", + "STATE_STORE_SHORT_WRITE" + ); + } + offset += written; + } +} + +function atomicCreateExclusive(targetPath, payload, options = {}) { + const mode = Object.prototype.hasOwnProperty.call(options, "mode") ? options.mode : DEFAULT_FILE_MODE; const temporary = `${targetPath}.new-${process.pid}-${crypto.randomBytes(8).toString("hex")}`; const writeTo = (target, flag) => { - const fd = fs.openSync(target, flag); - try { fs.writeSync(fd, payload); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } + /* CodeRabbit PR #33 review round-4 follow-up "close the no-hard-link fallback's + * creation-time exposure window": the WAL half of this finding was closed by passing + * DEFAULT_FILE_MODE to fs.openSync's creation-time mode argument (see recovery.js's + * appendDurableLine, commit 7932754) -- this primitive still opened both the + * temporary file AND (on filesystems without hard-link support) the FINAL targetPath + * itself with the umask-derived default mode, relying solely on the fchmodSync below + * to narrow it afterward. For the temporary file that gap is latent (its name is + * unguessable), but for the no-hard-link fallback -- which writes directly to the + * predictable, discoverable targetPath -- it is a real window during which another + * local user could read a freshly-created intent/WAL record before applyFileMode + * narrows it. Pass mode to fs.openSync itself so the file never has a wider mode than + * intended, even momentarily; applyFileMode is kept unconditionally as belt-and-braces + * (this file's own established defense-in-depth pattern) and as the sole enforcement + * point for a caller that opts out via {mode: null}. */ + const fd = fs.openSync(target, flag, mode === null || mode === undefined ? undefined : mode); + try { writeFullySync(fd, payload); applyFileMode(fd, mode, target); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } }; writeTo(temporary, "wx"); try { @@ -217,11 +354,23 @@ function atomicCreateExclusive(targetPath, payload) { // Unconditional atomic overwrite (temp file + fsync + rename + best-effort directory // fsync) for a caller that already holds verified exclusive ownership of `targetPath` // and needs to update the record's content in place. -function atomicOverwriteFile(targetPath, content, dirPath) { +// +// `options.mode` (default DEFAULT_FILE_MODE, 0640) is applied via fchmodSync to the +// TEMPORARY file before the rename -- rename(2) replaces the target's directory entry +// with the temporary file's own inode, so the target ends up with whatever mode the +// temporary file had, REGARDLESS of what mode a pre-existing target used to have (see +// this file's own atomic-primitives test suite for a case pinning exactly that: a +// pre-existing 0644 target overwritten with the default options ends up 0640, not +// 0644). Pass `{ mode: null }` to opt out entirely and keep the pre-existing, +// umask-governed default untouched (see applyFileMode's own doc comment for why +// gateway.js's operator-facing status file needs exactly that). +function atomicOverwriteFile(targetPath, content, dirPath, options = {}) { + const mode = Object.prototype.hasOwnProperty.call(options, "mode") ? options.mode : DEFAULT_FILE_MODE; const temporary = `${targetPath}.tmp-${process.pid}-${crypto.randomBytes(8).toString("hex")}`; const fd = fs.openSync(temporary, "wx"); try { - fs.writeSync(fd, content); + writeFullySync(fd, content); + applyFileMode(fd, mode, temporary); fs.fsyncSync(fd); } finally { fs.closeSync(fd); } try { @@ -1570,12 +1719,23 @@ api.systemLeaseClock = systemLeaseClock; /* Exported so writer-claim.js (a separate, standing process-lifetime claim mechanism -- * see .plans/v0.5.0/GATEWAY-MULTI-INSTANCE-HANDOFF.md FR-1..FR-4) reuses these SAME * primitives rather than a second near-identical copy, per that plan's NFR-2. */ +api.writeFullySync = writeFullySync; api.atomicCreateExclusive = atomicCreateExclusive; api.atomicOverwriteFile = atomicOverwriteFile; api.pidAlive = pidAlive; api.validateNamedRecord = validateNamedRecord; api.recordLeaseClockConstruction = recordLeaseClockConstruction; +/* Access-model constants/helpers (see this file's own doc comments above + * atomicCreateExclusive/atomicOverwriteFile): exported so the gateway startup + * permission-tightening pass (scripts/gateway/startup-permissions.js) re-chmods + * EXISTING sensitive files/directories to the identical target mode these two + * primitives now default NEW files to, rather than a second, driftable copy of the + * numbers or the real-error-vs-unsupported-platform distinction. */ +api.DEFAULT_FILE_MODE = DEFAULT_FILE_MODE; +api.DEFAULT_DIR_MODE = DEFAULT_DIR_MODE; +api.chmodPathOrFail = chmodPathOrFail; + module.exports = api; function sleep(ms) { diff --git a/scripts/writer-claim.js b/scripts/writer-claim.js index 3dfad778..6eac8d8f 100644 --- a/scripts/writer-claim.js +++ b/scripts/writer-claim.js @@ -269,6 +269,10 @@ class WriterClaim { this._claimToken = null; this._timer = null; this._lastHeartbeatError = null; + /* Set by acquire() and renew() (never by maybeRenew() directly -- it only reads + * this to decide whether to call renew()). Used by maybeRenew() below to time-gate + * renewal for callers doing long synchronous work while holding this claim. */ + this._lastRenewAt = null; // Optional: see startHeartbeat()'s doc comment. Also settable after construction. this.onClaimLost = typeof options.onClaimLost === "function" ? options.onClaimLost : null; } @@ -350,6 +354,7 @@ class WriterClaim { try { const { token } = this._writeNewClaim(now); this._claimToken = token; + this._lastRenewAt = now; return this.status(); } catch (error) { if (error.code !== "EEXIST") throw error; @@ -378,6 +383,7 @@ class WriterClaim { }); if (decision.outcome === "own") { this._claimToken = observed.claim_token; + this._lastRenewAt = now; return this.status(); } if (decision.outcome === "refuse") { @@ -457,12 +463,56 @@ class WriterClaim { fs.ftruncateSync(fd, 0); fs.writeSync(fd, content, 0, "utf8"); fs.fsyncSync(fd); + this._lastRenewAt = now; return updated; } finally { fs.closeSync(fd); } } + /* Round-1 fix-plan item 1 (lease keepalive, generalized): a time-gated renew() for a + * caller doing long synchronous work while holding this claim -- WAL replay loops, the + * startup chain-integrity walk, `chain.reconcileHead`'s per-entry walk (the last two + * land in later commits and will call this same method; it is written now so they + * have it to call). startHeartbeat()'s own doc comment above discloses exactly why a + * bare setInterval is not sufficient on its own: it only fires when the event loop is + * free, and a caller doing synchronous fs work across many small iterations can starve + * it past staleAfterMs before a single renewal lands. Calling renew() unconditionally + * on every iteration would fix that but adds an fsync per iteration -- for a WAL replay + * loop over thousands of events that makes the cure slower than the disease. This + * renews only once at least heartbeatMs has elapsed since the LAST successful renewal + * from ANY source sharing this same WriterClaim instance -- acquire(), the background + * heartbeat, a previous maybeRenew(), or a previous direct renew() call, all update the + * same `_lastRenewAt` -- so a caller with many small iterations pays at most roughly one + * renewal per heartbeatMs, not one per iteration. + * + * Uses this.clock.now() -- NEVER Date.now() -- for the elapsed-time check, the same + * instance-level clock renew()/acquire() already use and that this class's constructor + * enforces via LEASE_CLOCK_REQUIRED/BAD_LEASE_CLOCK (see above): a caller under test can + * drive this deterministically with a manual clock (tests/_harness/clock.js), and a + * caller mixing in Date.now() here would silently defeat that for exactly the code path + * meant to be pinned by a chosen instant. + * + * A caller that knows a SINGLE unit of work can itself exceed staleAfterMs (a single + * oversized WAL event, or -- more importantly -- the actual write that mutates chain + * state, e.g. immediately before chain.appendSession/chain.repairMissingChainEntry) + * must call renew() directly and unconditionally instead of relying on this time gate; + * see gateway.js's own call sites for exactly that case. This method's own gate is a + * per-iteration convenience, not a substitute for that unconditional pre-write renewal. + * + * DISCLOSED, NOT FIXED HERE: neither this method nor renew() provides atomic fencing. + * A successful renewal and the shared-state write it is meant to protect remain two + * separate operations with no OS-level atomicity between them -- a contender could in + * principle take the claim in the gap between this call returning and the next line of + * caller code running. This narrows the exposure window from staleAfterMs down to + * roughly one iteration's (or one write's) worth of work; it does not close it. Do not + * describe this, in a comment, a commit message, or to an operator, as "preventing + * corruption" -- say "narrows the window" instead. */ + maybeRenew() { + if (this._lastRenewAt !== null && this._now() - this._lastRenewAt < this.heartbeatMs) return; + this.renew(); + } + /* CodeRabbit review (PR #27) found two problems here, both left the module silently * pretending to be the sole attesting writer after it no longer was: * @@ -481,27 +531,37 @@ class WriterClaim { * long synchronous work while holding this claim can starve the heartbeat past * staleAfterMs, exactly the failure mode state-store.js's own lock renewal * (_assertStillOwned) was hardened against by making renewal synchronous at - * durable writes. WriterClaim has NOT had the equivalent fix applied, because - * Phase 0 does not yet wire this claim into any long-running synchronous - * operation (no production entry point calls acquire()/startHeartbeat() as of - * this PR -- see KNOWN-LIMITATIONS.md). This is disclosed here rather than - * silently left implicit: whichever change wires WriterClaim into a real process - * lifecycle must either keep every synchronous critical section well under - * staleAfterMs, or add a synchronous renewal call at the boundaries of any long - * synchronous phase, mirroring state-store.js's fix, before relying on this - * heartbeat alone for correctness. */ + * durable writes. Round-1 fix-plan item 1 (lease keepalive, generalized) is that + * equivalent fix: gateway.js now wires acquire()/startHeartbeat() into the real + * process lifecycle immediately after acquire() succeeds (before any startup work + * runs), and calls this class's own maybeRenew()/renew() at the boundaries of every + * long synchronous phase held under this claim (WAL replay loops, immediately + * before every chain-mutating write) -- see gateway.js's recoverCrashedSessions/ + * abandonConnection and their shared createLeaseGuard() helper. + * + * Round-1 fix-plan item 1 also corrects a narrower bug in this method: previously only + * `error.code === "WRITER_CLAIM_LOST"` halted the heartbeat -- any OTHER renew() + * failure (a raw fs error such as EIO from the openSync in renew() above, or + * CORRUPT_CLAIM) fell through this check silently, leaving `_lastHeartbeatError` set + * but the timer running, unobserved except by a caller that happens to poll status(). + * A renewal failure for ANY reason means this process can no longer be certain it is + * the sole attesting writer -- lease ownership is process-global, so treating only the + * one named code as fatal while quietly retrying every other failure is exactly the + * "silently continue after losing exclusivity" bug convention this codebase otherwise + * refuses to allow (see the halt-loudly convention cited above). Every renew() failure + * now halts the heartbeat and either reports via onClaimLost or rethrows uncaught. */ startHeartbeat() { if (this._timer) return this._timer; this._timer = setInterval(() => { try { this.renew(); this._lastHeartbeatError = null; } catch (error) { this._lastHeartbeatError = error; - if (error.code === "WRITER_CLAIM_LOST") { - this.stopHeartbeat(); - this._claimToken = null; - if (typeof this.onClaimLost === "function") this.onClaimLost(error); - else throw error; // unhandled: halt loudly rather than keep writing - } + // Any renew() failure is fatal to the heartbeat, not only WRITER_CLAIM_LOST -- + // see this method's own doc comment above. + this.stopHeartbeat(); + this._claimToken = null; + if (typeof this.onClaimLost === "function") this.onClaimLost(error); + else throw error; // unhandled: halt loudly rather than keep writing } }, this.heartbeatMs); if (typeof this._timer.unref === "function") this._timer.unref(); diff --git a/tests/gateway/chain/run-tests.js b/tests/gateway/chain/run-tests.js index 90934d5c..82b80977 100644 --- a/tests/gateway/chain/run-tests.js +++ b/tests/gateway/chain/run-tests.js @@ -116,9 +116,24 @@ function incompleteAppendDetected() { fs.appendFileSync(chain.chainPath(dir), JSON.stringify(entry2) + "\n"); // HEAD.json still points at entry1 -- exactly the "stale head" incomplete-append shape. + /* Round-2 fix-round follow-up (docs/contracts/chain-validity.md C2 SS2 row 7, SS5): + * this on-disk shape -- HEAD exactly one hash-verified step behind a fully valid + * chain tail -- is EXACTLY C2's row-7 single-step lagging-HEAD case, which the + * canonical validator (chain.validateChain, which this walk now delegates to + * instead of re-deriving its own classification) correctly reports as + * `reconcilable` / `headAction: "advance"`, NOT a structural failure: per C2 SS3, + * this is the expected shape of "the process died between two of its own writes" + * and is auto-caught-up by the append-time check on the very next append (see + * appendTimeCheckAutoAdvancesASingleStepLaggingHead below), not genuine corruption + * needing an operator's own recovery-runbook intervention. The OLD assertions here + * (`status === "failed"`, reason matching /incomplete append/) predate C2's binding + * 2026-09-15 decision and were themselves the less-correct classification this + * refactor exists to fix -- updated to the canonical validator's more correct + * verdict, not weakened. See statusWalkAndAppendTimeCheckAgreeOnSingleStepLaggingHead + * below for the explicit divergence test C2 SS5 asks for. */ const result = verifyDir(dir); - check("incomplete-append-detected-as-failed", result.status === "failed", JSON.stringify(result)); - check("incomplete-append-reason-names-incomplete-append", /incomplete append/.test(result.reason || ""), JSON.stringify(result)); + check("incomplete-append-is-reconcilable-not-a-structural-failure", result.status === "reconcilable" && result.headAction === "advance", JSON.stringify(result)); + check("incomplete-append-reason-names-the-single-step-lagging-head-signature", /one step behind/.test(result.reason || ""), JSON.stringify(result)); } /* SS8 test 16: mutate a middle entry's entry_sha256. */ @@ -168,6 +183,115 @@ function missingGenesisPrefixDetected() { check("missing-genesis-prefix-reason-names-seq-not-one", /seq=2, expected 1/.test(result.reason || ""), JSON.stringify(result)); } +/* Round-1 fix-plan item 1 (lease keepalive, generalized), round-2 fix pass (Paul's + * 2026-09-16 decision): commit 1's own plan named this walk, alongside reconcileHead's + * startup walk and the WAL-replay loops, as a long synchronous per-entry phase that + * needs the shared maybeRenew() keepalive -- confirms the wiring actually reaches every + * entry of a multi-entry chain, the same way commit 1's own writer-claim unit test + * (grFix1_maybeRenewIsTimeGatedByTheClaimsOwnClock) proves the underlying primitive's + * own time-gating, but here at this walk's own call-site level. */ +function walkGatewaySessionsCallsMaybeRenewOncePerEntry() { + const dir = freshDir("walk-maybe-renew-per-entry"); + const keys = makeKeys(); + for (let i = 0; i < 4; i++) chain.appendSession(dir, sealTrivialSession(`conn-${i}`, keys)); + + let renewCalls = 0; + const result = verifyDir(dir, { maybeRenew: () => { renewCalls++; } }); + check("walk-verifies-with-maybe-renew-wired-in", result.status === "verified", JSON.stringify(result)); + check("walk-calls-maybe-renew-once-per-chain-entry", renewCalls === 4, String(renewCalls)); + // Note: this assertion alone is satisfied whether maybeRenew fires interleaved with + // chain.validateChain's own per-entry walk OR front-loaded in a disconnected pre-pass + // before validateChain ever runs -- a fully-valid chain visits every entry either way. + // walkGatewaySessionsMaybeRenewInterleavedWithValidateChainNotFrontLoaded below is what + // actually distinguishes the two (round-3 fix pass, defect 1). +} + +/* Round-3 fix pass (defect 1): proves maybeRenew calls are genuinely INTERLEAVED with + * chain.validateChain's own per-entry hash/link/sequence walk, not front-loaded in a + * separate pre-pass over the whole raw chain before chain.validateChain is ever invoked. + * A tampered entry at index 2 of a 5-entry chain makes chain.validateChain's own loop + * stop (refuse) after visiting entries 0, 1, and 2 -- entries 3 and 4 are never reached. + * Under the (defective) front-loaded pre-pass this fix corrects, maybeRenew would have + * been called once per RAW entry (5 times) regardless of where -- or whether -- + * chain.validateChain's own structural walk ever got to look at any of them. Under the + * fixed, interleaved wiring, the call count is bounded by how far the real per-entry + * walk actually got: exactly 3, never 5. */ +function walkGatewaySessionsMaybeRenewInterleavedWithValidateChainNotFrontLoaded() { + const dir = freshDir("walk-maybe-renew-interleaved"); + const keys = makeKeys(); + for (let i = 0; i < 5; i++) chain.appendSession(dir, sealTrivialSession(`conn-${i}`, keys)); + const lines = fs.readFileSync(chain.chainPath(dir), "utf8").trim().split("\n").map((l) => JSON.parse(l)); + lines[2].entry_sha256 = "e".repeat(64); // tamper entry index 2 (of 0..4) -- mid-chain. + fs.writeFileSync(chain.chainPath(dir), lines.map((l) => JSON.stringify(l)).join("\n") + "\n"); + + let renewCalls = 0; + const result = verifyDir(dir, { maybeRenew: () => { renewCalls++; } }); + check("walk-interleaved-still-detects-the-tampered-entry", result.status === "failed" && /TAMPERED/.test(result.reason || ""), JSON.stringify(result)); + check( + "walk-maybe-renew-calls-bounded-by-real-progress-not-front-loaded-across-whole-chain", + renewCalls === 3, + `expected exactly 3 (entries 0,1,2 visited before the walk stopped at the tampered entry), got ${renewCalls}` + ); +} + +/* Round-1 fix-plan item 1, round-2 fix pass: mirrors + * recoverAbortsOnLeaseTakeoverDuringWalReplayNotJustOneConnection (tests/gateway/ + * recovery/run-tests.js) at this walk's own level. A lease taken over by a competing + * writer, detected by this walk's own per-entry maybeRenew() call partway through a + * multi-entry chain, must abort the WHOLE walk with the exact detected error -- never + * downgraded to this walk's own ordinary fail-closed `{ status: "failed" }` evidence-only + * report, which is exactly the per-connection "flag for operator review, continue" + * class of handling createLeaseGuard's own doc comment (scripts/gateway/gateway.js) says + * a detected lease loss must never receive. That distinction is the whole point of this + * test: entry[2]'s own hash chain is perfectly intact (unlike every OTHER failure this + * suite exercises above, which the walk legitimately reports as evidence and returns + * from) -- the only reason this run does not return a "verified" result is the + * lease-loss abort firing first, part way through, and propagating instead of being + * reported. */ +function walkAbortsOnLeaseTakeoverMidWalkNotDowngradedToAFailedReport() { + const dir = freshDir("walk-lease-takeover-mid-walk"); + const keys = makeKeys(); + for (let i = 0; i < 5; i++) chain.appendSession(dir, sealTrivialSession(`conn-${i}`, keys)); + + let callCount = 0; + const takeoverError = new Error("simulated competing writer took over this state_dir mid-walk"); + const ctx = { + chain: chain.readChain(dir), + head: chain.readHead(dir), + computeEntrySha256: chain.computeEntrySha256, + bundleExists: (id) => fs.existsSync(chain.bundlePath(dir, id)), + maybeRenew: () => { + callCount++; + if (callCount === 3) throw takeoverError; // strictly mid-walk: entries 0-1 already visited, 3-4 never reached. + }, + isLeaseError: (error) => error === takeoverError, + }; + + let threw = null; + let returned = null; + try { + returned = walkGatewaySessions(ctx); + } catch (error) { + threw = error; + } + check("walk-lease-takeover-aborts-with-the-exact-detected-error", threw === takeoverError, threw && threw.message); + check("walk-lease-takeover-never-returns-a-downgraded-failed-report", returned === null, JSON.stringify(returned)); + + // Without ctx.isLeaseError, the identical maybeRenew failure IS this walk's own + // ordinary fail-closed contract (a caller that never wires in lease recognition gets + // exactly today's pre-fix behavior for any other injected-function failure, not a + // silent behavior change) -- proves the rethrow above is conditional on isLeaseError + // being provided, not a blanket "any maybeRenew throw aborts" rule. + callCount = 0; + const ctxNoLeaseRecognition = { ...ctx, isLeaseError: undefined }; + const fallback = walkGatewaySessions(ctxNoLeaseRecognition); + check( + "walk-maybe-renew-failure-without-isLeaseError-still-fails-closed-not-aborts", + fallback.status === "failed" && /simulated competing writer/.test(fallback.reason || ""), + JSON.stringify(fallback) + ); +} + function bundleIdCollisionRefused() { const dir = freshDir("collision"); const keys = makeKeys(); @@ -185,6 +309,993 @@ function emptyChainIsNotApplicable() { check("empty-chain-not-applicable", result.status === "not-applicable", JSON.stringify(result)); } +/* ================================================================================== + * C2 (docs/contracts/chain-validity.md) -- commit 3: validateChain + readChainTail. + * ================================================================================== */ + +function mkEntry(seq, prev, bundleId) { + const partial = { schema_version: "1.0", seq, bundle_id: bundleId, prev_entry_sha256: prev }; + return { ...partial, entry_sha256: chain.computeEntrySha256(partial) }; +} +function mkHeadFor(entry) { + return { schema_version: "1.0", seq: entry.seq, bundle_id: entry.bundle_id, entry_sha256: entry.entry_sha256 }; +} +function line(entry) { + return JSON.stringify(entry); +} +function writeChainFile(dir, text) { + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + fs.writeFileSync(chain.chainPath(dir), text); +} + +/* ---- validateChain: rows 1/2 (empty states) ---- */ + +function validateChainEmptyGenesisIsNotAFailure() { + const result = chain.validateChain([], null); + check("validate-chain-empty-genesis-is-not-a-failure", result.status === "empty", JSON.stringify(result)); +} + +function validateChainEmptyWithHeadRefuses() { + const e1 = mkEntry(1, null, "b1"); + const result = chain.validateChain([], mkHeadFor(e1)); + check("validate-chain-empty-with-head-refuses", result.status === "refuse" && result.class === "empty-chain-with-head", JSON.stringify(result)); +} + +/* ---- validateChain: rows 5/6/8 (structural/integrity failures -- refuse-and-latch) ---- */ + +function validateChainDetectsSequenceGap() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const e3 = mkEntry(3, e2.entry_sha256, "b3"); + const result = chain.validateChain([e1, e3], null); // e2 deleted -> gap + check("validate-chain-detects-sequence-gap", result.status === "refuse" && result.class === "sequence-gap", JSON.stringify(result)); +} + +function validateChainDetectsDuplicateSeqAsFork() { + const e1 = mkEntry(1, null, "b1"); + const e2a = mkEntry(2, e1.entry_sha256, "b2a"); + const e2b = mkEntry(2, e1.entry_sha256, "b2b"); // same seq, different bundle+hash + const result = chain.validateChain([e1, e2a, e2b], null); + check("validate-chain-detects-duplicate-seq-as-fork", result.status === "refuse" && result.class === "fork", JSON.stringify(result)); +} + +function validateChainDetectsTamperedEntry() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const tampered = { ...e2, entry_sha256: "9".repeat(64) }; + const result = chain.validateChain([e1, tampered], null); + check("validate-chain-detects-tampered-entry", result.status === "refuse" && result.class === "tampered" && result.at === 1, JSON.stringify(result)); +} + +function validateChainDetectsBrokenLink() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, "d".repeat(64), "b2"); // internally hash-consistent, wrong link + const result = chain.validateChain([e1, e2], null); + check("validate-chain-detects-broken-link", result.status === "refuse" && result.class === "broken-link", JSON.stringify(result)); +} + +function validateChainDetectsInvalidGenesisSeq() { + const forged = mkEntry(2, null, "b1"); // hash-consistent, but not seq=1 + const result = chain.validateChain([forged], null); + check("validate-chain-detects-invalid-genesis-seq", result.status === "refuse" && result.class === "invalid-genesis", JSON.stringify(result)); +} + +function validateChainDetectsInvalidGenesisPrev() { + const forged = mkEntry(1, "a".repeat(64), "b1"); // seq=1 but prev != null + const result = chain.validateChain([forged], null); + check("validate-chain-detects-invalid-genesis-prev", result.status === "refuse" && result.class === "invalid-genesis", JSON.stringify(result)); +} + +function validateChainRefusesMalformedInteriorRecordNotJustTheLast() { + const e1 = mkEntry(1, null, "b1"); + const e3 = mkEntry(3, "irrelevant-since-interior-is-what-fails", "b3"); + // null in the MIDDLE (not the physical last position) must never get the + // torn-tail accommodation -- always refuse, per C2 row 8. + const result = chain.validateChain([e1, null, e3], null); + check("validate-chain-refuses-malformed-interior-record", result.status === "refuse" && result.class === "malformed-interior" && result.at === 1, JSON.stringify(result)); +} + +/* ---- validateChain: row 7 (mid-run lagging HEAD, single-step, hash-verified) ---- */ + +function validateChainAutoAdvancesSingleStepLaggingHead() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const e3 = mkEntry(3, e2.entry_sha256, "b3"); + const result = chain.validateChain([e1, e2, e3], mkHeadFor(e2)); + check("validate-chain-auto-advances-single-step-lagging-head", result.status === "reconcilable" && result.headAction === "advance" && result.tail.seq === 3, JSON.stringify(result)); + check("validate-chain-single-step-lag-emits-a-loud-anomaly", typeof result.anomaly === "string" && result.anomaly.length > 0, JSON.stringify(result)); +} + +function validateChainRefusesMultiStepLaggingHead() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const e3 = mkEntry(3, e2.entry_sha256, "b3"); + const result = chain.validateChain([e1, e2, e3], mkHeadFor(e1)); // two steps behind + check("validate-chain-refuses-multi-step-lagging-head", result.status === "refuse" && result.class === "multi-step-lag", JSON.stringify(result)); +} + +function validateChainRefusesHeadAheadOfTail() { + const e1 = mkEntry(1, null, "b1"); + const fakeAhead = { schema_version: "1.0", seq: 2, bundle_id: "ghost", entry_sha256: "b".repeat(64) }; + const result = chain.validateChain([e1], fakeAhead); + check("validate-chain-refuses-head-ahead-of-tail", result.status === "refuse" && result.class === "head-ahead-of-tail", JSON.stringify(result)); +} + +function validateChainRefusesHeadSameSeqDifferentHashAsFork() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const forkedHead = { schema_version: "1.0", seq: 2, bundle_id: "b2-other", entry_sha256: "c".repeat(64) }; + const result = chain.validateChain([e1, e2], forkedHead); + check("validate-chain-refuses-head-same-seq-different-hash-as-fork", result.status === "refuse" && result.class === "fork", JSON.stringify(result)); +} + +function validateChainRefusesOneStepBackHeadThatIsNotAGenuineAncestor() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + // seq is exactly one behind tail (looks like row 7 at a glance) but its hash does + // not actually match tail.prev_entry_sha256 -- must NOT auto-catch-up. + const notAnAncestor = { schema_version: "1.0", seq: 1, bundle_id: "b1-impostor", entry_sha256: "e".repeat(64) }; + const result = chain.validateChain([e1, e2], notAnAncestor); + check("validate-chain-refuses-one-step-back-head-that-is-not-a-genuine-ancestor", result.status === "refuse" && result.class === "head-not-ancestor", JSON.stringify(result)); +} + +/* ---- validateChain: rows 3/4 (HEAD absent/unusable -- rebuild from verified tail) ---- */ + +function validateChainRebuildsMissingHeadFromVerifiedTail() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const result = chain.validateChain([e1, e2], null); + check("validate-chain-rebuilds-missing-head-from-verified-tail", result.status === "reconcilable" && result.headAction === "rebuild" && result.tail.seq === 2, JSON.stringify(result)); +} + +function validateChainDoesNotRebuildWhenTheChainItselfDoesNotValidate() { + const e1 = mkEntry(1, null, "b1"); + // Shape-valid (64-hex prev), but wrong link AND a sequence gap -- HEAD being + // absent must never bypass the structural walk and rebuild blindly. + const e3 = mkEntry(3, "d".repeat(64), "b3"); + const result = chain.validateChain([e1, e3], null); + check("validate-chain-does-not-rebuild-when-the-chain-itself-does-not-validate", result.status === "refuse", JSON.stringify(result)); +} + +/* ---- validateChain: row 9 (torn/unparseable FINAL record) ---- */ + +function validateChainTruncatesTornFinalRecordAndRebuildsHead() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + // `null` stands in for "this physical line could not be parsed at all" (per + // validateChain's own documented input contract). + const result = chain.validateChain([e1, e2, null], null); + check("validate-chain-truncates-torn-final-record-and-rebuilds-head", result.status === "reconcilable" && result.truncatedTail === true && result.headAction === "rebuild" && result.tail.seq === 2, JSON.stringify(result)); +} + +function validateChainTornFinalRecordWithHeadAlreadyAtTheTruncatedTailStillReportsReconciliation() { + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + // HEAD already matches what the truncated chain's tail will be -- this must NOT + // be silently reported as "valid" (nothing to do): the on-disk garbage still + // needs isolating/truncating, so it must surface as a reconciliation action. + const result = chain.validateChain([e1, e2, null], mkHeadFor(e2)); + check("validate-chain-torn-tail-with-head-already-consistent-still-reconciles-not-silently-valid", result.status === "reconcilable" && result.truncatedTail === true, JSON.stringify(result)); +} + +function validateChainTornSoleGenesisRecordWithNoHeadReconcilesToEmptyTail() { + const result = chain.validateChain([null], null); + check("validate-chain-torn-sole-genesis-record-with-no-head-reconciles-to-empty-tail", result.status === "reconcilable" && result.tail === null && result.truncatedTail === true, JSON.stringify(result)); +} + +function validateChainTornSoleRecordWithHeadPresentRefuses() { + const ghostHead = { schema_version: "1.0", seq: 1, bundle_id: "ghost", entry_sha256: "f".repeat(64) }; + const result = chain.validateChain([null], ghostHead); + check("validate-chain-torn-sole-record-with-head-present-refuses", result.status === "refuse", JSON.stringify(result)); +} + +function validateChainNeverConflatesTornTailWithInteriorCorruption() { + // A shape-VALID final record whose hash simply does not recompute is TAMPERING, + // not a torn write -- must be refused via the ordinary hash check, never given + // the row-9 accommodation. + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const tamperedTail = { ...e2, entry_sha256: "7".repeat(64) }; + const result = chain.validateChain([e1, tamperedTail], null); + check("validate-chain-never-conflates-a-tampered-tail-with-a-torn-tail", result.status === "refuse" && result.class === "tampered", JSON.stringify(result)); +} + +/* ---- readChainTail: absent / empty / corrupt stay three distinct outcomes ---- */ + +function readChainTailAbsentWhenFileDoesNotExist() { + const dir = freshDir("tail-absent"); + fs.mkdirSync(dir, { recursive: true }); // no gateway-sessions/ dir at all + const result = chain.readChainTail(dir); + check("read-chain-tail-absent-when-file-does-not-exist", result.status === "absent", JSON.stringify(result)); +} + +function readChainTailEmptyWhenFileIsZeroBytes() { + const dir = freshDir("tail-zero-bytes"); + writeChainFile(dir, ""); + const result = chain.readChainTail(dir); + check("read-chain-tail-empty-when-file-is-zero-bytes", result.status === "empty", JSON.stringify(result)); +} + +function readChainTailEmptyWhenFileIsOnlyBlankLines() { + const dir = freshDir("tail-blank-only"); + writeChainFile(dir, "\n\n\n"); + const result = chain.readChainTail(dir); + check("read-chain-tail-empty-when-file-is-only-blank-lines", result.status === "empty", JSON.stringify(result)); +} + +function readChainTailSucceedsWithTrailingNewline() { + const dir = freshDir("tail-trailing-nl"); + const e1 = mkEntry(1, null, "b1"); + writeChainFile(dir, line(e1) + "\n"); + const result = chain.readChainTail(dir); + check("read-chain-tail-succeeds-with-trailing-newline", result.status === "ok" && result.hadTrailingNewline === true && result.entry.bundle_id === "b1", JSON.stringify(result)); +} + +function readChainTailSucceedsWithoutTrailingNewline() { + const dir = freshDir("tail-no-trailing-nl"); + const e1 = mkEntry(1, null, "b1"); + writeChainFile(dir, line(e1)); // no trailing \n + const result = chain.readChainTail(dir); + check("read-chain-tail-succeeds-without-trailing-newline", result.status === "ok" && result.hadTrailingNewline === false && result.entry.bundle_id === "b1", JSON.stringify(result)); +} + +function readChainTailReturnsTheSameEntryRegardlessOfTrailingNewline() { + const dir1 = freshDir("tail-nl-a"); + const dir2 = freshDir("tail-nl-b"); + const e1 = mkEntry(1, null, "same-content"); + writeChainFile(dir1, line(e1) + "\n"); + writeChainFile(dir2, line(e1)); + const r1 = chain.readChainTail(dir1); + const r2 = chain.readChainTail(dir2); + check("read-chain-tail-trailing-newline-does-not-change-the-parsed-entry", r1.status === "ok" && r2.status === "ok" && JSON.stringify(r1.entry) === JSON.stringify(r2.entry), JSON.stringify({ r1, r2 })); +} + +function readChainTailSkipsBlankLinesBeforeTheLastEntry() { + const dir = freshDir("tail-blank-between"); + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + writeChainFile(dir, line(e1) + "\n\n" + line(e2) + "\n"); + const result = chain.readChainTail(dir); + check("read-chain-tail-skips-blank-lines-before-the-last-entry", result.status === "ok" && result.entry.bundle_id === "b2", JSON.stringify(result)); +} + +function readChainTailFindsTheLastEntryWhenTheWindowStartsMidRecord() { + const dir = freshDir("tail-mid-record"); + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const e3 = mkEntry(3, e2.entry_sha256, "b3"); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n" + line(e3) + "\n"); + // A tiny initial window guarantees the first read starts strictly inside e3's own + // line (no newline visible), forcing at least one grow-and-retry before success. + const result = chain.readChainTail(dir, { initialWindowBytes: 8 }); + check("read-chain-tail-finds-the-last-entry-when-the-window-starts-mid-record", result.status === "ok" && result.entry.bundle_id === "b3", JSON.stringify(result)); +} + +function readChainTailGrowsPastTheInitialWindowForAnOversizedFinalRecord() { + const dir = freshDir("tail-oversized-final"); + const e1 = mkEntry(1, null, "b1"); + const bigBundleId = "b2-" + "x".repeat(9000); // final record now well over 8 KiB + const e2 = mkEntry(2, e1.entry_sha256, bigBundleId); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n"); + const result = chain.readChainTail(dir); // default 8 KiB initial window + check("read-chain-tail-grows-past-the-initial-window-for-an-oversized-final-record", result.status === "ok" && result.entry.bundle_id === bigBundleId, JSON.stringify({ status: result.status, matched: result.status === "ok" && result.entry.bundle_id === bigBundleId })); +} + +function readChainTailFindsALargeFinalRecordWhenANewlineIsAlreadyInTheFirstWindow() { + // Distinct from the grow case above: here the record is large but the FIRST read + // already contains the newline that starts it (no growth needed) -- exercises the + // "final record larger than the window, but the window still contains a newline" + // boundary without conflating it with the retry path. + const dir = freshDir("tail-large-first-window"); + const e1 = mkEntry(1, null, "b1"); + const bigBundleId = "b2-" + "y".repeat(4000); // large, but smaller than the window + const e2 = mkEntry(2, e1.entry_sha256, bigBundleId); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n"); + const result = chain.readChainTail(dir); // default 8 KiB window comfortably covers this + check("read-chain-tail-finds-a-large-final-record-already-inside-the-first-window", result.status === "ok" && result.entry.bundle_id === bigBundleId, JSON.stringify({ status: result.status })); +} + +function readChainTailFailsClosedWhenNoNewlineIsFoundWithinTheCappedWindow() { + const dir = freshDir("tail-uncapped"); + const e1 = mkEntry(1, null, "b1"); + const hugeBundleId = "z".repeat(5000); // final record itself exceeds the test's tiny cap + const e2 = mkEntry(2, e1.entry_sha256, hugeBundleId); + writeChainFile(dir, line(e1) + "\n" + line(e2)); // no trailing newline either + const result = chain.readChainTail(dir, { initialWindowBytes: 64, maxEntryBytes: 512 }); + check("read-chain-tail-fails-closed-when-no-newline-boundary-is-found-within-the-capped-window", result.status === "corrupt", JSON.stringify(result)); + check("read-chain-tail-fail-closed-does-not-fall-back-to-an-earlier-valid-record", !(result.entry && result.entry.bundle_id === "b1"), JSON.stringify(result)); +} + +function readChainTailReportsCorruptForATruncatedFinalRecord() { + const dir = freshDir("tail-truncated-final"); + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + const full = line(e1) + "\n" + line(e2) + "\n"; + const tornPoint = full.length - 5; // cut the last record off mid-field + writeChainFile(dir, full.slice(0, tornPoint)); + const result = chain.readChainTail(dir); + check("read-chain-tail-reports-corrupt-for-a-truncated-final-record", result.status === "corrupt", JSON.stringify(result)); + check("read-chain-tail-truncated-final-record-does-not-silently-return-the-earlier-valid-entry", !(result.entry && result.entry.bundle_id === "b1"), JSON.stringify(result)); +} + +function readChainTailReportsCorruptForAMalformedFinalRecord() { + const dir = freshDir("tail-malformed-final"); + const e1 = mkEntry(1, null, "b1"); + writeChainFile(dir, line(e1) + "\n" + "{not even close to json\n"); + const result = chain.readChainTail(dir); + check("read-chain-tail-reports-corrupt-for-a-malformed-final-record", result.status === "corrupt", JSON.stringify(result)); +} + +function readChainTailReportsCorruptWhenTheFinalRecordFailsItsShapeCheck() { + const dir = freshDir("tail-bad-shape-final"); + const e1 = mkEntry(1, null, "b1"); + const badShape = { schema_version: "1.0", seq: 2, bundle_id: "b2" }; // missing hash fields, but VALID json + writeChainFile(dir, line(e1) + "\n" + JSON.stringify(badShape) + "\n"); + const result = chain.readChainTail(dir); + check("read-chain-tail-reports-corrupt-when-the-final-record-fails-its-shape-check", result.status === "corrupt", JSON.stringify(result)); +} + +function readChainTailHandlesUtf8BoundariesSplitAcrossTheWindowEdge() { + const dir = freshDir("tail-utf8-boundary"); + // Multi-byte bundle_id (emoji + accented chars are 2-4 byte UTF-8 sequences) on + // the record BEFORE the last one, positioned so a small window's left edge is + // very likely to land inside one of its multi-byte characters -- the last + // entry's own content must still parse correctly regardless, because only bytes + // strictly after the boundary newline are ever decoded. + const e1 = mkEntry(1, null, "héllo-🎉-bündle-" + "€".repeat(40)); + const e2 = mkEntry(2, e1.entry_sha256, "plain-ascii-tail-résumé-日本語"); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n"); + // Small window forced to start somewhere inside e1's multi-byte content. + const result = chain.readChainTail(dir, { initialWindowBytes: 20 }); + check("read-chain-tail-handles-utf8-boundaries-split-across-the-window-edge", result.status === "ok" && result.entry.bundle_id === "plain-ascii-tail-résumé-日本語", JSON.stringify(result)); +} + +function readChainTailUtf8LastEntryItselfSurvivesGrowth() { + const dir = freshDir("tail-utf8-last-entry-grows"); + const e1 = mkEntry(1, null, "b1"); + // The LAST entry itself contains multi-byte UTF-8 content and is large enough + // that the window must grow at least once to capture all of it. + const unicodeBundleId = "🎉-" + "日".repeat(3000); + const e2 = mkEntry(2, e1.entry_sha256, unicodeBundleId); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n"); + const result = chain.readChainTail(dir, { initialWindowBytes: 64 }); + check("read-chain-tail-utf8-last-entry-itself-survives-growth", result.status === "ok" && result.entry.bundle_id === unicodeBundleId, JSON.stringify({ status: result.status })); +} + +function readChainTailHandlesShortReads() { + const dir = freshDir("tail-short-reads"); + const e1 = mkEntry(1, null, "b1"); + const e2 = mkEntry(2, e1.entry_sha256, "b2"); + writeChainFile(dir, line(e1) + "\n" + line(e2) + "\n"); + + const realReadSync = fs.readSync; + let calls = 0; + fs.readSync = function shortReadingReadSync(fd, buffer, offset, length, position) { + calls++; + const cappedLength = Math.min(length, 3); // force many short reads, 3 bytes at a time + return realReadSync(fd, buffer, offset, cappedLength, position); + }; + let result; + try { + result = chain.readChainTail(dir); + } finally { + fs.readSync = realReadSync; + } + check("read-chain-tail-handles-short-reads", result.status === "ok" && result.entry.bundle_id === "b2", JSON.stringify(result)); + check("read-chain-tail-short-reads-actually-exercised-multiple-calls", calls > 1, `calls=${calls}`); +} + +function readChainTailAbsentEmptyCorruptAreThreeDistinctOutcomes() { + const absentDir = freshDir("distinct-absent"); + fs.mkdirSync(absentDir, { recursive: true }); + const emptyDir = freshDir("distinct-empty"); + writeChainFile(emptyDir, ""); + const corruptDir = freshDir("distinct-corrupt"); + writeChainFile(corruptDir, "{not json\n"); + + const absent = chain.readChainTail(absentDir); + const empty = chain.readChainTail(emptyDir); + const corrupt = chain.readChainTail(corruptDir); + const statuses = [absent.status, empty.status, corrupt.status]; + check("read-chain-tail-absent-empty-corrupt-are-three-distinct-outcomes", new Set(statuses).size === 3 && statuses.includes("absent") && statuses.includes("empty") && statuses.includes("corrupt"), JSON.stringify(statuses)); +} + +/* ================================================================================== + * Round-1 fix-plan commit 4 -- reconcileHead (startup) + the classified append-time + * HEAD/tail check (chain.appendSession / chain.repairMissingChainEntry). Constructs a + * genuinely stale HEAD through the real public API (chain.appendSession + a targeted + * HEAD.json overwrite mirroring exactly the crash window between chain.appendSession's + * own step 2 (chain.jsonl append) and step 3 (HEAD.json update) -- never by disabling + * the safety check itself. + * ================================================================================== */ + +function makeStaleHeadFixture(dir, keys, n) { + // Appends `n` real, distinct sessions, then rolls HEAD.json back to name an EARLIER + // entry -- exactly the on-disk shape a crash between chain.appendSession's own + // chain.jsonl-append and HEAD.json-update steps leaves behind, reproduced through the + // real public API rather than by hand-crafting a chain.jsonl fixture. + const entries = []; + for (let i = 0; i < n; i++) { + entries.push(chain.appendSession(dir, sealTrivialSession(`stale-head-${i}`, keys))); + } + return entries; +} + +function appendTimeCheckAutoAdvancesASingleStepLaggingHead() { + const dir = freshDir("append-time-single-lag"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + // Roll HEAD back to entry[0] -- exactly one step behind the real tail (entry[1]). + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 })); + check("append-time-lag-fixture-head-still-stale", chain.readHead(dir).seq === 1, JSON.stringify(chain.readHead(dir))); + + const logs = []; + const newEntry = chain.appendSession(dir, sealTrivialSession("stale-head-third", keys), { log: (l) => logs.push(l) }); + check("append-time-lag-auto-advance-new-entry-is-seq-3", newEntry.seq === 3, JSON.stringify(newEntry)); + check("append-time-lag-auto-advance-new-entry-links-to-entry2", newEntry.prev_entry_sha256 === entries[1].entry_sha256, JSON.stringify(newEntry)); + const headAfter = chain.readHead(dir); + check("append-time-lag-auto-advance-head-ends-at-entry3", headAfter.seq === 3 && headAfter.bundle_id === newEntry.bundle_id, JSON.stringify(headAfter)); + check("append-time-lag-auto-advance-chain-has-exactly-3-entries-no-fork", chain.readChain(dir).length === 3, JSON.stringify(chain.readChain(dir).map((e) => e.seq))); + const anomalyLog = logs.find((l) => l.includes("gateway_chain_head_lag_auto_advanced")); + check("append-time-lag-auto-advance-logs-a-loud-anomaly", Boolean(anomalyLog), JSON.stringify(logs)); +} + +/* C2 (docs/contracts/chain-validity.md) SS5's own required proof: "one implementation, + * not three copies" -- reconcileHead, the append-time check, and this walk must all + * reach the SAME classification for the same on-disk state, because all three now + * delegate to the identical chain.validateChain. This constructs C2's row 7 (single- + * step lagging HEAD, hash-verified ancestor) -- chosen because it is already the + * best-tested case elsewhere (appendTimeCheckAutoAdvancesASingleStepLaggingHead + * above) and is unambiguous per the contract: NOT corruption at either call site -- + * and asserts the status walk (checks/register-gateway-sessions.js, via verifyDir) + * and the append-time check (chain.checkHeadAgainstTailOrRepair, the exact function + * chain.appendSession calls before every append) agree: neither refuses/latches what + * the other treats as a silently-repairable lag. */ +function statusWalkAndAppendTimeCheckAgreeOnSingleStepLaggingHead() { + const dir = freshDir("divergence-single-lag"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + const staleHead = { schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 }; + fs.writeFileSync(chain.headPath(dir), JSON.stringify(staleHead)); + + // The status walk, on the stale-HEAD state, BEFORE anything repairs it on disk. + const walkResult = verifyDir(dir); + check( + "divergence-status-walk-classifies-single-step-lag-as-reconcilable-not-failed", + walkResult.status === "reconcilable" && walkResult.headAction === "advance", + JSON.stringify(walkResult) + ); + + // The append-time check, on the exact same on-disk state -- called directly (rather + // than via chain.appendSession) so this test exercises the check in isolation, + // without also appending a new entry, which would change what state the two are + // being compared against. + chain._resetChainIntegrityFailureForTests(); + const logs = []; + let threw = null; + let repairedHead = null; + try { + repairedHead = chain.checkHeadAgainstTailOrRepair(dir, staleHead, (l) => logs.push(l)); + } catch (error) { + threw = error; + } + check("divergence-append-time-check-does-not-refuse-the-same-state", threw === null, threw ? `${threw.code}: ${threw.message}` : "threw unexpectedly"); + check("divergence-append-time-check-does-not-latch-the-same-state", chain.getChainIntegrityFailure() === null, JSON.stringify(chain.getChainIntegrityFailure())); + + // Agreement, stated explicitly: both call sites, given the identical on-disk state, + // land on the SAME verified tail (entry[1], seq=2) as what HEAD should point at -- + // neither one refuses/latches what the other one silently repairs. + check( + "divergence-both-call-sites-agree-on-the-verified-tail", + Boolean(repairedHead) && repairedHead.seq === entries[1].seq && repairedHead.entry_sha256 === entries[1].entry_sha256, + JSON.stringify(repairedHead) + ); + const anomalyLog = logs.find((l) => l.includes("gateway_chain_head_lag_auto_advanced")); + check("divergence-append-time-check-logs-the-same-auto-advance-anomaly-the-walk-reports", Boolean(anomalyLog), JSON.stringify(logs)); +} + +function appendTimeCheckRefusesATwoStepLaggingHead() { + const dir = freshDir("append-time-two-step-lag"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 3); + // Roll HEAD back TWO steps behind the real tail (entry[2]) -- not the auto-repairable + // single-step case. + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 })); + + chain._resetChainIntegrityFailureForTests(); + let threw = null; + try { + chain.appendSession(dir, sealTrivialSession("two-step-lag-new", keys)); + } catch (error) { + threw = error; + } + check("append-time-two-step-lag-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_HEAD_DIVERGED", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("append-time-two-step-lag-chain-unchanged-no-fourth-entry", chain.readChain(dir).length === 3, JSON.stringify(chain.readChain(dir).map((e) => e.seq))); + check("append-time-two-step-lag-head-untouched", chain.readHead(dir).seq === 1, JSON.stringify(chain.readHead(dir))); + const latched = chain.getChainIntegrityFailure(); + check("append-time-two-step-lag-latches-the-integrity-failure", Boolean(latched) && latched.class === "multi-step-lag", JSON.stringify(latched)); +} + +function appendTimeCheckRefusesAForkedHead() { + const dir = freshDir("append-time-fork"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + // A genuine fork: same seq/bundle_id as the real tail, but a bogus hash that does not + // match anything actually in chain.jsonl. + const forked = Object.assign({}, entries[1], { entry_sha256: "d".repeat(64) }); + fs.writeFileSync(chain.headPath(dir), JSON.stringify(forked)); + + let threw = null; + try { + chain.appendSession(dir, sealTrivialSession("fork-new", keys)); + } catch (error) { + threw = error; + } + check("append-time-fork-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_HEAD_DIVERGED", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("append-time-fork-chain-unchanged", chain.readChain(dir).length === 2, JSON.stringify(chain.readChain(dir).map((e) => e.seq))); +} + +function reconcileHeadEmptyChainIsANoOpWithZeroWrites() { + const dir = freshDir("reconcile-empty"); + fs.mkdirSync(dir, { recursive: true }); // no gateway-sessions/ dir at all yet + const result = chain.reconcileHead(dir); + check("reconcile-empty-action-none", result.action === "none", JSON.stringify(result)); + check("reconcile-empty-writes-no-head-file", !fs.existsSync(chain.headPath(dir)), "HEAD.json was written for a genuinely empty chain"); + check("reconcile-empty-writes-no-chain-file", !fs.existsSync(chain.chainPath(dir)), "chain.jsonl was written for a genuinely empty chain"); +} + +function reconcileHeadRefusesAMultiStepLaggingHeadAtStartupToo() { + // Round-1 fix-plan commit 4: multi-step lag is a REFUSE case per C2 row 7's own last + // sentence, at BOTH call sites -- reconcileHead delegates entirely to the same + // classifyHeadAgainstTail/validateChain classification the O(1) append-time check + // uses, it does not get a weaker/more-permissive version just because it can afford a + // full walk. Only an EXACT single-step, hash-matching lag ever auto-catches-up. + const dir = freshDir("reconcile-multi-lag"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 4); + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 })); + + chain._resetChainIntegrityFailureForTests(); + let threw = null; + try { + chain.reconcileHead(dir); + } catch (error) { + threw = error; + } + check("reconcile-multi-step-lag-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("reconcile-multi-step-lag-head-untouched", chain.readHead(dir).bundle_id === entries[0].bundle_id, JSON.stringify(chain.readHead(dir))); + const latched = chain.getChainIntegrityFailure(); + check("reconcile-multi-step-lag-latches-the-integrity-failure", Boolean(latched) && latched.class === "multi-step-lag", JSON.stringify(latched)); +} + +function reconcileHeadRebuildsFromAMalformedHeadFile() { + const dir = freshDir("reconcile-malformed-head"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + fs.writeFileSync(chain.headPath(dir), "{not even close to json"); + + const result = chain.reconcileHead(dir); + check("reconcile-malformed-head-action-advanced", result.action === "advanced" && result.to.seq === 2, JSON.stringify(result)); + check("reconcile-malformed-head-rebuilt-correctly", chain.readHead(dir).bundle_id === entries[1].bundle_id, JSON.stringify(chain.readHead(dir))); +} + +/* Codex PR #33 review "renew the claim while validating the startup chain": reconcileHead + * renewed the writer claim exactly ONCE before handing the whole chain to validateChain -- + * so on a long chain the synchronous walk itself could outlast the 45s staleness window + * and let another same-host gateway steal the claim mid-walk, after which this function + * would still write HEAD.json. The status/health walk already had the per-entry hook + * (walkGatewaySessionsCallsMaybeRenewOncePerEntry above); this is the same wiring for the + * startup walk. 5 valid entries => 1 pre-walk renew + 5 per-entry renews. A chain that + * validates clean returns before the repair write, so no extra renew is expected there. + * + * Round-N fix (frontier-panel finding #8): readChainForValidation now streams the chain + * file in chunks (streamChainFileLines) instead of reading it whole into a string, and + * accepts its own onProgress hook wired to the same maybeRenew closure so a chain large + * enough to outlast the staleness window is renewed DURING the read, not just before/after + * it. This whole 5-entry fixture fits in a single 1MB read chunk, so the streaming read + * contributes exactly one additional renew call beyond the pre-walk + per-entry total: + * 1 pre-walk + 1 read-phase (one chunk) + 5 per-entry = 7. */ +function reconcileHeadRenewsTheClaimPerEntryAcrossTheStartupWalk() { + const dir = freshDir("reconcile-per-entry-renew"); + const keys = makeKeys(); + for (let i = 0; i < 5; i++) chain.appendSession(dir, sealTrivialSession(`renew-${i}`, keys)); + + let renewCalls = 0; + const result = chain.reconcileHead(dir, { maybeRenew: () => { renewCalls++; } }); + check("reconcile-per-entry-renew-chain-is-valid", result.action === "none", JSON.stringify(result)); + check( + "reconcile-renews-the-claim-once-per-chain-entry-not-only-once-before-the-walk", + renewCalls === 7, + `expected 7 (1 pre-walk + 1 read-phase + 5 per-entry), got ${renewCalls}` + ); +} + +/* Second half of the same review item ("revalidate ownership before the repair write"): + * the per-entry hook cannot cover a walk of 0-1 entries, so the HEAD.json repair write + * gets its own unconditional renew immediately before it, mirroring + * gateway.js#abandonConnection's own pre-appendSession renew. 2 entries with a one-step + * stale HEAD => 1 pre-walk + 2 per-entry + 1 pre-write = 4, plus the same read-phase renew + * described above (Round-N fix, finding #8) since this fixture's tiny chain file also + * fits in a single streamed chunk => 1 pre-walk + 1 read-phase + 2 per-entry + 1 pre-write = 5. */ +function reconcileHeadRenewsTheClaimAgainImmediatelyBeforeTheRepairWrite() { + const dir = freshDir("reconcile-pre-write-renew"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 })); + + let renewCalls = 0; + const result = chain.reconcileHead(dir, { maybeRenew: () => { renewCalls++; } }); + check("reconcile-pre-write-renew-advanced-the-head", result.action === "advanced" && result.to.seq === 2, JSON.stringify(result)); + check( + "reconcile-renews-the-claim-immediately-before-the-head-repair-write", + renewCalls === 5, + `expected 5 (1 pre-walk + 1 read-phase + 2 per-entry + 1 pre-write), got ${renewCalls}` + ); +} + +/* And the point of renewing at all: a lease lost PARTWAY THROUGH the walk must abort the + * walk where it happened, never fall through to a HEAD.json write this process no longer + * owns the right to make. Fails the SECOND maybeRenew call -- the first entry of the walk, + * after the pre-walk renew already succeeded. */ +function reconcileHeadPerEntryLeaseLossAbortsBeforeTheRepairWrite() { + const dir = freshDir("reconcile-mid-walk-lease-loss"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + const staleHead = { schema_version: entries[0].schema_version, seq: entries[0].seq, bundle_id: entries[0].bundle_id, entry_sha256: entries[0].entry_sha256 }; + fs.writeFileSync(chain.headPath(dir), JSON.stringify(staleHead)); + + let renewCalls = 0; + let threw = null; + try { + chain.reconcileHead(dir, { + maybeRenew: () => { + renewCalls++; + if (renewCalls === 2) throw Object.assign(new Error("simulated lease takeover mid startup chain walk"), { code: "WRITER_CLAIM_LOST" }); + }, + }); + } catch (error) { + threw = error; + } + check("reconcile-mid-walk-lease-loss-propagates", threw !== null && threw.code === "WRITER_CLAIM_LOST", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("reconcile-mid-walk-lease-loss-leaves-head-untouched", chain.readHead(dir).seq === 1, JSON.stringify(chain.readHead(dir))); +} + +function reconcileHeadRefusesAGenuineFork() { + const dir = freshDir("reconcile-fork"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + const forked = Object.assign({}, entries[1], { entry_sha256: "c".repeat(64) }); + fs.writeFileSync(chain.headPath(dir), JSON.stringify(forked)); + + chain._resetChainIntegrityFailureForTests(); + let threw = null; + try { + chain.reconcileHead(dir); + } catch (error) { + threw = error; + } + check("reconcile-fork-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("reconcile-fork-head-untouched", chain.readHead(dir).entry_sha256 === "c".repeat(64), JSON.stringify(chain.readHead(dir))); + const latched = chain.getChainIntegrityFailure(); + check("reconcile-fork-latches-the-integrity-failure", Boolean(latched) && latched.class === "fork", JSON.stringify(latched)); +} + +/* Round-1 fix-plan commit 6 (classified false positives): an OPERATIONAL failure + * merely opening chain.jsonl (EACCES/EPERM -- exactly what a misapplied or + * in-progress commit-2 group-readable chmod pass produces) must refuse THIS append + * but must NOT engage the process-wide admission latch. Simulated via a targeted + * fs.openSync patch (this sandbox runs as root, where a real chmod would not actually + * deny the read) rather than real OS permissions -- mirrors tests/gateway/ + * startup-permissions/run-tests.js's own withPatched/codeError fault-injection style. */ +function checkHeadAgainstTailOperationalPermissionFailureRefusesButDoesNotLatch() { + const dir = freshDir("tail-eacces-operational"); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + chain._resetChainIntegrityFailureForTests(); + + const targetPath = chain.chainPath(dir); + const originalOpenSync = fs.openSync; + fs.openSync = function (p, ...rest) { + if (p === targetPath) { + const err = new Error("permission denied"); + err.code = "EACCES"; + throw err; + } + return originalOpenSync.apply(fs, [p, ...rest]); + }; + let threw = null; + try { + chain.checkHeadAgainstTailOrRepair(dir, null, () => {}); + } catch (error) { + threw = error; + } finally { + fs.openSync = originalOpenSync; + } + check( + "tail-eacces-refuses-this-append-with-the-operational-code", + threw !== null && threw.code === "GATEWAY_CHAIN_TAIL_UNREADABLE_OPERATIONAL", + threw ? `${threw.code}: ${threw.message}` : "did not throw" + ); + check("tail-eacces-does-not-latch-admission", chain.getChainIntegrityFailure() === null, JSON.stringify(chain.getChainIntegrityFailure())); +} + +/* Companion regression lock: a non-permission open failure (anything other than + * EACCES/EPERM) for the exact same "could not even open chain.jsonl" shape must still + * refuse AND latch -- proves the operational carve-out above is narrowly scoped to + * permission codes, not a general "any open failure is fine" loophole. */ +function checkHeadAgainstTailGenuineUnreadableTailStillLatches() { + const dir = freshDir("tail-genuine-unreadable"); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + chain._resetChainIntegrityFailureForTests(); + + const targetPath = chain.chainPath(dir); + const originalOpenSync = fs.openSync; + fs.openSync = function (p, ...rest) { + if (p === targetPath) { + const err = new Error("I/O error"); + err.code = "EIO"; + throw err; + } + return originalOpenSync.apply(fs, [p, ...rest]); + }; + let threw = null; + try { + chain.checkHeadAgainstTailOrRepair(dir, null, () => {}); + } catch (error) { + threw = error; + } finally { + fs.openSync = originalOpenSync; + } + check( + "tail-eio-refuses-this-append-with-the-diverged-code", + threw !== null && threw.code === "GATEWAY_CHAIN_HEAD_DIVERGED", + threw ? `${threw.code}: ${threw.message}` : "did not throw" + ); + const latched = chain.getChainIntegrityFailure(); + check("tail-eio-latches-admission", Boolean(latched) && latched.class === "tail-unreadable", JSON.stringify(latched)); + chain._resetChainIntegrityFailureForTests(); +} + +function reconcileHeadAheadRefuses() { + const dir = freshDir("reconcile-head-ahead"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 1); + const ahead = { schema_version: "1.0", seq: 5, bundle_id: "ghost", entry_sha256: "b".repeat(64) }; + fs.writeFileSync(chain.headPath(dir), JSON.stringify(ahead)); + + let threw = null; + try { + chain.reconcileHead(dir); + } catch (error) { + threw = error; + } + check("reconcile-head-ahead-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", threw ? `${threw.code}: ${threw.message}` : "did not throw"); +} + +function reconcileHeadRefusesWhenChainFailsStructuralValidation() { + const dir = freshDir("reconcile-tampered"); + const keys = makeKeys(); + const entries = makeStaleHeadFixture(dir, keys, 2); + // Tamper the interior (first) entry directly in chain.jsonl. + const lines = fs.readFileSync(chain.chainPath(dir), "utf8").trim().split("\n").map((l) => JSON.parse(l)); + lines[0].entry_sha256 = "9".repeat(64); + fs.writeFileSync(chain.chainPath(dir), lines.map((e) => JSON.stringify(e)).join("\n") + "\n"); + + let threw = null; + try { + chain.reconcileHead(dir); + } catch (error) { + threw = error; + } + check("reconcile-tampered-chain-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", threw ? `${threw.code}: ${threw.message}` : "did not throw"); +} + +function reconcileHeadRefusesEmptyChainWithHeadPresent() { + const dir = freshDir("reconcile-empty-with-head"); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: "1.0", seq: 1, bundle_id: "ghost", entry_sha256: "a".repeat(64) })); + // chain.jsonl itself does not exist -- HEAD.json names a tail that cannot possibly + // exist. Zero entries, but NOT the empty-chain-is-fine case (C2 row 2). + + let threw = null; + try { + chain.reconcileHead(dir); + } catch (error) { + threw = error; + } + check("reconcile-empty-with-head-refuses", threw !== null && threw.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", threw ? `${threw.code}: ${threw.message}` : "did not throw"); +} + +/* ================================================================================== + * Round-N fix (frontier-panel finding #8 checklist): streamChainFileLines (the shared + * chunked-byte-read primitive readChain/readChainForValidation are built on) tested + * directly at the primitive level -- chunk-boundary correctness, UTF-8 safety, renewal + * firing during the read itself (not only after it), immediate abort on lost ownership + * mid-read (the competing-claimant-after-a-stale-lease case), and unchanged behavior for + * a file with no trailing newline / an absent file. Exported specifically so these can be + * driven with small, fast, deterministic chunkBytes overrides instead of a multi-hundred- + * MB fixture (checklist: "a mocked fs / small chunk size that pretends >512MB without + * checking in a gigabyte fixture" -- streamChainFileLines's own memory profile is + * O(chunk size), not O(file size), so proving correctness across MANY small forced chunk + * boundaries generalizes to any file size by construction, without needing an actual + * multi-hundred-MB file on disk). + * ================================================================================== */ + +function collectStreamedLines(dir, options) { + const lines = []; + const progress = []; + chain.streamChainFileLines(dir, { + ...options, + onLine: (buf, idx, meta) => lines.push({ idx, meta, text: buf ? buf.toString("utf8") : null }), + onProgress: (n) => progress.push(n), + }); + return { lines, progress }; +} + +function streamChainFileLinesHandlesMultiByteUtf8SplitAcrossATinyChunkBoundary() { + const dir = freshDir("stream-utf8-chunk"); + const entries = [ + { schema_version: "1.0", seq: 1, bundle_id: "plain-ascii" }, + { schema_version: "1.0", seq: 2, bundle_id: "résumé-日本語-emoji-🎉-more" }, + { schema_version: "1.0", seq: 3, bundle_id: "back-to-ascii" }, + ]; + writeChainFile(dir, entries.map((e) => JSON.stringify(e)).join("\n") + "\n"); + + // chunkBytes=3 forces the raw byte reader to split well inside the multi-byte UTF-8 + // characters above on nearly every read -- streamChainFileLines only ever decodes a + // complete assembled LINE (never a mid-chunk partial buffer), so this must not corrupt + // the decoded text regardless of exactly where each chunk boundary lands. + const { lines, progress } = collectStreamedLines(dir, { chunkBytes: 3 }); + check("stream-utf8-forces-many-physical-chunks", progress.length > 10, `only ${progress.length} chunk(s) -- fixture too small to prove the boundary case`); + check("stream-utf8-parses-all-three-lines", lines.length === 3, JSON.stringify(lines)); + check( + "stream-utf8-multibyte-content-survives-the-chunk-split-intact", + lines.every((l, i) => !l.meta.oversized && JSON.parse(l.text).bundle_id === entries[i].bundle_id), + JSON.stringify(lines) + ); +} + +function streamChainFileLinesRenewsOncePerPhysicalChunkAcrossManyChunks() { + const dir = freshDir("stream-renew-many-chunks"); + const entries = Array.from({ length: 8 }, (_, i) => ({ schema_version: "1.0", seq: i + 1, bundle_id: `chunked-renew-${i}` })); + writeChainFile(dir, entries.map((e) => JSON.stringify(e)).join("\n") + "\n"); + + let renewCalls = 0; + const lines = []; + chain.streamChainFileLines(dir, { + chunkBytes: 5, // tiny -- forces dozens of physical read syscalls for this small fixture + onProgress: () => { renewCalls++; }, + onLine: (buf) => lines.push(JSON.parse(buf.toString("utf8"))), + }); + check("stream-renew-parses-every-entry-despite-tiny-chunks", lines.length === 8 && lines.every((l, i) => l.bundle_id === entries[i].bundle_id), JSON.stringify(lines)); + check( + "stream-renew-fires-more-than-once-i.e.-genuinely-during-the-read-not-only-once-at-the-end", + renewCalls > 8, + `expected many renew opportunities across chunk boundaries, got ${renewCalls}` + ); +} + +/* The competing-claimant-after-a-stale-lease case at the primitive level: once + * onProgress (the caller's maybeRenew hook) detects lost ownership mid-read and throws, + * the read must abort AT THAT POINT -- no further physical chunks are read and no further + * lines are handed to onLine. This is what actually prevents the former (now-stale) owner + * from completing its read and going on to act as if it still held the claim: the error + * propagates by reference, unmodified, out of the read entirely. */ +function streamChainFileLinesAbortsImmediatelyWhenOnProgressDetectsLostOwnership() { + const dir = freshDir("stream-lost-ownership-mid-read"); + const entries = Array.from({ length: 6 }, (_, i) => ({ schema_version: "1.0", seq: i + 1, bundle_id: `pre-takeover-${i}` })); + writeChainFile(dir, entries.map((e) => JSON.stringify(e)).join("\n") + "\n"); + + const takeoverError = Object.assign(new Error("simulated competing claimant won the lease after an over-TTL stall"), { code: "WRITER_CLAIM_LOST" }); + let progressCalls = 0; + const linesSeenBeforeAbort = []; + let threw = null; + try { + chain.streamChainFileLines(dir, { + chunkBytes: 4, // many chunks -- so the second progress call is still mid-file, not the last one + onLine: (buf) => linesSeenBeforeAbort.push(buf ? buf.toString("utf8") : null), + onProgress: () => { + progressCalls++; + if (progressCalls === 2) throw takeoverError; + }, + }); + } catch (error) { + threw = error; + } + check("stream-lost-ownership-propagates-the-exact-error-by-reference", threw === takeoverError, threw && threw.message); + check( + "stream-lost-ownership-does-not-finish-reading-the-rest-of-the-file", + linesSeenBeforeAbort.length < entries.length, + `expected an early abort (fewer than ${entries.length} lines), got ${linesSeenBeforeAbort.length}` + ); +} + +function streamChainFileLinesToleratesFileWithNoTrailingNewline() { + const dir = freshDir("stream-no-trailing-newline"); + const entries = [ + { schema_version: "1.0", seq: 1, bundle_id: "first" }, + { schema_version: "1.0", seq: 2, bundle_id: "last-no-newline" }, + ]; + // Deliberately no trailing "\n" -- the final unterminated line is still a real record. + writeChainFile(dir, entries.map((e) => JSON.stringify(e)).join("\n")); + + const { lines } = collectStreamedLines(dir, {}); + check("stream-no-trailing-newline-still-parses-both-lines", lines.length === 2, JSON.stringify(lines)); + check( + "stream-no-trailing-newline-final-unterminated-line-is-not-dropped", + lines[1] && JSON.parse(lines[1].text).bundle_id === "last-no-newline", + JSON.stringify(lines) + ); +} + +function streamChainFileLinesNoOpWhenChainFileIsAbsent() { + const dir = freshDir("stream-absent-file"); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); // directory exists, chain.jsonl does not + const { lines, progress } = collectStreamedLines(dir, {}); + check("stream-absent-file-calls-onLine-zero-times", lines.length === 0, JSON.stringify(lines)); + check("stream-absent-file-calls-onProgress-zero-times", progress.length === 0, JSON.stringify(progress)); +} + +/* Locks in the finding #8 checklist's distinct-error-code requirement: an oversized + * record is a size/resource condition (GATEWAY_CHAIN_ENTRY_TOO_LARGE), never conflated + * with CORRUPT_GATEWAY_CHAIN -- the catch-all reserved for genuine content corruption + * (malformed JSON, bad hash linkage). Both halves are pinned so a future edit cannot + * silently re-merge the two failure classes. */ +function readChainRejectsOversizedRecordWithADistinctCode() { + const dir = freshDir("readchain-oversized"); + const bigBundleId = "x".repeat(chain.MAX_ENTRY_BYTES + 100); + writeChainFile(dir, JSON.stringify({ schema_version: "1.0", seq: 1, bundle_id: bigBundleId }) + "\n"); + let threw = null; + try { + chain.readChain(dir); + } catch (error) { + threw = error; + } + check( + "readchain-oversized-record-throws-a-distinct-too-large-code-not-corrupt", + threw !== null && threw.code === "GATEWAY_CHAIN_ENTRY_TOO_LARGE", + threw ? `${threw.code}: ${threw.message}` : "did not throw" + ); +} + +function readChainStillLabelsGenuineJsonCorruptionAsCorrupt() { + const dir = freshDir("readchain-corrupt-json"); + writeChainFile(dir, '{"schema_version": "1.0", "seq": 1, "bundle_id": "ok"}\n{not even close to json\n'); + let threw = null; + try { + chain.readChain(dir); + } catch (error) { + threw = error; + } + check( + "readchain-malformed-json-record-still-reports-corrupt-not-too-large", + threw !== null && threw.code === "CORRUPT_GATEWAY_CHAIN", + threw ? `${threw.code}: ${threw.message}` : "did not throw" + ); +} + +/* Integration-level sanity check through the PUBLIC readChain API (default 1MB chunking, + * no chunkBytes override) with a real file large enough to force more than one physical + * chunk: proves the default production configuration -- not just the primitive with a + * tiny forced chunkBytes -- genuinely spans multiple chunks and still parses every entry + * correctly, and that onProgress (the lease-renewal hook both real call sites wire in) + * actually fires more than once for it. This is the closest cheap proxy for "a chain + * exceeding the old whole-file materialization ceiling parses successfully": the read + * path here is provably chunked (progress.length > 1), and streamChainFileLines's own + * memory profile is chunk-size-invariant, so this generalizes to any file size without + * needing an actual 512MB+ fixture checked into the repo. */ +function readChainHandlesARealMultiChunkFileThroughTheDefaultPublicApi() { + const dir = freshDir("readchain-multi-chunk-real"); + // ~40 entries of ~35KB padding each => comfortably over the default 1MB chunk size, + // each individual entry still well under MAX_ENTRY_BYTES (64KB) so none is rejected. + const entries = Array.from({ length: 40 }, (_, i) => ({ schema_version: "1.0", seq: i + 1, bundle_id: `multi-chunk-${i}`, padding: "p".repeat(35 * 1024) })); + writeChainFile(dir, entries.map((e) => JSON.stringify(e)).join("\n") + "\n"); + + const progress = []; + const result = chain.readChain(dir, { onProgress: (n) => progress.push(n) }); + check("readchain-multi-chunk-real-parses-every-entry", result.length === 40 && result.every((e, i) => e.bundle_id === entries[i].bundle_id), `parsed ${result.length} of 40`); + check("readchain-multi-chunk-real-genuinely-spans-more-than-one-physical-chunk", progress.length > 1, `expected >1 chunk read, got ${progress.length}`); +} + function main() { tenSessionsInSequence(); fixtureDataVerifiesIndependently(); @@ -195,6 +1306,81 @@ function main() { bundleIdCollisionRefused(); emptyChainIsNotApplicable(); + validateChainEmptyGenesisIsNotAFailure(); + validateChainEmptyWithHeadRefuses(); + validateChainDetectsSequenceGap(); + validateChainDetectsDuplicateSeqAsFork(); + validateChainDetectsTamperedEntry(); + validateChainDetectsBrokenLink(); + validateChainDetectsInvalidGenesisSeq(); + validateChainDetectsInvalidGenesisPrev(); + validateChainRefusesMalformedInteriorRecordNotJustTheLast(); + validateChainAutoAdvancesSingleStepLaggingHead(); + validateChainRefusesMultiStepLaggingHead(); + validateChainRefusesHeadAheadOfTail(); + validateChainRefusesHeadSameSeqDifferentHashAsFork(); + validateChainRefusesOneStepBackHeadThatIsNotAGenuineAncestor(); + validateChainRebuildsMissingHeadFromVerifiedTail(); + validateChainDoesNotRebuildWhenTheChainItselfDoesNotValidate(); + validateChainTruncatesTornFinalRecordAndRebuildsHead(); + validateChainTornFinalRecordWithHeadAlreadyAtTheTruncatedTailStillReportsReconciliation(); + validateChainTornSoleGenesisRecordWithNoHeadReconcilesToEmptyTail(); + validateChainTornSoleRecordWithHeadPresentRefuses(); + validateChainNeverConflatesTornTailWithInteriorCorruption(); + + readChainTailAbsentWhenFileDoesNotExist(); + readChainTailEmptyWhenFileIsZeroBytes(); + readChainTailEmptyWhenFileIsOnlyBlankLines(); + readChainTailSucceedsWithTrailingNewline(); + readChainTailSucceedsWithoutTrailingNewline(); + readChainTailReturnsTheSameEntryRegardlessOfTrailingNewline(); + readChainTailSkipsBlankLinesBeforeTheLastEntry(); + readChainTailFindsTheLastEntryWhenTheWindowStartsMidRecord(); + readChainTailGrowsPastTheInitialWindowForAnOversizedFinalRecord(); + readChainTailFindsALargeFinalRecordWhenANewlineIsAlreadyInTheFirstWindow(); + readChainTailFailsClosedWhenNoNewlineIsFoundWithinTheCappedWindow(); + readChainTailReportsCorruptForATruncatedFinalRecord(); + readChainTailReportsCorruptForAMalformedFinalRecord(); + readChainTailReportsCorruptWhenTheFinalRecordFailsItsShapeCheck(); + readChainTailHandlesUtf8BoundariesSplitAcrossTheWindowEdge(); + readChainTailUtf8LastEntryItselfSurvivesGrowth(); + readChainTailHandlesShortReads(); + readChainTailAbsentEmptyCorruptAreThreeDistinctOutcomes(); + + appendTimeCheckAutoAdvancesASingleStepLaggingHead(); + statusWalkAndAppendTimeCheckAgreeOnSingleStepLaggingHead(); + appendTimeCheckRefusesATwoStepLaggingHead(); + appendTimeCheckRefusesAForkedHead(); + checkHeadAgainstTailOperationalPermissionFailureRefusesButDoesNotLatch(); + checkHeadAgainstTailGenuineUnreadableTailStillLatches(); + reconcileHeadEmptyChainIsANoOpWithZeroWrites(); + reconcileHeadRefusesAMultiStepLaggingHeadAtStartupToo(); + reconcileHeadRebuildsFromAMalformedHeadFile(); + reconcileHeadRefusesAGenuineFork(); + reconcileHeadAheadRefuses(); + reconcileHeadRefusesWhenChainFailsStructuralValidation(); + reconcileHeadRefusesEmptyChainWithHeadPresent(); + + // Codex PR #33 review round 2 (2026-09-17): startup-walk lease keepalive. + reconcileHeadRenewsTheClaimPerEntryAcrossTheStartupWalk(); + reconcileHeadRenewsTheClaimAgainImmediatelyBeforeTheRepairWrite(); + reconcileHeadPerEntryLeaseLossAbortsBeforeTheRepairWrite(); + + // Round-N fix (frontier-panel finding #8 checklist): streamChainFileLines primitive + + // readChain error-code tests. + streamChainFileLinesHandlesMultiByteUtf8SplitAcrossATinyChunkBoundary(); + streamChainFileLinesRenewsOncePerPhysicalChunkAcrossManyChunks(); + streamChainFileLinesAbortsImmediatelyWhenOnProgressDetectsLostOwnership(); + streamChainFileLinesToleratesFileWithNoTrailingNewline(); + streamChainFileLinesNoOpWhenChainFileIsAbsent(); + readChainRejectsOversizedRecordWithADistinctCode(); + readChainStillLabelsGenuineJsonCorruptionAsCorrupt(); + readChainHandlesARealMultiChunkFileThroughTheDefaultPublicApi(); + + walkGatewaySessionsCallsMaybeRenewOncePerEntry(); + walkGatewaySessionsMaybeRenewInterleavedWithValidateChainNotFrontLoaded(); + walkAbortsOnLeaseTakeoverMidWalkNotDowngradedToAFailedReport(); + const passed = results.filter((r) => r.status === "PASS").length; const failed = results.filter((r) => r.status === "FAIL").length; console.log(`SUMMARY passed=${passed} failed=${failed} skipped=0`); diff --git a/tests/gateway/e2e/run-tests.js b/tests/gateway/e2e/run-tests.js index d1c007af..9f865677 100644 --- a/tests/gateway/e2e/run-tests.js +++ b/tests/gateway/e2e/run-tests.js @@ -36,6 +36,8 @@ const { writeConfirmedMode } = require("../_fixtures/mode-file.js"); const { walkGatewaySessions } = require(path.join(ROOT, "checks", "register-gateway-sessions.js")); const chain = require(path.join(ROOT, "scripts", "gateway", "chain.js")); const { verifyBundle } = require(path.join(ROOT, "scripts", "gsa-verify.js")); +const recovery = require(path.join(ROOT, "scripts", "gateway", "recovery.js")); +const { WriterClaim } = require(path.join(ROOT, "scripts", "writer-claim.js")); let failures = 0; const results = []; @@ -582,6 +584,115 @@ async function httpAgentSessionsAreIdBasedNotSocketBased() { await gw.exitCode(); } +/** Polls the on-disk quarantine directory (proxy.js#quarantineSealedBundle) rather + * than sleeping a fixed duration -- matches this suite's own "signal, not sleep" + * discipline (see waitForHttpPort above). A quarantined bundle appearing is the + * externally-observable proof that a real closeConnection ran chain.appendSession, + * hit the genuine chain-integrity failure, and (per round-1 fix-plan commit 6) latched + * admission -- agent-transport.js's own DELETE handler fires closeConnection + * fire-and-forget (never awaited before the 204 response), so this cannot be inferred + * from the DELETE response alone. */ +function waitForQuarantinedBundle(stateDir, timeoutMs = 5000) { + const quarantineDir = path.join(chain.sessionsDir(stateDir), "quarantine"); + return new Promise((resolve, reject) => { + const deadline = Date.now() + timeoutMs; + const poll = () => { + if (fs.existsSync(quarantineDir) && fs.readdirSync(quarantineDir).length > 0) { + resolve(); + return; + } + if (Date.now() > deadline) { + reject(new Error("timed out waiting for a quarantined bundle to appear")); + return; + } + setTimeout(poll, 25); + }; + poll(); + }); +} + +function httpDelete(port, token, sessionId) { + return new Promise((resolve, reject) => { + const req = http.request({ host: "127.0.0.1", port, method: "DELETE", headers: { authorization: `Bearer ${token}`, "mcp-session-id": sessionId } }, (res) => { + res.resume(); + res.on("end", () => resolve(res.statusCode)); + }); + req.on("error", reject); + req.end(); + }); +} + +/** Round-1 fix-plan commit 6, full-stack wiring proof: a genuine chain-integrity + * failure discovered by a RUNNING gateway's own append-time check (commit 4, + * chain.js#checkHeadAgainstTailOrRepair) must immediately refuse every subsequent new + * session admission on that same process -- not merely be recorded for the next + * restart. Uses the HTTP agent transport (unlike the stdio-based tests above, it + * supports multiple independent, still-running sessions on one live gateway process), + * and corrupts HEAD.json on disk from this TEST process -- deliberately not via any + * chain.js private test hook, since chain.js's own process-local latch lives inside + * the SEPARATE gateway subprocess and can only actually be exercised by making that + * subprocess's own code discover the divergence itself. */ +async function chainIntegrityLatchRefusesNewHttpSessionAdmission() { + const root = freshRoot("chain-integrity-http"); + writeConfirmedMode(root, "standalone"); + const tokenPath = path.join(root, "agent-token.txt"); + fs.writeFileSync(tokenPath, "a-fake-but-long-enough-bearer-token-value"); + const { configPath, stateDir } = writeGatewayConfig(root, { agent_listen: { transport: "http", token_ref: tokenPath } }); + const gw = spawnGateway(root, configPath); + const port = await waitForHttpPort(gw); + const token = fs.readFileSync(tokenPath, "utf8").trim(); + + // Session A: a normal, fully clean session, sealed via an explicit DELETE -- gives + // the chain a real, verified first entry to fork HEAD against below. + const initA = await httpPost(port, token, { jsonrpc: "2.0", id: 1, method: "initialize", params: { clientInfo: { name: "agent-A", version: "1.0" } } }); + const sessionA = initA.headers["mcp-session-id"]; + await httpNotify(port, token, { jsonrpc: "2.0", method: "notifications/initialized" }, undefined, sessionA); + await httpPost(port, token, { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "fixture_echo", arguments: {} } }, undefined, sessionA); + const deleteAStatus = await httpDelete(port, token, sessionA); + check("e2e-chain-integrity-session-a-deletes-cleanly", deleteAStatus === 204, String(deleteAStatus)); + + const realHead = chain.readHead(stateDir); + check("e2e-chain-integrity-fixture-session-a-actually-sealed", Boolean(realHead) && realHead.seq === 1, JSON.stringify(realHead)); + + // Corrupt HEAD.json into a genuine fork -- the same "worse than single-step lag" + // divergence commit 4's append-time check refuses and latches on -- directly on + // disk, exactly like the startup-hard-fail test above does for reconcileHead. + const forkedHead = Object.assign({}, realHead, { entry_sha256: "9".repeat(64) }); + fs.writeFileSync(chain.headPath(stateDir), JSON.stringify(forkedHead)); + + // Session B: admission itself is unaffected (nothing has re-checked HEAD since + // session A's own clean append) -- it is session B's own CLOSE that is the append + // which discovers the fork, refuses it (quarantining the sealed bundle), and + // latches admission for the whole process. + const initB = await httpPost(port, token, { jsonrpc: "2.0", id: 1, method: "initialize", params: { clientInfo: { name: "agent-B", version: "1.0" } } }); + const sessionB = initB.headers["mcp-session-id"]; + check("e2e-chain-integrity-session-b-opens-fine-before-the-close-time-check-reruns", typeof sessionB === "string" && sessionB.length > 0, JSON.stringify(initB.body)); + await httpNotify(port, token, { jsonrpc: "2.0", method: "notifications/initialized" }, undefined, sessionB); + await httpPost(port, token, { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "fixture_echo", arguments: {} } }, undefined, sessionB); + const deleteBStatus = await httpDelete(port, token, sessionB); + check("e2e-chain-integrity-session-b-delete-still-returns-204-even-though-its-append-was-refused", deleteBStatus === 204, String(deleteBStatus)); + + // Wait for externally-observable proof that closeConnection's own chain.appendSession + // ran and was refused (see waitForQuarantinedBundle's own doc comment on why this + // cannot be inferred from the DELETE response alone). + await waitForQuarantinedBundle(stateDir); + check("e2e-chain-integrity-chain-not-advanced-past-session-a", chain.readChain(stateDir).length === 1, JSON.stringify(chain.readChain(stateDir).map((e) => e.seq))); + + // Session C: a brand-new admission attempt AFTER the latch -- commit 6's own primary + // enforcement point. Must be refused immediately by openConnection, surfaced by + // agent-transport.js's existing openSession() try/catch as a 503 naming the reason. + const initC = await httpPost(port, token, { jsonrpc: "2.0", id: 1, method: "initialize", params: { clientInfo: { name: "agent-C", version: "1.0" } } }); + check( + "e2e-chain-integrity-latch-refuses-new-session-admission", + Boolean(initC.body && initC.body.error && /chain-integrity/i.test(initC.body.error.message) && /GATEWAY_CHAIN_INTEGRITY_FAILED|chain-integrity failure is latched/i.test(initC.body.error.message)), + JSON.stringify(initC.body) + ); + check("e2e-chain-integrity-latch-mints-no-session-id-for-the-refused-attempt", !initC.headers["mcp-session-id"], JSON.stringify(initC.headers)); + + gw.child.kill(); + await gw.exitCode(); +} + /** Codex PR #29 review "validate initialize before allocating an HTTP session": a * headerless request whose method is "initialize" but whose envelope is otherwise * malformed (missing "jsonrpc": "2.0" here) previously still got a real session with a @@ -714,6 +825,62 @@ async function cleanSigtermDrainsAndExitsZero() { check("e2e-sigterm-session-still-finalized-and-persisted", head && head.seq === 1, JSON.stringify(head)); } +/* Codex PR #33 review "release the writer claim when intent cleanup aborts shutdown": + * closeConnection's own recovery.listIntentsForConnection can throw on a genuinely + * unreadable/corrupt intent FILE (a real fs-level problem, distinct from the content + * issues it already tolerates) -- previously unguarded in gateway.js's shutdown loop, so + * that exception escaped doStop() entirely, skipping every remaining step including + * writerClaim.release() and leaking a stale claim that blocks the next start. + * + * Round-N fix (frontier-panel finding #1) changed WHERE this failure surfaces: before + * that fix, closeConnection's own intent-fencing loop threw before this.sessions.delete + * ran, leaving a ghost session that gateway.js's shutdown loop (iterating + * proxy.sessions.keys()) would then redundantly retry and log via its own "SHUTDOWN + * CLOSE FAILURE" wording -- this test originally asserted on THAT retry's log line. + * Finding #1's fix makes closeConnection always clean up this.sessions/agentInitialized + * (try/finally), which removes the ghost session and therefore the redundant shutdown- + * loop retry entirely: the SAME fixture's single stdio-disconnect handler call now fails, + * cleans up, and is the ONLY attempt -- so it never reaches gateway.js's shutdown loop at + * all (the connection is already gone from proxy.sessions by the time that loop runs). + * agent-transport.js's own stdio-disconnect catch was updated in the same round to log + * this (previously-silent) failure via "CONNECTION CLOSE FAILURE" instead, so this test + * now looks for that wording -- still proving the failure was hit and handled, just via + * its new (and, post-fix, only) logging surface rather than the old redundant one. */ +async function sigtermStillReleasesClaimWhenAConnectionsCloseFails() { + if (process.platform === "win32") { + skip("e2e-sigterm-releases-claim-despite-close-failure", "Windows cannot deliver a real SIGTERM for graceful in-process handling -- same platform limitation as the other SIGTERM test."); + return; + } + const root = freshRoot("sigterm-close-failure"); + writeConfirmedMode(root, "standalone"); + const { configPath, stateDir } = writeGatewayConfig(root); + const gw = spawnGateway(root, configPath); + + gw.send({ jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + const initResp = await gw.nextMessage(); + check("e2e-close-failure-preinitialize-succeeded", initResp && initResp.id === 0 && initResp.result, JSON.stringify(initResp)); + + // Corrupt an intent record on disk -- listIntentsForConnection's own readdirSync scan + // reads EVERY intent file regardless of which connection it names, so this breaks the + // scan for the live stdio connection's own closeConnection() at shutdown, the same way + // a real fs-level I/O error would (a directory where a file is expected reproduces + // EISDIR, mirroring this suite's own existing "unreadable WAL" fixtures elsewhere). + fs.mkdirSync(recovery.intentsDir(stateDir), { recursive: true }); + fs.mkdirSync(recovery.intentPath(stateDir, "corrupt-intent")); + + gw.child.kill("SIGTERM"); + const code = await gw.exitCode(); + check("e2e-close-failure-still-exits-zero", code === 0, `exit code ${code}; stderr: ${gw.stderr()}`); + check("e2e-close-failure-logged-the-isolated-failure", gw.stderr().includes("CONNECTION CLOSE FAILURE"), gw.stderr()); + + // Decisive proof the claim was actually released (not merely that the process exited): + // a brand-new instance must be able to acquire it immediately, with no stale lease wait. + const fresh = new WriterClaim(stateDir, { hostId: "post-shutdown-check" }); + let acquireError = null; + try { fresh.acquire(); fresh.release(); } catch (error) { acquireError = error; } + check("e2e-close-failure-writer-claim-still-released", acquireError === null, acquireError && acquireError.message); +} + /* Cluster E (partial fix): checkModeGate's root must track configPath's own directory, * not process.cwd() -- `gateway.js --config /path/to/project-b/gateway.json` run with * cwd somewhere else entirely must still validate project B's own @@ -814,10 +981,12 @@ async function main() { await agentHttpListenerRejectsNonPostMethod(); await agentHttpListenerBindFailureRejectedCleanly(); await httpAgentSessionsAreIdBasedNotSocketBased(); + await chainIntegrityLatchRefusesNewHttpSessionAdmission(); await httpFailedInitializeDoesNotLeakSession(); await modeDormantExitsZero(); await secondInstanceRefused(); await cleanSigtermDrainsAndExitsZero(); + await sigtermStillReleasesClaimWhenAConnectionsCloseFails(); await modeGateRootTracksConfigPathNotCwd(); await statusCommandReadsRunningGatewaysStatusFile(); diff --git a/tests/gateway/lifecycle/run-tests.js b/tests/gateway/lifecycle/run-tests.js index e8dea613..35cb32a8 100644 --- a/tests/gateway/lifecycle/run-tests.js +++ b/tests/gateway/lifecycle/run-tests.js @@ -22,6 +22,8 @@ const { drainOpenSessions, startGateway } = require(path.join(ROOT, "scripts", " const { GatewayProxy } = require(path.join(ROOT, "scripts", "gateway", "proxy.js")); const { WriterClaim } = require(path.join(ROOT, "scripts", "writer-claim.js")); const { writeConfirmedMode } = require("../_fixtures/mode-file.js"); +const recovery = require(path.join(ROOT, "scripts", "gateway", "recovery.js")); +const FIXTURE_SERVER = path.join(ROOT, "tests", "gateway", "_fixtures", "fixture-mcp-server.js"); let failures = 0; const results = []; @@ -246,7 +248,13 @@ async function samplingForwardStillWorksBelowTheCap() { const { forwardDownstreamRequestToAgent } = require(path.join(ROOT, "scripts", "gateway", "gateway.js")); const sessionModule = require(path.join(ROOT, "scripts", "gateway", "session.js")); const s = sessionModule.createSession("agent-conn-2"); - const proxy = { sessions: new Map([["agent-conn-2", s]]), now: () => Date.now() }; + // Cluster H (frontier-panel review, 2026-09-14): forwardDownstreamRequestToAgent now + // unconditionally appends a CALL_START/CALL_RESULT pair to the recovery WAL for this + // path too (PR #33), so this fake proxy needs a real stateDir for recovery.appendWalEvent + // to write into -- this test predates that addition and previously got away with a + // bare {sessions, now} stand-in. + const stateDir = freshDir("sampling-forward-below-cap"); + const proxy = { sessions: new Map([["agent-conn-2", s]]), now: () => Date.now(), stateDir }; const agentPusher = { current: () => Promise.resolve({ role: "assistant", content: { type: "text", text: "ok" } }), connectionId: "agent-conn-2" }; const resp = await forwardDownstreamRequestToAgent( @@ -260,6 +268,69 @@ async function samplingForwardStillWorksBelowTheCap() { check("sampling-forward-records-the-completed-call-below-the-cap", s.calls.length === 1 && s.calls[0].model_call === true, JSON.stringify(s.calls)); } + +/* Codex PR #33 review "prevent shutdown diagnostics from aborting claim release": an + * earlier round of this PR wrapped each connection's closeConnection in a try/catch so one + * failing session could not abort shutdown -- but the HANDLER itself called the + * caller-supplied `log` callback directly. `log` is caller plumbing and is allowed to + * throw, so a throwing logger reintroduced the exact failure one level up: doStop rejected + * before the remaining sessions, the downstream closes, and writerClaim.release() ever ran, + * leaving an embedding process renewing an orphaned claim indefinitely. */ +async function aThrowingLoggerDuringShutdownStillReleasesTheWriterClaim() { + const root = freshDir("shutdown-throwing-logger"); + writeConfirmedMode(root, "standalone"); + const kp = crypto.generateKeyPairSync("ed25519"); + const keyPath = path.join(root, "signing-key.pem"); + fs.writeFileSync(keyPath, kp.privateKey.export({ type: "pkcs8", format: "pem" }).toString()); + const stateDir = path.join(root, "state"); + const tokenPath = path.join(root, "agent-token.txt"); + fs.writeFileSync(tokenPath, "shutdown-logger-test-token"); + const configPath = path.join(root, "gateway-config.json"); + fs.writeFileSync(configPath, JSON.stringify({ + schema_version: "1.0", + state_dir: stateDir, + downstream_servers: [{ name: "fixture", transport: "stdio", endpoint: `${process.execPath} ${FIXTURE_SERVER} --server-name fixture` }], + signing_key_ref: keyPath, + /* Deliberately the HTTP agent transport, not stdio: the stdio branch wires + * `stdioHandle.closed -> stop() -> process.exit(0)`, so stopping a stdio gateway + * in-process would terminate this whole test runner before it could report. The + * shutdown path under test (doStop's diagnostics and writerClaim.release()) is + * transport-independent. */ + agent_listen: { transport: "http", token_ref: tokenPath }, + }, null, 2)); + + // A logger that works during startup (so the gateway actually starts) but throws for + // every shutdown diagnostic -- precisely the caller-supplied-plumbing failure mode. + let shuttingDown = false; + const log = () => { if (shuttingDown) throw new Error("caller-supplied logger exploded during shutdown"); }; + + const handle = await startGateway({ configPath, root, log }); + check("throwing-logger-shutdown-gateway-actually-started", Boolean(handle) && handle.dormant !== true, JSON.stringify(handle && { dormant: handle.dormant })); + + // Open a real session AND make its close fail, so the catch handler (the one that used + // to call the throwing logger directly) is genuinely entered during shutdown. + handle.proxy.openConnection("conn-shutdown"); + await handle.proxy.handleMessage("conn-shutdown", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + fs.mkdirSync(recovery.intentsDir(stateDir), { recursive: true }); + fs.mkdirSync(recovery.intentPath(stateDir, "corrupt-intent"), { recursive: true }); + + shuttingDown = true; + let stopError = null; + try { + await handle.stop("test shutdown"); + } catch (error) { + stopError = error; + } + check("throwing-logger-during-shutdown-does-not-reject-stop", stopError === null, stopError && stopError.message); + + // Decisive proof the claim was actually released: a brand-new instance must be able to + // acquire it immediately, with no stale-lease wait. + const fresh = new WriterClaim(stateDir, { hostId: "post-shutdown-check" }); + let acquireError = null; + try { fresh.acquire(); fresh.release(); } catch (error) { acquireError = error; } + check("throwing-logger-during-shutdown-still-releases-the-writer-claim", acquireError === null, acquireError && acquireError.message); +} + async function main() { await drainWaitsForInFlightCallToComplete(); await drainTimesOutAndReportsIncomplete(); @@ -267,6 +338,7 @@ async function main() { await samplingForwardRefusesOnceCompletedCallCapReached(); await samplingForwardStillWorksBelowTheCap(); await startupWatchdogTimesOutOnHungDownstreamConnect(); + await aThrowingLoggerDuringShutdownStillReleasesTheWriterClaim(); const passed = results.filter((r) => r.status === "PASS").length; const failed = results.filter((r) => r.status === "FAIL").length; diff --git a/tests/gateway/proxy/run-tests.js b/tests/gateway/proxy/run-tests.js index 28a523e1..2a06f2c0 100644 --- a/tests/gateway/proxy/run-tests.js +++ b/tests/gateway/proxy/run-tests.js @@ -23,6 +23,8 @@ const path = require("path"); const ROOT = path.resolve(__dirname, "../../.."); const { GatewayProxy, isMalformedToolCallResult } = require(path.join(ROOT, "scripts", "gateway", "proxy.js")); const chain = require(path.join(ROOT, "scripts", "gateway", "chain.js")); +const recovery = require(path.join(ROOT, "scripts", "gateway", "recovery.js")); +const session = require(path.join(ROOT, "scripts", "gateway", "session.js")); let failures = 0; const results = []; @@ -74,6 +76,26 @@ function fakeConnection(impl) { }; } +/** Like fakeConnection, but also records every call's full argument list -- needed for + * the Option C tests below to assert on `idempotencyKey` (call()'s 5th argument) and on + * how many times a logical operation was actually dispatched downstream. */ +function fakeConnectionCapturing(impl) { + let closed = false; + const calls = []; + return { + transport: "fake", + calls, + call: async (method, params, timeoutMs, onIdAssigned, idempotencyKey) => { + calls.push({ method, params, idempotencyKey }); + if (closed) throw Object.assign(new Error("closed"), { code: "GATEWAY_DOWNSTREAM_DISCONNECTED" }); + return impl(method, params); + }, + close: () => { closed = true; }, + isClosed: () => closed, + whenClosed: () => new Promise(() => {}), + }; +} + function makeProxy(dir, connections, mergedTools, toolOwners, extra) { return new GatewayProxy({ connections, @@ -289,6 +311,63 @@ async function stopAcceptingNewSessionsRefusesNewButNotExisting() { check("already-open-session-still-finalizes-after-stop", entry !== null, "existing session was not finalized"); } +/* Round-1 fix-plan commit 6: getChainIntegrityFailure defaults to `() => null` + * (never latched), mirroring isWriterClaimValid's own always-valid default -- every + * existing test in this file that constructs a bare GatewayProxy (no chain-integrity + * concern of its own) must stay unaffected by this commit. */ +function getChainIntegrityFailureDefaultsToNeverLatchedSoBareProxiesAreUnaffected() { + const dir = freshDir("chain-integrity-default"); + const proxy = makeProxy(dir, new Map(), [], new Map()); + let threw = null; + try { proxy.openConnection("conn-1"); } catch (error) { threw = error; } + check("chain-integrity-default-does-not-block-admission", threw === null, threw && threw.message); + check("chain-integrity-default-session-actually-opened", proxy.sessions.has("conn-1"), "conn-1 missing from proxy.sessions"); +} + +/* Round-1 fix-plan commit 6: PRIMARY enforcement point. A latched chain-integrity + * failure must refuse every NEW admission (mirroring stopAcceptingNewSessions' + * own new-vs-existing split above) while never force-closing a session already open + * before the latch fired -- belt-and-braces on top of the append-time check, not a + * second drain mechanism. */ +async function chainIntegrityLatchRefusesNewAdmissionButNotExisting() { + const dir = freshDir("chain-integrity-latch"); + const conn = fakeConnection(async () => ({ ok: true })); + let latched = null; + const proxy = makeProxy(dir, new Map([["srv", conn]]), [], new Map(), { getChainIntegrityFailure: () => latched }); + proxy.openConnection("conn-1"); + + latched = { status: "refuse", class: "tampered", reason: "test: genuine structural corruption at seq 2", at: "2026-09-16T00:00:00.000Z" }; + let threw = null; + try { proxy.openConnection("conn-2"); } catch (error) { threw = error; } + check("chain-integrity-latch-refuses-new-admission", threw && threw.code === "GATEWAY_CHAIN_INTEGRITY_FAILED", threw && threw.code); + check( + "chain-integrity-latch-error-names-the-reason-not-a-generic-message", + threw && threw.message.includes("test: genuine structural corruption at seq 2") && threw.message.includes("class=tampered"), + threw && threw.message + ); + check("chain-integrity-latch-second-connection-never-published", !proxy.sessions.has("conn-2"), "conn-2 leaked into proxy.sessions despite the throw"); + + const entry = await proxy.closeConnection("conn-1", "already-open session still finalizes despite the latch"); + check("chain-integrity-latch-does-not-force-close-existing-sessions", entry !== null, "existing session was not finalized"); +} + +/* Round-1 fix-plan commit 6: the naming rationale itself -- getChainIntegrityFailure + * must be consulted for its DETAIL OBJECT (reason/class/at), not coerced to a bare + * boolean, so the thrown error can actually name what happened. */ +function chainIntegrityLatchCarriesTheDiagnosticDetailNotJustABoolean() { + const dir = freshDir("chain-integrity-detail"); + const proxy = makeProxy(dir, new Map(), [], new Map(), { + getChainIntegrityFailure: () => ({ status: "refuse", class: "sequence-gap", reason: "entry[4] seq=6 expected 5", at: "2026-09-16T01:02:03.000Z" }), + }); + let threw = null; + try { proxy.openConnection("conn-1"); } catch (error) { threw = error; } + check( + "chain-integrity-detail-message-includes-reason-class-and-timestamp", + Boolean(threw) && threw.message.includes("entry[4] seq=6 expected 5") && threw.message.includes("class=sequence-gap") && threw.message.includes("2026-09-16T01:02:03.000Z"), + threw && threw.message + ); +} + /* Board decision 2026-09-04, PR #29 review "honor MCP tool-level error results": a * `tools/call` result carrying `isError: true` on the RESULT itself (not a thrown * transport/RPC error) must be recorded as an error in the session, but the JSON-RPC @@ -490,13 +569,1337 @@ async function nullJsonRpcIdDoesNotCrash() { await proxy.closeConnection("conn-1", "test cleanup"); } +/* Option C (Codex PR #29 Finding 1 + 2, external-panel-reviewed design -- see + * option-c-hardened-design.md): recovery.js's WAL + idempotency-intent wiring inside + * proxy.js's real dispatch path. These use fakeConnectionCapturing (not the plain + * fakeConnection above) specifically to observe how many times a logical operation was + * actually dispatched downstream and what idempotency key (if any) it carried. */ + +async function idempotencyKeyPropagatedToRealToolCallOnly() { + const dir = freshDir("idem-key"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + const expectedKey = recovery.computeIntentKey("conn-1", "echo", { a: 1 }); + check( + "idempotency-key-propagated-to-downstream-tools-call", + conn.calls.length === 1 && conn.calls[0].idempotencyKey === expectedKey, + JSON.stringify(conn.calls) + ); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +async function inFlightRetryBlockedAsAmbiguousRetry() { + const dir = freshDir("in-flight-retry"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + // Deliberately not awaited: an async function body runs synchronously up to its first + // `await` (here, `conn.call(...)`), so by the time control returns to this line the + // intent has already been durably created as "dispatched" -- no extra tick needed. + const firstPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + const retryResp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + check("in-flight-retry-blocked-with-ambiguous-retry-code", retryResp && retryResp.error && retryResp.error.code === -32080, JSON.stringify(retryResp)); + check("in-flight-retry-did-not-redispatch-downstream", conn.calls.length === 1, JSON.stringify(conn.calls)); + resolveCall({ ok: true }); + await firstPromise; + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Frontier-panel decision (Paul, 2026-09-10, cluster E1): completed-intent replay now + * requires the retry to present the SAME caller-supplied idempotency key + * (params._meta.idempotencyKey) the original dispatch carried -- argument equality alone + * is no longer sufficient to trigger a silent replay. */ +async function completedCallReplaysCachedResultOnRetryWithMatchingKey() { + const dir = freshDir("replay"); + const conn = fakeConnectionCapturing(async () => ({ value: 42 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const params = { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "caller-key-1" } }; + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + const retry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params }); + check("retry-of-completed-call-replays-cached-result-with-matching-key", retry && retry.result && retry.result.value === 42, JSON.stringify(retry)); + check("retry-of-completed-call-did-not-redispatch-downstream-with-matching-key", conn.calls.length === 1, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* E1's actual fix: WITHOUT a caller-supplied idempotency key, a repeated identical call + * after completion is no longer silently replayed -- it dispatches as a new, independent + * call (this is precisely the two reviewers' finding: argument equality alone cannot + * distinguish an intentional second call from a lost-response retry). The intent's + * generation is bumped and the downstream-facing idempotency key is suffixed so a + * downstream implementing its own dedup does not mistake this for the earlier call. */ +async function retryOfCompletedCallWithoutKeyDispatchesIndependently() { + const dir = freshDir("no-key-supersede"); + const conn = fakeConnectionCapturing(async () => ({ value: 42 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const params = { name: "echo", arguments: { a: 1 } }; // no _meta.idempotencyKey + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + const retry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params }); + check("retry-without-key-redispatched-downstream", conn.calls.length === 2, JSON.stringify(conn.calls)); + check("retry-without-key-still-returns-a-real-result", Boolean(retry && retry.result && retry.result.value === 42), JSON.stringify(retry)); + const intentKey = recovery.computeIntentKey("conn-1", "echo", { a: 1 }); + check( + "retry-without-key-downstream-call-uses-generation-suffixed-key", + conn.calls[0].idempotencyKey === intentKey && conn.calls[1].idempotencyKey === `${intentKey}.g2`, + JSON.stringify(conn.calls) + ); + const finalIntent = recovery.readIntent(dir, intentKey); + check("retry-without-key-intent-generation-bumped", Boolean(finalIntent) && finalIntent.generation === 2 && finalIntent.state === "completed", JSON.stringify(finalIntent)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* A retry presenting a DIFFERENT idempotency key than the original dispatch is also not a + * caller-signaled retry of that specific attempt -- treated the same as no key at all. */ +async function retryWithMismatchedKeyDispatchesIndependently() { + const dir = freshDir("mismatched-key"); + const conn = fakeConnectionCapturing(async () => ({ value: 7 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "key-A" } } }); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "key-B" } } }); + check("retry-with-mismatched-key-redispatched-downstream", conn.calls.length === 2, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +async function ambiguousOutcomeBlocksRetryUntilOperatorResolves() { + const dir = freshDir("ambiguous"); + const conn = fakeConnectionCapturing(async () => { throw Object.assign(new Error("boom"), { code: "GATEWAY_DOWNSTREAM_RPC_ERROR" }); }); + const mergedTools = [{ name: "flaky", server: "srv", schema: {} }]; + const toolOwners = new Map([["flaky", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const first = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "flaky", arguments: {} } }); + check("first-attempt-surfaces-the-real-transport-error", Boolean(first && first.error), JSON.stringify(first)); + const retry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "flaky", arguments: {} } }); + check("ambiguous-retry-blocked-with-downstream-outcome-unknown-code", retry && retry.error && retry.error.code === -32081, JSON.stringify(retry)); + check("ambiguous-retry-did-not-redispatch-downstream", conn.calls.length === 1, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +async function closeConnectionFencesInFlightIntentAsAmbiguous() { + const dir = freshDir("close-fence"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const inFlight = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: {} } }); + const intentKey = recovery.computeIntentKey("conn-1", "slow", {}); + await proxy.closeConnection("conn-1", "agent hung up mid-call"); + const intent = recovery.readIntent(dir, intentKey); + check("close-connection-fences-in-flight-intent-as-ambiguous", Boolean(intent) && intent.state === "ambiguous", JSON.stringify(intent)); + resolveCall({ ok: true }); // let the now-orphaned call settle so nothing is left hanging + await inFlight.catch(() => {}); +} + +async function walRecordsLifecycleEventsAndIsCleanedUpOnClose() { + const dir = freshDir("wal-lifecycle"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: {} } }); + const eventsBeforeClose = recovery.readWalEvents(dir, "conn-1").map((e) => e.type); + check( + "wal-records-session-start-initialize-call-start-call-result-in-order", + JSON.stringify(eventsBeforeClose) === JSON.stringify(["SESSION_START", "INITIALIZE", "CALL_START", "CALL_RESULT"]), + JSON.stringify(eventsBeforeClose) + ); + await proxy.closeConnection("conn-1", "test cleanup"); + const eventsAfterClose = recovery.readWalEvents(dir, "conn-1"); + check("wal-deleted-after-a-clean-close", eventsAfterClose.length === 0, JSON.stringify(eventsAfterClose)); +} + +// --------------------------------------------------------------------------- +// PR #33 round-2 fixes. +// --------------------------------------------------------------------------- + +/* Codex PR #33 review "append blocked-retry anomalies to the WAL": a blocked duplicate + * used to be recorded in-memory only (session.recordAnomaly) -- a crash before this + * connection's own close would silently drop that attestation from a recovered/sealed + * bundle. Now also durably WAL-logged and given a normal structured completion log. */ +async function blockedRetryAnomalyIsDurablyRecorded() { + const dir = freshDir("blocked-anomaly-wal"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const logLines = []; + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { log: (line) => logLines.push(line) }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const firstPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + const walEvents = recovery.readWalEvents(dir, "conn-1"); + const anomalyEvent = walEvents.find((e) => e.type === "ANOMALY"); + check("blocked-retry-anomaly-appended-to-wal", Boolean(anomalyEvent) && anomalyEvent.kind === "GATEWAY_AMBIGUOUS_RETRY" && anomalyEvent.tool === "slow", JSON.stringify(walEvents)); + const parsedLogs = logLines.map((l) => { try { return JSON.parse(l); } catch (error) { return null; } }); + const blockedLog = parsedLogs.find((l) => l && l.event === "gateway_call_completed" && l.status === "blocked"); + check("blocked-retry-gets-a-structured-completion-log", Boolean(blockedLog) && blockedLog.tool === "slow", JSON.stringify(logLines)); + resolveCall({ ok: true }); + await firstPromise; + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Round-1 fix-round follow-up (2026-09-15, "anomaly cap tests and the WAL/disk-growth + * follow-up ticket", tying back to the still-open 2026-09-14 Cluster H recommendation): + * MAX_ANOMALIES_PER_SESSION (session.js) bounds session.anomalies -- the in-memory array + * that becomes the sealed bundle's anomalies -- but proxy.js's blocked-retry path (the + * "block" branch above) appends an ANOMALY event to the WAL UNCONDITIONALLY, every single + * time it is hit, regardless of what session.recordAnomaly returned. A blocked-retry + * attempt never goes through MAX_PENDING_CALLS_PER_SESSION or + * MAX_COMPLETED_CALLS_PER_SESSION either -- it is rejected before ever being admitted as a + * pending or completed call -- so nothing in this codebase bounds how many ANOMALY WAL + * lines one connection can generate. Proves both halves directly against the real + * dispatch path (not just session.js in isolation): the in-memory/sealed-bundle side + * plateaus at the cap while the WAL keeps growing past it. This is deliberately NOT + * treated as closing out the broader disk-growth risk -- see the WAL/disk-growth ticket + * in KNOWN-LIMITATIONS.md; neither this event-count cap nor WAL rotation alone bounds + * disk usage when event SIZE varies. */ +async function blockedRetryAnomaliesAreUncappedInTheWalDespiteTheSessionCap() { + const session = require(path.join(ROOT, "scripts", "gateway", "session.js")); + const { MAX_ANOMALIES_PER_SESSION } = session; + const dir = freshDir("anomaly-cap-vs-wal"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + // Keeps the original call in flight for the whole test, so every subsequent identical + // call is blocked as an ambiguous retry rather than dispatched or admitted anywhere. + const firstPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + await new Promise((resolve) => setImmediate(resolve)); + + const blockedAttempts = MAX_ANOMALIES_PER_SESSION + 5; + for (let i = 0; i < blockedAttempts; i++) { + const resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 100 + i, method: "tools/call", params: { name: "slow", arguments: { x: 1 } } }); + if (!(resp && resp.error && resp.error.code === -32080)) { + check("every-retry-in-this-loop-is-actually-blocked-as-ambiguous", false, `attempt ${i}: ${JSON.stringify(resp)}`); + break; + } + } + + const s = proxy.sessions.get("conn-1"); + check( + "session-side-anomalies-plateau-at-the-cap-plus-terminal-marker", + Boolean(s) && s.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1, + `length=${s && s.anomalies.length}, blockedAttempts=${blockedAttempts}` + ); + + const walAnomalyCount = recovery.readWalEvents(dir, "conn-1").filter((e) => e.type === "ANOMALY").length; + check( + "wal-anomaly-count-is-not-bounded-by-the-session-cap-and-matches-every-blocked-attempt", + walAnomalyCount === blockedAttempts, + `walAnomalyCount=${walAnomalyCount}, blockedAttempts=${blockedAttempts}, sessionCap=${MAX_ANOMALIES_PER_SESSION}` + ); + check( + "wal-anomaly-count-exceeds-what-the-session-cap-would-allow-if-it-applied-to-the-wal-too", + walAnomalyCount > MAX_ANOMALIES_PER_SESSION + 1, + `walAnomalyCount=${walAnomalyCount}` + ); + + resolveCall({ ok: true }); + await firstPromise; + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "emit complete step logs for replayed and blocked calls": a + * completed-intent replay previously logged only the special-purpose + * gateway_intent_replayed event, missing the normal step/status/duration shape every + * other handled call gets. */ +async function replayedCallGetsAStructuredCompletionLogToo() { + const dir = freshDir("replay-structured-log"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const logLines = []; + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { log: (line) => logLines.push(line) }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const params = { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "structured-log-key" } }; + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + logLines.length = 0; // only care about the retry's own logging below + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params }); + const parsedLogs = logLines.map((l) => { try { return JSON.parse(l); } catch (error) { return null; } }); + const replayedLog = parsedLogs.find((l) => l && l.event === "gateway_call_completed" && l.status === "replayed"); + check("replayed-call-gets-a-structured-completion-log", Boolean(replayedLog) && replayedLog.tool === "echo", JSON.stringify(logLines)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "persist cached replays in the session trace": an idempotency-key + * cache hit returned the cached tool result to the agent without recordCallStart/ + * recordCallResult, so the agent observed a successful tools/call that appeared nowhere in + * the session's own trace. It must now appear as a real, ordered step that is EXPLICITLY + * marked as a replay rather than passed off as a fresh downstream dispatch. */ +async function cachedReplayIsRecordedAsAnExplicitlyReplayedStep() { + const dir = freshDir("replay-trace"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { log: () => {} }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const params = { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "replay-trace-key" } }; + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + const retry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params }); + + const s = proxy.sessions.get("conn-1"); + const recorded = s.calls.slice().sort((a, b) => a.seq - b.seq); + check( + "cached-replay-dispatched-downstream-exactly-once", + conn.calls.filter((c) => c.method === "tools/call").length === 1, + JSON.stringify(conn.calls) + ); + check( + "cached-replay-is-recorded-as-its-own-step-in-the-trace", + recorded.length === 2 && recorded[1].tool === "echo", + `recorded ${recorded.length} call(s): ${JSON.stringify(recorded.map((c) => c.tool))}` + ); + check( + "cached-replay-step-is-explicitly-marked-replayed", + recorded.length === 2 && recorded[1].replayed === true && typeof recorded[1].replayed_from_intent_key === "string", + JSON.stringify(recorded[1] || null) + ); + check( + "first-dispatch-step-is-not-marked-replayed", + recorded.length === 2 && recorded[0].replayed === undefined, + JSON.stringify(recorded[0] || null) + ); + check("cached-replay-still-returns-the-cached-result", retry && retry.result && retry.result.value === 1, JSON.stringify(retry)); + + // The replayed step must also survive a crash: it is in the WAL, marked, so + // gateway.js#recoverCrashedSessions reconstructs the same shape a clean close would. + const walEvents = recovery.readWalEvents(dir, "conn-1"); + const replayedResultEvents = walEvents.filter((e) => e.type === "CALL_RESULT" && e.replayed === true); + check( + "cached-replay-is-durably-recorded-in-the-wal-as-a-replay", + replayedResultEvents.length === 1 && replayedResultEvents[0].result && replayedResultEvents[0].result.value === 1, + JSON.stringify(walEvents.map((e) => ({ type: e.type, replayed: e.replayed }))) + ); + + // And the sealed bundle must carry the marker through toSealableSession. + const sealable = session.toSealableSession(s); + check( + "sealed-trace-carries-the-replay-marker", + sealable.calls.length === 2 && sealable.calls[1].replayed === true, + JSON.stringify(sealable.calls) + ); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "emit one log line for each replayed step": a cached replay used to + * emit BOTH gateway_intent_replayed and gateway_call_completed for one handled step, + * double-counting it for any consumer keeping to this repo's one-line-per-step invariant. */ +async function cachedReplayEmitsExactlyOneLogLine() { + const dir = freshDir("replay-one-log"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const logLines = []; + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { log: (line) => logLines.push(line) }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const params = { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "one-log-key" } }; + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + logLines.length = 0; // only the retry's own logging matters below + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params }); + check("cached-replay-emits-exactly-one-log-line", logLines.length === 1, JSON.stringify(logLines)); + const parsed = logLines.map((l) => { try { return JSON.parse(l); } catch (error) { return null; } }); + check( + "the-single-replay-log-line-carries-the-replay-metadata", + parsed.length === 1 && parsed[0] && parsed[0].event === "gateway_call_completed" && parsed[0].status === "replayed" && + parsed[0].replayed === true && typeof parsed[0].intent_key === "string" && parsed[0].step !== null, + JSON.stringify(logLines) + ); + check( + "no-separate-gateway-intent-replayed-line-is-emitted-any-more", + !parsed.some((l) => l && l.event === "gateway_intent_replayed"), + JSON.stringify(logLines) + ); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "continue sealing when the closing WAL append fails": closeConnection + * removes the session from both live maps BEFORE appending the CLOSING WAL event, so an + * unguarded throw there left a completed, healthy session permanently absent from the + * signed chain with nothing in memory left to retry from. */ +async function closingWalAppendFailureStillSealsTheSession() { + const dir = freshDir("closing-wal-fails"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { log: () => {} }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + + // Fail ONLY the CLOSING append, on an otherwise completely healthy connection. + const realAppend = recovery.appendWalEvent; + recovery.appendWalEvent = (stateDir, connectionId, event) => { + if (event && event.type === "CLOSING") throw Object.assign(new Error("simulated ENOSPC on the CLOSING append"), { code: "ENOSPC" }); + return realAppend(stateDir, connectionId, event); + }; + let entry = null; + let thrown = null; + try { + entry = await proxy.closeConnection("conn-1", "test close"); + } catch (error) { + thrown = error; + } finally { + recovery.appendWalEvent = realAppend; + } + check("closing-wal-append-failure-does-not-throw-out-of-closeConnection", thrown === null, thrown && thrown.message); + check("closing-wal-append-failure-still-appends-the-session-to-the-chain", Boolean(entry) && entry.seq === 1, JSON.stringify(entry)); + const head = chain.readHead(dir); + check("closing-wal-append-failure-still-advances-HEAD", Boolean(head) && head.seq === 1, JSON.stringify(head)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, entry.bundle_id), "utf8")); + const raw = JSON.stringify(bundle); + check( + "closing-wal-append-failure-is-attested-in-the-sealed-bundle", + raw.includes("GATEWAY_RECOVERY_WAL_APPEND_FAILED"), + raw.slice(0, 400) + ); +} + +/* Frontier-panel decision (Paul, 2026-09-10, cluster E2): a reconnecting agent gets a + * brand-new connectionId, so its retry of "the same" logical operation computes a + * DIFFERENT intentKey than a still-unresolved intent its crashed predecessor connection + * left behind. A tool with an unresolved (dispatched/ambiguous) intent belonging to a + * connection startup recovery flagged pendingOperatorReview is quarantined for every + * connection until an operator resolves it -- regardless of connectionId or arguments. */ +async function toolQuarantinedWhileACrashedConnectionIsPendingOperatorReview() { + const dir = freshDir("quarantine"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "email", server: "srv", schema: {} }]; + const toolOwners = new Map([["email", "srv"]]); + // Simulate a crash-left "ambiguous" intent belonging to a now-defunct connection, the + // way gateway.js#recoverCrashedSessions would leave one on disk at startup. + const staleIntentKey = recovery.computeIntentKey("crashed-conn", "email", { to: "x" }); + recovery.createIntentIfAbsent(dir, staleIntentKey, { + connection_id: "crashed-conn", + tool: "email", + arguments: { to: "x" }, + state: "dispatched", + dispatched_at: Date.now(), + }); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { + pendingOperatorReviewConnections: ["crashed-conn"], + }); + proxy.openConnection("conn-2"); // a brand-new (e.g. reconnecting) connection + await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-2"); + const resp = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "email", arguments: { to: "y" } } }); + check("tool-quarantined-for-different-connection-and-arguments", Boolean(resp && resp.error && resp.error.code === -32082), JSON.stringify(resp)); + check("tool-quarantined-did-not-redispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-2", "test cleanup"); +} + +/* The quarantine must not over-block: a DIFFERENT tool on the same gateway, and the SAME + * tool once its owning connection is no longer in pendingOperatorReview (i.e. resolved), + * must dispatch normally. */ +async function toolQuarantineScopedToTheAffectedToolAndConnectionOnly() { + const dir = freshDir("quarantine-scope"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "email", server: "srv", schema: {} }, { name: "increment", server: "srv", schema: {} }]; + const toolOwners = new Map([["email", "srv"], ["increment", "srv"]]); + const staleIntentKey = recovery.computeIntentKey("crashed-conn", "email", { to: "x" }); + recovery.createIntentIfAbsent(dir, staleIntentKey, { + connection_id: "crashed-conn", + tool: "email", + arguments: { to: "x" }, + state: "dispatched", + dispatched_at: Date.now(), + }); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { + pendingOperatorReviewConnections: ["crashed-conn"], + }); + proxy.openConnection("conn-2"); + await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-2"); + const otherTool = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "increment", arguments: {} } }); + check("unrelated-tool-not-quarantined", Boolean(otherTool && otherTool.result && otherTool.result.ok === true), JSON.stringify(otherTool)); + // Simulate the operator resolving the crashed connection (recovery-abandon deletes the + // intent; recovery-resolve --confirmed executed/not-executed sets a terminal state -- + // either way this connection no longer belongs in pendingOperatorReviewConnections on + // the NEXT gateway startup, which is what a fresh GatewayProxy instance below models). + recovery.deleteIntent(dir, staleIntentKey); + const proxy2 = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { pendingOperatorReviewConnections: [] }); + proxy2.openConnection("conn-3"); + await proxy2.handleMessage("conn-3", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy2, "conn-3"); + const afterResolve = await proxy2.handleMessage("conn-3", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "email", arguments: { to: "y" } } }); + check("tool-dispatches-normally-once-crashed-connection-resolved", Boolean(afterResolve && afterResolve.result && afterResolve.result.ok === true), JSON.stringify(afterResolve)); + await proxy.closeConnection("conn-2", "test cleanup"); + await proxy2.closeConnection("conn-3", "test cleanup"); +} + +/* Cluster A (cross-connection replay of a proven-completed call): a reconnecting agent + * gets a brand-new connectionId, so its retry of "the same" logical operation computes a + * DIFFERENT intentKey than its own crashed/closed predecessor connection's completed + * call (intentKey is scoped to connectionId -- see computeIntentKey's own header). + * Presenting the SAME caller idempotency key the original call carried must still + * replay that real result rather than re-executing the downstream side effect. */ +async function reconnectWithMatchingIdempotencyKeyReplaysCrossConnectionCompletedCall() { + const dir = freshDir("cross-conn-replay"); + const conn = fakeConnectionCapturing(async () => ({ value: 7 })); + const mergedTools = [{ name: "charge", server: "srv", schema: {} }]; + const toolOwners = new Map([["charge", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + const params = { name: "charge", arguments: { amount: 5 }, _meta: { idempotencyKey: "customer-key-1" } }; + + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + await proxy.closeConnection("conn-1", "conn-1 done"); + + // A brand-new connection (e.g. the agent reconnected after a crash) presents the SAME + // idempotency key for the SAME logical operation. + proxy.openConnection("conn-2"); + await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-2"); + const resp = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + check("cross-connection-replay-returns-the-original-result", Boolean(resp && resp.result && resp.result.value === 7), JSON.stringify(resp)); + check("cross-connection-replay-does-not-redispatch-downstream", conn.calls.length === 1, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-2", "test cleanup"); +} + +/* The other half of the same fix: WITHOUT a matching (or any) idempotency key, a + * reconnecting agent's call to the same tool+arguments is a genuinely new, independent + * dispatch -- never silently collapsed onto an old connection's result just because the + * arguments happen to match, mirroring the live same-connection rule exactly. */ +async function reconnectWithoutMatchingIdempotencyKeyDispatchesIndependently() { + const dir = freshDir("cross-conn-no-replay"); + let callCount = 0; + const conn = fakeConnectionCapturing(async () => { callCount += 1; return { value: callCount }; }); + const mergedTools = [{ name: "charge", server: "srv", schema: {} }]; + const toolOwners = new Map([["charge", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + const params = { name: "charge", arguments: { amount: 5 } }; // no _meta.idempotencyKey + + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + await proxy.closeConnection("conn-1", "conn-1 done"); + + proxy.openConnection("conn-2"); + await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-2"); + const resp = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + check("cross-connection-without-key-dispatches-independently", Boolean(resp && resp.result && resp.result.value === 2), JSON.stringify(resp)); + check("cross-connection-without-key-redispatches-downstream", conn.calls.length === 2, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-2", "test cleanup"); +} + +/* "Extend the existing quarantine mechanism ... to also cover this retained-completed- + * intent case": a tool quarantined because some OTHER crashed connection left it with an + * unresolved outcome must stay blocked even for a reconnecting caller that also happens + * to present a valid, matching idempotency key for an unrelated, already-completed + * signature on that same tool -- the retained-signature replay path must never bypass + * the quarantine gate. */ +async function retainedSignatureReplayStillBlockedByQuarantine() { + const dir = freshDir("cross-conn-quarantine"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "email", server: "srv", schema: {} }]; + const toolOwners = new Map([["email", "srv"]]); + const params = { name: "email", arguments: { to: "a" }, _meta: { idempotencyKey: "email-key-1" } }; + + // First, a real completed call on its own connection -- this is what populates the + // cross-connection retained-signature store the quarantine gate must still override. + const proxy1 = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy1.openConnection("conn-1"); + await proxy1.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy1, "conn-1"); + await proxy1.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + await proxy1.closeConnection("conn-1", "conn-1 done"); + + // Now simulate a DIFFERENT, still-crashed connection leaving this exact tool + // quarantined (an unresolved intent for the SAME tool, different arguments -- the + // quarantine check is scoped to the tool, not the arguments; see proxy.js's own doc + // comment on it). + const staleIntentKey = recovery.computeIntentKey("crashed-conn", "email", { to: "z" }); + recovery.createIntentIfAbsent(dir, staleIntentKey, { + connection_id: "crashed-conn", + tool: "email", + arguments: { to: "z" }, + state: "dispatched", + dispatched_at: Date.now(), + }); + const proxy2 = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { + pendingOperatorReviewConnections: ["crashed-conn"], + }); + proxy2.openConnection("conn-2"); + await proxy2.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy2, "conn-2"); + const resp = await proxy2.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params }); + check("quarantine-blocks-even-a-matching-retained-signature-replay", Boolean(resp && resp.error && resp.error.code === -32082), JSON.stringify(resp)); + check("quarantine-blocks-retained-signature-replay-without-redispatch", conn.calls.length === 1, JSON.stringify(conn.calls)); // only conn-1's original call + await proxy2.closeConnection("conn-2", "test cleanup"); +} + +/* Codex PR #33 review "undo the fence when CALL_START persistence fails": the intent is + * created (dispatched) before this WAL append -- if the append itself fails (e.g. a full + * disk), the call never actually reaches conn.call(). Without a rollback, the intent + * stays "dispatched" forever (every retry permanently blocked) and, worse, the exception + * used to escape handleMessage entirely, breaking its documented "never throws" contract. + * + * Round-2 fix pass update: this append failure now also poisons the CONNECTION (see + * poisonWalOnFailure above), per docs/contracts/wal-append-failure-semantics.md (C1 SS2) + * -- "the caller should treat the refusal exactly like a second append failure," which + * for a retry on the SAME connection means every subsequent appendWalEvent attempt is + * refused too, not just this one intent's own fence. "Stays retryable" therefore now + * means retryable on a fresh connection (close and reopen, per the poisoned error's own + * wording), not a bare retry on the same, now-poisoned one -- this test asserts both. */ +async function callStartWalAppendFailureRollsBackAndStaysRetryable() { + const dir = freshDir("call-start-wal-failure"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const realAppendWalEvent = recovery.appendWalEvent; + let failNextCallStart = true; + recovery.appendWalEvent = (...args) => { + if (failNextCallStart && args[2] && args[2].type === "CALL_START") { + failNextCallStart = false; + throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + } + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("call-start-wal-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("call-start-wal-failure-returns-a-retryable-jsonrpc-error", Boolean(resp && resp.error && resp.error.code === -32000), JSON.stringify(resp)); + check("call-start-wal-failure-did-not-dispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + const intentKey = recovery.computeIntentKey("conn-1", "echo", { a: 1 }); + check("call-start-wal-failure-rolled-back-the-intent", recovery.readIntent(dir, intentKey) === null, "intent still present"); + + // Round-2 fix pass: a bare retry on the SAME (now-poisoned) connection must be + // refused, not silently allowed through onto a possibly-torn WAL line. + const sameConnRetry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + check( + "call-start-wal-failure-retry-on-the-same-connection-is-refused-poisoned", + Boolean(sameConnRetry && sameConnRetry.error && /poisoned/i.test(sameConnRetry.error.message)), + JSON.stringify(sameConnRetry) + ); + check("call-start-wal-failure-retry-on-the-same-connection-did-not-dispatch", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); + + // A fresh connection (close-and-reopen, exactly as the poisoned error's own wording + // instructs) gets its own fresh WAL and dispatches normally. + proxy.openConnection("conn-2"); + await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-2"); + const freshConnRetry = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + check("call-start-wal-failure-retry-on-a-fresh-connection-dispatches-normally", Boolean(freshConnRetry && freshConnRetry.result && freshConnRetry.result.ok === true), JSON.stringify(freshConnRetry)); + await proxy.closeConnection("conn-2", "test cleanup"); +} + +/* Codex PR #33 review "guard the post-dispatch intent update": a concurrent removal of + * this call's intent (an operator's recovery-resolve, or a race with closeConnection) + * while conn.call() is still in flight used to make the post-dispatch updateIntent throw + * GATEWAY_RECOVERY_INTENT_NOT_FOUND uncaught -- preventing the JSON-RPC response for a + * call that DID complete. */ +async function postDispatchUpdateSkippedWhenIntentConcurrentlyRemoved() { + const dir = freshDir("intent-removed-midflight"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const callPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: { a: 1 } } }); + const intentKey = recovery.computeIntentKey("conn-1", "slow", { a: 1 }); + recovery.deleteIntent(dir, intentKey); // simulate the concurrent removal + resolveCall({ ok: true }); + let resp, threw = null; + try { + resp = await callPromise; + } catch (error) { + threw = error; + } + check("post-dispatch-update-does-not-throw-when-intent-concurrently-removed", threw === null, threw && threw.message); + check("post-dispatch-update-still-returns-the-real-result-to-the-agent", Boolean(resp && resp.result && resp.result.ok === true), JSON.stringify(resp)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* CodeRabbit PR #33 review "guard this WAL append the same way the CALL_START append is + * guarded": the ANOMALY append on a blocked-retry path (GATEWAY_AMBIGUOUS_RETRY etc.) was + * unguarded, unlike CALL_START's own established pattern -- a WAL failure here used to + * escape handleMessage entirely instead of still returning the block response. */ +async function anomalyWalAppendFailureDoesNotEscapeHandleMessage() { + const dir = freshDir("anomaly-wal-failure"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const firstPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: { a: 1 } } }); + await new Promise((resolve) => setImmediate(resolve)); + + const realAppendWalEvent = recovery.appendWalEvent; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type === "ANOMALY") throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "slow", arguments: { a: 1 } } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("anomaly-wal-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("anomaly-wal-failure-still-returns-the-block-response", Boolean(resp && resp.error && resp.error.code === -32080), JSON.stringify(resp)); + /* Round-1 fix-round follow-up (2026-09-15): settle, with a test rather than an + * assumption, whether session.recordAnomaly consumes the MAX_ANOMALIES_PER_SESSION + * budget BEFORE this path's WAL append (and its outcome) or after -- prior review + * rounds disagreed on this. proxy.js calls session.recordAnomaly(...) unconditionally, + * THEN attempts the (here, deliberately failing) WAL append in its own try/catch -- + * so the in-memory/sealed-bundle accounting must already be done by the time the WAL + * append is even attempted, independent of whether that append succeeds. */ + const sAfterWalFailure = proxy.sessions.get("conn-1"); + check( + "anomaly-budget-was-consumed-in-memory-before-the-failed-wal-append-was-even-attempted", + Boolean(sAfterWalFailure) && sAfterWalFailure.anomalies.length === 1 && sAfterWalFailure.anomalies[0].kind === "GATEWAY_AMBIGUOUS_RETRY", + JSON.stringify(sAfterWalFailure && sAfterWalFailure.anomalies) + ); + resolveCall({ ok: true }); + await firstPromise; + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* CodeRabbit PR #33 review "add a not_executed branch to the live dispatch guard": an + * intent an operator confirmed via recovery-resolve --confirmed not-executed used to fall + * through to the generic "could not establish a durable dispatch intent" AMBIGUOUS_RETRY + * after two wasted attempts, instead of a response that reflects what actually happened. */ +async function notExecutedIntentReturnsDedicatedBlockCode() { + const dir = freshDir("not-executed-branch"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "email", server: "srv", schema: {} }]; + const toolOwners = new Map([["email", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const intentKey = recovery.computeIntentKey("conn-1", "email", { to: "x" }); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-1", tool: "email", arguments: { to: "x" }, state: "dispatched", dispatched_at: Date.now() }); + recovery.resolveIntentNotExecuted(dir, intentKey); + const resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "email", arguments: { to: "x" } } }); + check("not-executed-intent-returns-dedicated-code", Boolean(resp && resp.error && resp.error.code === -32084), JSON.stringify(resp)); + check("not-executed-intent-did-not-dispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "reject duplicate request IDs before creating intents": an agent + * reusing an in-flight JSON-RPC id for a genuinely different operation used to get a + * durable "dispatched" intent created BEFORE session.recordCallStart's own duplicate-id + * check ran (and threw, uncaught) -- orphaning that intent forever for a call that never + * actually dispatched. */ +async function duplicateJsonRpcIdRejectedBeforeIntentCreated() { + const dir = freshDir("duplicate-id"); + let resolveFirst; + const conn = fakeConnectionCapturing((method, params) => { + if (params && params.name === "slow") return new Promise((resolve) => { resolveFirst = resolve; }); + return Promise.resolve({ ok: true }); + }); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }, { name: "other", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"], ["other", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const firstPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: {} } }); + await new Promise((resolve) => setImmediate(resolve)); + + let resp, threw = null; + try { + // Same id (1) reused for a DIFFERENT tool while the first is still pending. + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "other", arguments: {} } }); + } catch (error) { + threw = error; + } + check("duplicate-id-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("duplicate-id-rejected-with-a-jsonrpc-error", Boolean(resp && resp.error), JSON.stringify(resp)); + check("duplicate-id-did-not-dispatch-the-second-call", conn.calls.length === 1, JSON.stringify(conn.calls)); + const otherIntentKey = recovery.computeIntentKey("conn-1", "other", {}); + check("duplicate-id-did-not-create-an-orphaned-intent", recovery.readIntent(dir, otherIntentKey) === null, "intent unexpectedly present"); + resolveFirst({ ok: true }); + await firstPromise; + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* CodeRabbit/Codex PR #33 review "restore the prior completed intent when CALL_START WAL + * persistence fails": when a completed intent is superseded by a new, independent dispatch + * (no matching idempotency key) and THAT dispatch's own CALL_START WAL append then fails, + * the rollback used to unconditionally delete the intent -- discarding the earlier + * generation's real cached result. A later request carrying the ORIGINAL idempotency key + * would then find no record and dispatch again instead of replaying it. */ +async function supersedeCallStartWalFailureRestoresPriorCompletedIntent() { + const dir = freshDir("supersede-wal-failure"); + const conn = fakeConnectionCapturing(async () => ({ value: 1 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const originalParams = { name: "echo", arguments: { a: 1 }, _meta: { idempotencyKey: "first-attempt" } }; + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: originalParams }); + const intentKey = recovery.computeIntentKey("conn-1", "echo", { a: 1 }); + const beforeSupersede = recovery.readIntent(dir, intentKey); + check("supersede-setup-first-call-completed", Boolean(beforeSupersede) && beforeSupersede.state === "completed" && beforeSupersede.idempotency_key === "first-attempt", JSON.stringify(beforeSupersede)); + + const realAppendWalEvent = recovery.appendWalEvent; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type === "CALL_START") throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + // No idempotencyKey -> this is treated as a new, independent dispatch that supersedes + // the completed record above, generation 2 -- whose own CALL_START append then fails. + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("supersede-wal-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("supersede-wal-failure-returns-a-retryable-jsonrpc-error", Boolean(resp && resp.error && resp.error.code === -32000), JSON.stringify(resp)); + check("supersede-wal-failure-did-not-redispatch-downstream", conn.calls.length === 1, JSON.stringify(conn.calls)); + const restored = recovery.readIntent(dir, intentKey); + check( + "supersede-wal-failure-restored-the-original-completed-record", + Boolean(restored) && restored.state === "completed" && restored.idempotency_key === "first-attempt" && restored.generation === 1 && restored.cached_result && restored.cached_result.value === 1, + JSON.stringify(restored) + ); + // The original idempotency key must still replay the FIRST call's cached result, not + // dispatch a third time. + const replay = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 3, method: "tools/call", params: originalParams }); + check("supersede-wal-failure-original-key-still-replays-not-redispatches", conn.calls.length === 1 && Boolean(replay && replay.result && replay.result.value === 1), JSON.stringify({ calls: conn.calls, replay })); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* CodeRabbit PR #33 review "fail closed when the quarantine scan cannot read an intent": + * listAllIntents propagating a read failure for ANY intent file on disk (not just the one + * relevant to this dispatch) used to reject handleMessage with a generic -32603 instead of + * this quarantine check's own documented -32082, and the tool was not actually blocked. */ +async function quarantineScanFailureFailsClosed() { + const dir = freshDir("quarantine-scan-failure"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "email", server: "srv", schema: {} }]; + const toolOwners = new Map([["email", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { + pendingOperatorReviewConnections: ["crashed-conn"], + }); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const realListAllIntents = recovery.listAllIntents; + recovery.listAllIntents = () => { throw Object.assign(new Error("simulated corrupt intent"), { code: "GATEWAY_RECOVERY_INTENT_CORRUPT" }); }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "email", arguments: { to: "x" } } }); + } catch (error) { + threw = error; + } finally { + recovery.listAllIntents = realListAllIntents; + } + check("quarantine-scan-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("quarantine-scan-failure-fails-closed-with-quarantine-code", Boolean(resp && resp.error && resp.error.code === -32082), JSON.stringify(resp)); + check("quarantine-scan-failure-did-not-dispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "block dispatch when a retained signature is unreadable": this catch + * claimed to fail closed but converted the failure to `null` -- a CACHE MISS that fell + * straight through to a fresh dispatched intent and a real downstream call. An EIO or a + * corrupt signature file therefore repeated an already-completed side effect at exactly the + * moment its deduplication evidence could not be verified. */ +async function unreadableRetainedSignatureBlocksInsteadOfDispatching() { + const dir = freshDir("retained-signature-unreadable"); + const conn = fakeConnectionCapturing(async () => ({ value: 7 })); + const mergedTools = [{ name: "charge", server: "srv", schema: {} }]; + const toolOwners = new Map([["charge", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const realReadCompletedSignature = recovery.readCompletedSignature; + recovery.readCompletedSignature = () => { + throw Object.assign(new Error("EIO: simulated unreadable retained signature"), { code: "GATEWAY_RECOVERY_SIGNATURE_UNREADABLE" }); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "charge", arguments: { amount: 5 }, _meta: { idempotencyKey: "customer-key-1" } } }); + } catch (error) { + threw = error; + } finally { + recovery.readCompletedSignature = realReadCompletedSignature; + } + check("unreadable-retained-signature-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("unreadable-retained-signature-fails-closed-with-its-own-code", Boolean(resp && resp.error && resp.error.code === -32083), JSON.stringify(resp)); + check("unreadable-retained-signature-did-not-dispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "make the post-fence diagnostic non-throwing": on the supersede path + * the durable intent is already flipped to "dispatched" before this diagnostic is logged. + * A caller-supplied logger that throws there was caught by the surrounding block's catch + * and rethrown, aborting before recordCallStart / the CALL_START WAL append / the + * downstream dispatch -- leaving a false in-flight fence that blocks every future retry of + * an effect that never occurred. */ +async function throwingLoggerOnTheSupersedePathDoesNotStrandTheFence() { + const dir = freshDir("supersede-logger-throws"); + let callCount = 0; + const conn = fakeConnectionCapturing(async () => { callCount += 1; return { value: callCount }; }); + const mergedTools = [{ name: "charge", server: "srv", schema: {} }]; + const toolOwners = new Map([["charge", "srv"]]); + let loggerThrows = 0; + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { + log: (payload) => { + if (typeof payload === "string" && payload.includes("gateway_intent_signature_reused_without_key")) { + loggerThrows += 1; + throw new Error("simulated caller-supplied logger failure"); + } + }, + }); + const args = { amount: 5 }; + + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "charge", arguments: args, _meta: { idempotencyKey: "key-1" } } }); + + // Same connection, same tool+arguments, NO idempotency key -> the supersede path, which + // is where the post-fence diagnostic (and the throwing logger) lives. + const resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "charge", arguments: args } }); + check("supersede-logger-throw-was-actually-exercised", loggerThrows === 1, String(loggerThrows)); + check("supersede-logger-throw-does-not-abort-the-dispatch", Boolean(resp && resp.result && resp.result.value === 2), JSON.stringify(resp)); + check("supersede-logger-throw-still-reaches-downstream", conn.calls.length === 2, JSON.stringify(conn.calls)); + const intentKey = recovery.computeIntentKey("conn-1", "charge", args); + const intent = recovery.readIntent(dir, intentKey); + check("supersede-logger-throw-leaves-no-false-in-flight-fence", Boolean(intent) && intent.state === "completed", JSON.stringify(intent)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Codex PR #33 review "commit initialization only after its WAL event succeeds": a WAL + * failure on the INITIALIZE append used to be unguarded (escaping handleMessage) AND ran + * after agentInitialized was already flipped true, so a retry would be rejected as a + * repeat initialize even though it was never durably recorded. */ +async function initializeWalFailureDoesNotEscapeAndRollsBack() { + const dir = freshDir("initialize-wal-failure"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + + const realAppendWalEvent = recovery.appendWalEvent; + let failNext = true; + recovery.appendWalEvent = (...args) => { + if (failNext && args[2] && args[2].type === "INITIALIZE") { + failNext = false; + throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + } + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("initialize-wal-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("initialize-wal-failure-returns-a-retryable-jsonrpc-error", Boolean(resp && resp.error && resp.error.code === -32000), JSON.stringify(resp)); + check("initialize-wal-failure-did-not-flip-agentInitialized", proxy.agentInitialized.get("conn-1") === "none", `agentInitialized unexpectedly advanced: ${proxy.agentInitialized.get("conn-1")}`); + + // Round-2 fix pass: this append failure now also poisons the CONNECTION (see + // poisonWalOnFailure above) -- a bare retry of initialize on the SAME connection must + // be refused too, not silently allowed through onto a possibly-torn WAL line. + const sameConnRetry = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "initialize", params: {} }); + check( + "initialize-retry-on-the-same-connection-is-refused-poisoned", + Boolean(sameConnRetry && sameConnRetry.error && /poisoned/i.test(sameConnRetry.error.message)), + JSON.stringify(sameConnRetry) + ); + await proxy.closeConnection("conn-1", "test cleanup"); + + // A fresh connection (close-and-reopen) gets its own fresh WAL and initializes + // normally. + proxy.openConnection("conn-2"); + const freshConnRetry = await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + check("initialize-retry-on-a-fresh-connection-succeeds-normally", Boolean(freshConnRetry && freshConnRetry.result && freshConnRetry.result.protocolVersion), JSON.stringify(freshConnRetry)); + await proxy.closeConnection("conn-2", "test cleanup"); +} + +/* Codex PR #33 review "roll back session publication when its first WAL write fails": a + * SESSION_START WAL failure in openConnection used to be unguarded and ran AFTER the + * session was already published into this.sessions/agentInitialized -- leaving an + * unreachable "ghost" session counted by openSessionCount() even though the caller (who + * never gets a usable connectionId back) saw openConnection throw. */ +async function openConnectionWalFailureDoesNotPublishGhostSession() { + const dir = freshDir("open-connection-wal-failure"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + + const realAppendWalEvent = recovery.appendWalEvent; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type === "SESSION_START") throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + return realAppendWalEvent(...args); + }; + let threw = null; + try { + proxy.openConnection("conn-1"); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("open-connection-wal-failure-still-throws-to-its-caller", threw !== null, "openConnection did not throw"); + check("open-connection-wal-failure-did-not-publish-a-ghost-session", proxy.openSessionCount() === 0, `openSessionCount()=${proxy.openSessionCount()}`); + check("open-connection-wal-failure-did-not-set-agentInitialized", proxy.agentInitialized.has("conn-1") === false, "agentInitialized unexpectedly set"); + + // A subsequent real openConnection for the SAME connectionId must work normally -- + // proof there is no leftover half-published state blocking it. + proxy.openConnection("conn-1"); + check("open-connection-retry-succeeds-normally-afterward", proxy.openSessionCount() === 1, `openSessionCount()=${proxy.openSessionCount()}`); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* Part of the same WAL-append-guarding cluster as CALL_START/INITIALIZE/ANOMALY above: the + * CALL_RESULT append (after a call has ALREADY completed downstream, with no "don't + * dispatch" option left) must be best-effort and never escape handleMessage -- the agent + * is still owed its real result even if this durability write fails. */ +async function callResultWalFailureDoesNotEscapeHandleMessage() { + const dir = freshDir("call-result-wal-failure"); + const conn = fakeConnectionCapturing(async () => ({ value: 99 })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const realAppendWalEvent = recovery.appendWalEvent; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type === "CALL_RESULT") throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: {} } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("call-result-wal-failure-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("call-result-wal-failure-still-returns-the-real-result", Boolean(resp && resp.result && resp.result.value === 99), JSON.stringify(resp)); + const intentKey = recovery.computeIntentKey("conn-1", "echo", {}); + const intent = recovery.readIntent(dir, intentKey); + check("call-result-wal-failure-intent-still-marked-completed", Boolean(intent) && intent.state === "completed", JSON.stringify(intent)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + /* Codex PR #29 review round 3 "retain sessions when persistence fails": closeConnection * used to discard the fully-sealed bundle the moment chain.appendSession threw, leaving - * only a summary log line. Two connections given identical initialize params and an - * identical call count deterministically produce the same bundle_id (gsa-mcp-shim.js - * hashes only {init, grantedTools, n} -- see chain.js's own "same agent reconnecting ... - * reproduces it" comment), so the second one's real chain.appendSession call genuinely - * throws GATEWAY_BUNDLE_ID_COLLISION here rather than a synthetic/forced failure. */ + * only a summary log line. Two connections given identical initialize params, an + * identical call count, AND (Cluster B: session_id is now folded into bundle_id -- see + * session.js#createSession/gsa-mcp-shim.js#sealBoundaryBundle) the SAME explicit + * session_id deterministically produce the same bundle_id, so the second one's real + * chain.appendSession call genuinely throws GATEWAY_BUNDLE_ID_COLLISION here rather than + * a synthetic/forced failure. Forcing an identical session_id across two otherwise- + * independent connections is now the only way to reproduce that collision on purpose + * (a real, live connection always gets its own random one) -- exactly mirroring what + * "two connections with identical content" meant before session_id existed. */ + +// --------------------------------------------------------------------------- +// Round-2 fix pass (2026-09-16): closing commit 5's WAL-poison enforcement gap. +// +// The independent verify pass on commit 5 (b14bf35c) found that gateway.js:142/ +// session.js:88 document a connection-wide contract -- once an appendWalEvent call +// fails for a connection, `s.walPoisoned` is set and EVERY later appendWalEvent +// attempt on that same connection must be refused before touching the filesystem +// (docs/contracts/wal-append-failure-semantics.md, C1 SS2-3) -- but that contract was +// only actually enforced on gateway.js's own downstream-initiated sampling-forward +// path. proxy.js's own appendWalEvent call sites (SESSION_START/INITIALIZE/ANOMALY/ +// CALL_START/CALL_RESULT/CLOSING) neither checked nor set it, so an agent-initiated +// call could still dispatch onto (and CLOSING would unconditionally append onto) a +// connection's own already-poisoned or already-torn WAL. The four tests below cover: +// (1) proxy.js's own append failures now actually SET s.walPoisoned (not just check +// it); (2) an already-poisoned connection refuses a brand-new CALL_START before +// touching the filesystem, attested as an anomaly; (3) the CLOSING append -- the +// guaranteed-violation path the verify pass named explicitly, since every connection +// eventually closes -- is skipped and attested rather than unconditionally attempted; +// (4) the fail-open CALL_RESULT path still returns the agent's real result when +// skipping its own append due to poisoning, attested the same way. +// --------------------------------------------------------------------------- + +/* Closes the other half of the gap: not just gating on s.walPoisoned before an append, + * but proxy.js's OWN append failures must actually SET it (matching gateway.js# + * poisonWalOnFailure's identical contract), so a second, otherwise-healthy call on the + * SAME now-poisoned connection is refused too, without the filesystem ever being + * touched a second time. */ +async function ownAppendFailurePoisonsConnectionForEverySubsequentCallSite() { + const dir = freshDir("own-failure-poisons"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const realAppendWalEvent = recovery.appendWalEvent; + let failNextCallStart = true; + recovery.appendWalEvent = (...args) => { + if (failNextCallStart && args[2] && args[2].type === "CALL_START") { + failNextCallStart = false; + throw Object.assign(new Error("simulated ENOSPC"), { code: "ENOSPC" }); + } + return realAppendWalEvent(...args); + }; + let firstResp; + try { + firstResp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("own-failure-first-call-refused-as-not-dispatched", Boolean(firstResp && firstResp.error && firstResp.error.code === -32000), JSON.stringify(firstResp)); + + const s = proxy.sessions.get("conn-1"); + check("own-CALL_START-failure-sets-walPoisoned-on-the-session", Boolean(s.walPoisoned) && s.walPoisoned.reason === "simulated ENOSPC", JSON.stringify(s.walPoisoned)); + + const attemptedTypes = []; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type) attemptedTypes.push(args[2].type); + return realAppendWalEvent(...args); + }; + let secondResp, threw = null; + try { + secondResp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: { a: 2 } } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("own-failure-second-call-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("own-failure-second-call-refused-via-the-poisoned-gate-not-a-fresh-append-attempt", !attemptedTypes.includes("CALL_START"), JSON.stringify(attemptedTypes)); + check("own-failure-second-call-returns-a-poisoned-wal-error", Boolean(secondResp && secondResp.error && /poisoned/i.test(secondResp.error.message)), JSON.stringify(secondResp)); + check("own-failure-connection-never-dispatched-downstream-at-all", conn.calls.length === 0, JSON.stringify(conn.calls)); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* An already-poisoned connection must refuse a brand-new tools/call's CALL_START append + * before ever touching the filesystem -- treated exactly like a caught append failure + * (rollback pendingCalls/intent, attest an anomaly, return "not dispatched -- safe to + * retry"), never silently dispatched onto a possibly-torn WAL line. */ +async function poisonedConnectionRefusesNewCallStartAndAttestsSkip() { + const dir = freshDir("poisoned-call-start"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + + const s = proxy.sessions.get("conn-1"); + s.walPoisoned = { reason: "simulated earlier ENOSPC", at: Date.now() }; + + const realAppendWalEvent = recovery.appendWalEvent; + const attemptedTypes = []; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type) attemptedTypes.push(args[2].type); + return realAppendWalEvent(...args); + }; + let resp, threw = null; + try { + resp = await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: { a: 1 } } }); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("poisoned-call-start-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("poisoned-call-start-returns-a-poisoned-wal-error", Boolean(resp && resp.error && resp.error.code === -32000 && /poisoned/i.test(resp.error.message)), JSON.stringify(resp)); + check("poisoned-call-start-never-attempted-the-CALL_START-append", !attemptedTypes.includes("CALL_START"), JSON.stringify(attemptedTypes)); + check("poisoned-call-start-did-not-dispatch-downstream", conn.calls.length === 0, JSON.stringify(conn.calls)); + const intentKey = recovery.computeIntentKey("conn-1", "echo", { a: 1 }); + check("poisoned-call-start-did-not-leave-an-orphaned-intent", recovery.readIntent(dir, intentKey) === null, "intent unexpectedly present"); + check( + "poisoned-call-start-skip-is-attested-as-an-anomaly", + s.anomalies.some((a) => a.kind === "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED" && a.tool === "echo"), + JSON.stringify(s.anomalies) + ); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* The fail-OPEN CALL_RESULT path (the downstream call has already genuinely completed -- + * there is no "don't return the result" option left) must still skip its own append when + * the connection is poisoned, attest the skip, and still hand back the real result. */ +async function poisonedConnectionSkipsCallResultAppendButStillReturnsResult() { + const dir = freshDir("poisoned-call-result"); + let resolveCall; + const conn = fakeConnectionCapturing(() => new Promise((resolve) => { resolveCall = resolve; })); + const mergedTools = [{ name: "slow", server: "srv", schema: {} }]; + const toolOwners = new Map([["slow", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + const callPromise = proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "slow", arguments: {} } }); + await new Promise((resolve) => setImmediate(resolve)); + + // Poison mid-flight (after this call's own CALL_START already durably appended, before + // its CALL_RESULT does) -- simulating an earlier failed append on this connection + // (e.g. a concurrent downstream-initiated sampling forward via gateway.js) without + // needing to force a real filesystem error on this specific append. + const s = proxy.sessions.get("conn-1"); + s.walPoisoned = { reason: "simulated mid-flight poisoning", at: Date.now() }; + + const realAppendWalEvent = recovery.appendWalEvent; + const attemptedTypes = []; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type) attemptedTypes.push(args[2].type); + return realAppendWalEvent(...args); + }; + resolveCall({ value: 7 }); + let resp, threw = null; + try { + resp = await callPromise; + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + check("poisoned-call-result-does-not-escape-handleMessage", threw === null, threw && threw.message); + check("poisoned-call-result-still-returns-the-real-result", Boolean(resp && resp.result && resp.result.value === 7), JSON.stringify(resp)); + check("poisoned-call-result-never-attempted-the-CALL_RESULT-append", !attemptedTypes.includes("CALL_RESULT"), JSON.stringify(attemptedTypes)); + check( + "poisoned-call-result-skip-is-attested-as-an-anomaly", + s.anomalies.some((a) => a.kind === "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED"), + JSON.stringify(s.anomalies) + ); + await proxy.closeConnection("conn-1", "test cleanup"); +} + +/* The explicitly-required regression: the independent verify pass on commit 5 named + * closeConnection's CLOSING append as a GUARANTEED violation path, since it ran with no + * guard at all and every open connection eventually closes. Poisons a connection's WAL, + * drives it through a real close, and asserts the CLOSING append is never attempted (not + * attempted-and-caught -- refused before touching the filesystem, per C1 SS2-3) and that + * the skip is attested in the sealed bundle's own anomaly list, not silently swallowed. */ +async function poisonedConnectionSkipsClosingAppendAndAttestsIt() { + const dir = freshDir("poisoned-closing"); + const conn = fakeConnectionCapturing(async () => ({ ok: true })); + const mergedTools = [{ name: "echo", server: "srv", schema: {} }]; + const toolOwners = new Map([["echo", "srv"]]); + const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); + proxy.openConnection("conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); + await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "echo", arguments: {} } }); + + // Poison the connection's WAL exactly the way a real appendWalEvent failure would + // (poisonWalOnFailure's own {reason, at} shape), simulating an earlier failure on this + // connection without needing to force a real filesystem error at close time itself. + const s = proxy.sessions.get("conn-1"); + s.walPoisoned = { reason: "simulated earlier ENOSPC", at: Date.now() }; + + const realAppendWalEvent = recovery.appendWalEvent; + const attemptedTypes = []; + recovery.appendWalEvent = (...args) => { + if (args[2] && args[2].type) attemptedTypes.push(args[2].type); + return realAppendWalEvent(...args); + }; + let entry, threw = null; + try { + entry = await proxy.closeConnection("conn-1", "test cleanup"); + } catch (error) { + threw = error; + } finally { + recovery.appendWalEvent = realAppendWalEvent; + } + + check("poisoned-closing-does-not-escape-closeConnection", threw === null, threw && threw.message); + check("poisoned-closing-still-seals-and-appends-the-session", Boolean(entry), "closeConnection returned null"); + check("poisoned-closing-never-attempts-the-CLOSING-append-at-all", !attemptedTypes.includes("CLOSING"), JSON.stringify(attemptedTypes)); + // `s` is the same in-memory session object closeConnection operated on (only removed + // from proxy.sessions, never replaced) -- session.recordAnomaly's push onto s.anomalies + // happens before finalizeSession seals it, so this is exactly what the sealed bundle's + // own decision_record.md "## Anomalies" section (scripts/gsa-mcp-shim.js) attests from. + check( + "poisoned-closing-skip-is-attested-as-an-anomaly", + s.anomalies.some((a) => a.kind === "GATEWAY_RECOVERY_WAL_APPEND_SKIPPED_POISONED" && a.detail === "simulated earlier ENOSPC"), + JSON.stringify(s.anomalies) + ); +} + async function persistenceFailureQuarantinesSealedBundle() { const dir = freshDir("quarantine"); const conn = fakeConnection(async () => ({ ok: true })); @@ -506,15 +1909,16 @@ async function persistenceFailureQuarantinesSealedBundle() { const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners, { onSealFailure: (s, error) => sealFailures.push(error), }); + const forcedSharedSessionId = "forced-shared-session-id-for-collision-test"; - proxy.openConnection("conn-1"); + proxy.openConnection("conn-1", { sessionId: forcedSharedSessionId }); await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "initialize", params: { clientInfo: { name: "x", version: "1" } } }); await sendInitializedNotification(proxy, "conn-1"); await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: {} } }); const entry1 = await proxy.closeConnection("conn-1", "test cleanup"); check("quarantine-setup-first-append-succeeds", Boolean(entry1 && typeof entry1.bundle_id === "string"), JSON.stringify(entry1)); - proxy.openConnection("conn-2"); + proxy.openConnection("conn-2", { sessionId: forcedSharedSessionId }); await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 1, method: "initialize", params: { clientInfo: { name: "x", version: "1" } } }); await sendInitializedNotification(proxy, "conn-2"); await proxy.handleMessage("conn-2", { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "echo", arguments: {} } }); @@ -596,6 +2000,7 @@ async function writerClaimCheckDefaultsToValidWhenNotProvided() { const proxy = makeProxy(dir, new Map([["srv", conn]]), mergedTools, toolOwners); // no isWriterClaimValid proxy.openConnection("conn-1"); await proxy.handleMessage("conn-1", { jsonrpc: "2.0", id: 1, method: "initialize", params: {} }); + await sendInitializedNotification(proxy, "conn-1"); const entry = await proxy.closeConnection("conn-1", "test cleanup"); check("append-not-refused-when-no-writer-claim-check-configured", Boolean(entry && typeof entry.bundle_id === "string"), JSON.stringify(entry)); } @@ -739,6 +2144,9 @@ async function main() { await downstreamDisconnectEmitsCompletionLog(); await completedCallHistoryCapped(); await stopAcceptingNewSessionsRefusesNewButNotExisting(); + getChainIntegrityFailureDefaultsToNeverLatchedSoBareProxiesAreUnaffected(); + await chainIntegrityLatchRefusesNewAdmissionButNotExisting(); + chainIntegrityLatchCarriesTheDiagnosticDetailNotJustABoolean(); await toolLevelErrorRecordedButNotProtocolError(); malformedResultPredicateMatrix(); await malformedResultRecordedAndResponseUnchanged(); @@ -750,6 +2158,50 @@ async function main() { await structuredLogEmittedPerCompletedCall(); await toolsListForwardsFullDescriptor(); await nullJsonRpcIdDoesNotCrash(); + await idempotencyKeyPropagatedToRealToolCallOnly(); + await inFlightRetryBlockedAsAmbiguousRetry(); + await completedCallReplaysCachedResultOnRetryWithMatchingKey(); + await retryOfCompletedCallWithoutKeyDispatchesIndependently(); + await retryWithMismatchedKeyDispatchesIndependently(); + await toolQuarantinedWhileACrashedConnectionIsPendingOperatorReview(); + await toolQuarantineScopedToTheAffectedToolAndConnectionOnly(); + await reconnectWithMatchingIdempotencyKeyReplaysCrossConnectionCompletedCall(); + await reconnectWithoutMatchingIdempotencyKeyDispatchesIndependently(); + await retainedSignatureReplayStillBlockedByQuarantine(); + await ambiguousOutcomeBlocksRetryUntilOperatorResolves(); + await closeConnectionFencesInFlightIntentAsAmbiguous(); + await walRecordsLifecycleEventsAndIsCleanedUpOnClose(); + + await blockedRetryAnomalyIsDurablyRecorded(); + await blockedRetryAnomaliesAreUncappedInTheWalDespiteTheSessionCap(); + await replayedCallGetsAStructuredCompletionLogToo(); + await cachedReplayIsRecordedAsAnExplicitlyReplayedStep(); + await cachedReplayEmitsExactlyOneLogLine(); + await closingWalAppendFailureStillSealsTheSession(); + await callStartWalAppendFailureRollsBackAndStaysRetryable(); + await postDispatchUpdateSkippedWhenIntentConcurrentlyRemoved(); + + // PR #33 review round 5 (post E1/E2) fixes: + await anomalyWalAppendFailureDoesNotEscapeHandleMessage(); + await notExecutedIntentReturnsDedicatedBlockCode(); + await duplicateJsonRpcIdRejectedBeforeIntentCreated(); + await supersedeCallStartWalFailureRestoresPriorCompletedIntent(); + await quarantineScanFailureFailsClosed(); + + // Codex PR #33 review round 2 (2026-09-17): + await unreadableRetainedSignatureBlocksInsteadOfDispatching(); + await throwingLoggerOnTheSupersedePathDoesNotStrandTheFence(); + await initializeWalFailureDoesNotEscapeAndRollsBack(); + await openConnectionWalFailureDoesNotPublishGhostSession(); + await callResultWalFailureDoesNotEscapeHandleMessage(); + + // Round-2 fix pass (2026-09-16): closing commit 5's WAL-poison enforcement gap. + await ownAppendFailurePoisonsConnectionForEverySubsequentCallSite(); + await poisonedConnectionRefusesNewCallStartAndAttestsSkip(); + await poisonedConnectionSkipsCallResultAppendButStillReturnsResult(); + await poisonedConnectionSkipsClosingAppendAndAttestsIt(); + + // From feature/track-1.2-standalone-gateway (merged into PR #33): await persistenceFailureQuarantinesSealedBundle(); await writerClaimRevalidatedImmediatelyBeforeAppend(); await writerClaimCheckDefaultsToValidWhenNotProvided(); diff --git a/tests/gateway/recovery/run-tests.js b/tests/gateway/recovery/run-tests.js new file mode 100644 index 00000000..f6b584bd --- /dev/null +++ b/tests/gateway/recovery/run-tests.js @@ -0,0 +1,2561 @@ +#!/usr/bin/env node +"use strict"; + +/* Regression suite for scripts/gateway/recovery.js (Option C: Codex PR #29 Finding 1 + * [session.js -- a completed call lives only in memory until the whole session closes] + * and Finding 2 [proxy.js -- nothing fences a retried tools/call against a duplicate + * downstream side effect], external-panel-reviewed design -- see + * option-c-hardened-design.md in the project record) and of gateway.js's own + * recoverCrashedSessions/abandonConnection startup-recovery + operator-override paths + * that replay recovery.js's WAL through session.js's real, unmodified recorder + * functions. + * + * Covers: + * - recovery.js's WAL primitives (append/read/list/delete), including the "torn tail + * line" tolerance a real crash mid-fsync produces. + * - recovery.js's intent-store primitives (create-if-absent, read, update, delete, + * list-for-connection, list-all, operator resolution) including canonicalization + * (key-order-independent hashing) and the race-safety of createIntentIfAbsent. + * - gateway.js#recoverCrashedSessions: the Finding-1 gap (a call completed downstream + * but the WAL never recorded CALL_RESULT before a crash) is closed by consulting the + * real intent record, NOT by guessing; an unresolved in-flight call is left for an + * operator rather than auto-sealed; a repeated recovery attempt on already-sealed + * data is a no-op (GATEWAY_BUNDLE_ID_COLLISION), not a crash. + * - gateway.js#abandonConnection: the explicit "give up on the fence, seal what we + * have" operator override, and that it actually releases the intent fence it gives + * up on (by design -- see its own doc comment in gateway.js). + */ + +const crypto = require("crypto"); +const fs = require("fs"); +const os = require("os"); +const path = require("path"); + +const ROOT = path.resolve(__dirname, "../../.."); +const recovery = require(path.join(ROOT, "scripts", "gateway", "recovery.js")); +const session = require(path.join(ROOT, "scripts", "gateway", "session.js")); +const chain = require(path.join(ROOT, "scripts", "gateway", "chain.js")); +const stateStore = require(path.join(ROOT, "scripts", "state-store.js")); +const { recoverCrashedSessions, abandonConnection, startGateway, runRecoveryResolveCli, forwardDownstreamRequestToAgent, buildHealthStatus, writeStatusFile } = require(path.join(ROOT, "scripts", "gateway", "gateway.js")); +const { WriterClaim } = require(path.join(ROOT, "scripts", "writer-claim.js")); + +let failures = 0; +const results = []; +function record(name, status, reason) { + console.log(status === "PASS" ? `PASS ${name}` : `FAIL ${name}+${reason || "unknown"}`); + results.push({ name, status, reason: reason || "" }); + if (status === "FAIL") failures++; +} +function check(name, cond, reason) { + record(name, cond ? "PASS" : "FAIL", reason); +} + +function freshDir(prefix) { + return fs.mkdtempSync(path.join(os.tmpdir(), `gs-gateway-recovery-${prefix}-`)); +} +function makeKeys() { + const kp = crypto.generateKeyPairSync("ed25519"); + return { privateKey: kp.privateKey, signer: "test-key", algo: "ed25519" }; +} +function silentLog() {} // most tests don't care about the operational log lines + +// Mirrors gsa-mcp-shim.js's own sha256Hex(JSON.stringify(...)) exactly -- the sealed +// execution_trace.jsonl only ever stores a call's result as this hash (privacy-preserving +// by design; see session.js's own doc comment on why), so a test proving replay recovered +// a SPECIFIC real result must compare against this same hash, not grep for raw text that +// the trace never contains. +function expectedResultSha256(result) { + return crypto.createHash("sha256").update(Buffer.from(JSON.stringify(result === undefined ? null : result), "utf8")).digest("hex"); +} + +// --------------------------------------------------------------------------- +// WAL primitives +// --------------------------------------------------------------------------- + +function walAppendAndReadRoundTrip() { + const dir = freshDir("wal-roundtrip"); + recovery.appendWalEvent(dir, "conn-1", { type: "SESSION_START", goal: "g" }); + recovery.appendWalEvent(dir, "conn-1", { type: "INITIALIZE", model: "m" }); + const events = recovery.readWalEvents(dir, "conn-1"); + check("wal-roundtrip-count", events.length === 2, String(events.length)); + check("wal-roundtrip-order-and-content", events[0].type === "SESSION_START" && events[0].goal === "g" && events[1].type === "INITIALIZE" && events[1].model === "m", JSON.stringify(events)); + check("wal-roundtrip-records-recorded_at", typeof events[0].recorded_at === "number", JSON.stringify(events[0])); +} + +function walReadOfMissingConnectionReturnsEmpty() { + const dir = freshDir("wal-missing"); + const events = recovery.readWalEvents(dir, "never-existed"); + check("wal-read-missing-connection-returns-empty-array", Array.isArray(events) && events.length === 0, JSON.stringify(events)); +} + +function walTornTailLineToleratedRestKept() { + const dir = freshDir("wal-torn"); + recovery.appendWalEvent(dir, "conn-1", { type: "SESSION_START", goal: null }); + recovery.appendWalEvent(dir, "conn-1", { type: "CALL_START", call_seq: 1, tool: "t", server: "s", arguments: {}, ts: 1 }); + // Simulate a crash mid-fsync of the THIRD line: append a torn (non-JSON) tail + // directly, bypassing appendWalEvent's own well-formed JSON.stringify. + fs.appendFileSync(recovery.walPath(dir, "conn-1"), '{"type":"CALL_RESULT","call_se'); + const events = recovery.readWalEvents(dir, "conn-1"); + check("wal-torn-tail-keeps-every-complete-line-before-it", events.length === 2 && events[0].type === "SESSION_START" && events[1].type === "CALL_START", JSON.stringify(events)); +} + +/* Codex PR #33 review "complete WAL writes before treating them as durable": fs.writeSync + * may write fewer bytes than the caller asked for in a single call. Before this fix, a + * short write here was fsync'd (and reported durable) exactly as if it had written the + * whole line -- readWalEvents' own "stop at the first malformed line" tolerance (its own + * header comment) would then treat that torn interior line as a crash tail and silently + * discard it AND every real, complete event appended after it. */ +function appendWalEventRetriesOnAShortWrite() { + const dir = freshDir("wal-short-write"); + const originalWriteSync = fs.writeSync; + let calls = 0; + fs.writeSync = function (fd, buffer, offset, length, position) { + calls++; + if (calls === 1) { + // Simulate a short write: only the first half of the requested bytes land. + const partial = Math.max(1, Math.floor(length / 2)); + return originalWriteSync(fd, buffer, offset, partial, position); + } + return originalWriteSync.apply(fs, arguments); + }; + try { + recovery.appendWalEvent(dir, "conn-1", { type: "SESSION_START", goal: "g" }); + } finally { + fs.writeSync = originalWriteSync; + } + check("wal-short-write-took-more-than-one-writeSync-call", calls > 1, String(calls)); + const events = recovery.readWalEvents(dir, "conn-1"); + check("wal-short-write-still-produces-one-complete-parseable-event", events.length === 1 && events[0].type === "SESSION_START" && events[0].goal === "g", JSON.stringify(events)); + // A second, real event appended afterward must not be lost the way a genuinely torn + // line would discard it -- confirms the whole first line landed intact on disk, not + // just that JSON.parse happened to tolerate a truncated tail by coincidence. + recovery.appendWalEvent(dir, "conn-1", { type: "INITIALIZE", model: "m" }); + const eventsAfter = recovery.readWalEvents(dir, "conn-1"); + check("wal-short-write-does-not-corrupt-subsequent-events", eventsAfter.length === 2 && eventsAfter[1].type === "INITIALIZE", JSON.stringify(eventsAfter)); +} + +function appendWalEventFailsClosedWhenWriteSyncMakesNoProgress() { + const dir = freshDir("wal-no-progress-write"); + const originalWriteSync = fs.writeSync; + fs.writeSync = function () { return 0; }; + let threw = null; + try { + recovery.appendWalEvent(dir, "conn-1", { type: "SESSION_START", goal: "g" }); + } catch (error) { + threw = error; + } finally { + fs.writeSync = originalWriteSync; + } + check("wal-no-progress-write-throws-rather-than-fsync-nothing", threw && threw.code === "GATEWAY_RECOVERY_SHORT_WRITE", threw && threw.code); +} + +/* Codex PR #33 review "restrict permissions on raw recovery records": these files/dirs + * hold raw goals, tool arguments, and cached results -- data the signed execution trace + * otherwise only ever stores as hashes. Under a common 022 umask, the previous default + * modes (0755 dirs, 0644 files) let any other local user on the host read them. POSIX-only + * (Windows has no equivalent permission bits to assert against). + * + * Codex PR #33 review round 2 "preserve group access when creating recovery state": the + * asserted modes are now the SHARED access model (stateStore.DEFAULT_DIR_MODE 0750 / + * DEFAULT_FILE_MODE 0640) that startup-permissions.js applies to these exact paths, not + * the single-user 0700/0600 literals recovery.js used to reset them to on every write -- + * which silently undid that startup pass and locked same-group ops tooling out again. The + * property under test is unchanged (never the umask-dependent world-readable default); + * only the group bit the later access model deliberately requires has changed. */ +function recoveryFilesAndDirsUseTheSharedAccessModel() { + if (process.platform === "win32") { + record("recovery-permissions-shared-access-model", "SKIP", "POSIX file mode bits do not apply on win32"); + return; + } + const dir = freshDir("permissions"); + recovery.appendWalEvent(dir, "conn-1", { type: "SESSION_START", goal: null }); + const walMode = fs.statSync(recovery.walPath(dir, "conn-1")).mode & 0o777; + check("recovery-wal-file-is-0640", walMode === 0o640, walMode.toString(8)); + check("recovery-wal-file-is-never-world-readable", (walMode & 0o007) === 0, walMode.toString(8)); + const activeDirMode = fs.statSync(recovery.activeDir(dir)).mode & 0o777; + check("recovery-active-dir-is-0750", activeDirMode === 0o750, activeDirMode.toString(8)); + + const intentKey = recovery.computeIntentKey("conn-1", "toolA", {}); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-1", tool: "toolA", arguments: {}, state: "dispatched", dispatched_at: 1 }); + const intentModeAfterCreate = fs.statSync(recovery.intentPath(dir, intentKey)).mode & 0o777; + check("recovery-intent-file-is-0640-after-create", intentModeAfterCreate === 0o640, intentModeAfterCreate.toString(8)); + const intentsDirMode = fs.statSync(recovery.intentsDir(dir)).mode & 0o777; + check("recovery-intents-dir-is-0750", intentsDirMode === 0o750, intentsDirMode.toString(8)); + + recovery.updateIntent(dir, intentKey, { state: "ambiguous", ambiguous_at: 2 }); + const intentModeAfterUpdate = fs.statSync(recovery.intentPath(dir, intentKey)).mode & 0o777; + check("recovery-intent-file-is-0640-after-update", intentModeAfterUpdate === 0o640, intentModeAfterUpdate.toString(8)); + + // The point of the fix: startup-permissions.js's pass and recovery.js's own per-write + // re-tightening must agree, so a write after startup cannot undo it. + check( + "recovery-runtime-modes-match-the-startup-permissions-pass", + activeDirMode === stateStore.DEFAULT_DIR_MODE && walMode === stateStore.DEFAULT_FILE_MODE, + `${activeDirMode.toString(8)}/${walMode.toString(8)} vs ${stateStore.DEFAULT_DIR_MODE.toString(8)}/${stateStore.DEFAULT_FILE_MODE.toString(8)}` + ); +} + +/* Codex PR #33 review "propagate real directory fsync failures": fsyncDir used to swallow + * EVERY error, so a genuine EIO while making a new WAL/intent's DIRECTORY ENTRY durable + * was reported to the caller as success -- the gateway could dispatch believing its fence + * was durable and restart to find no record of it. deleteIntent is the smallest public + * call that ends in fsyncDir and nothing else, so it isolates the behavior exactly. + * Only the directory open (flags "r") is faulted; the record's own unlink is left real. */ +function directoryFsyncRealIoFailurePropagates() { + const dir = freshDir("fsyncdir-eio"); + const intentKey = recovery.computeIntentKey("conn-fsync", "toolA", {}); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-fsync", tool: "toolA", arguments: {}, state: "dispatched", dispatched_at: 1 }); + + const realOpenSync = fs.openSync; + let threw = null; + fs.openSync = function (p, flags, mode) { + if (flags === "r") throw Object.assign(new Error("EIO: simulated directory fsync failure"), { code: "EIO" }); + return realOpenSync.call(fs, p, flags, mode); + }; + try { + recovery.deleteIntent(dir, intentKey); + } catch (error) { + threw = error; + } finally { + fs.openSync = realOpenSync; + } + check("directory-fsync-eio-propagates-instead-of-reporting-success", threw !== null && threw.code === "EIO", threw ? `${threw.code}: ${threw.message}` : "did not throw"); +} + +/* The other half: the cases this helper was actually written to tolerate must still be + * tolerated -- a directory that is simply gone (ENOENT, e.g. mid-teardown in a test) and a + * platform/filesystem that does not support fsync on a directory fd (EINVAL) are NOT real + * I/O faults and must not turn a durably-completed write into a failure. */ +function directoryFsyncUnsupportedAndMissingAreStillTolerated() { + for (const code of ["ENOENT", "EINVAL", "ENOTSUP", "EPERM"]) { + const dir = freshDir(`fsyncdir-tolerated-${code.toLowerCase()}`); + const intentKey = recovery.computeIntentKey("conn-fsync", "toolA", {}); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-fsync", tool: "toolA", arguments: {}, state: "dispatched", dispatched_at: 1 }); + + const realOpenSync = fs.openSync; + let threw = null; + fs.openSync = function (p, flags, mode) { + if (flags === "r") throw Object.assign(new Error(`${code}: simulated`), { code }); + return realOpenSync.call(fs, p, flags, mode); + }; + try { + recovery.deleteIntent(dir, intentKey); + } catch (error) { + threw = error; + } finally { + fs.openSync = realOpenSync; + } + check(`directory-fsync-${code.toLowerCase()}-is-still-tolerated`, threw === null, threw && `${threw.code}: ${threw.message}`); + check(`directory-fsync-${code.toLowerCase()}-still-completed-the-real-work`, recovery.readIntent(dir, intentKey) === null, "intent was not actually deleted"); + } +} + +/* PR #33/#29 merge-verification correction to frontier-panel finding #4: ensureDir's + * ancestor-fsync loop originally fsynced every newly-created ancestor's own CONTENTS + * (e.g. gateway-recovery/'s listing of active/) but never the pre-existing PARENT's + * listing of the topmost newly-created ancestor (e.g. state_dir's listing of + * gateway-recovery/ itself) -- so the very directory-entry loss this comment already + * claimed to prevent was still possible. On a completely fresh state_dir (only the + * top-level dir itself pre-exists), createIntentIfAbsent's ensureDir(intentsDir(...)) + * call must create BOTH gateway-recovery/ and gateway-recovery/intents/ in one + * mkdirSync(recursive:true) -- exactly the multi-level-ancestor case the fix targets. + * Spy on fs.openSync(path, "r") (fsyncDir's own open call) to capture every directory + * path actually fsynced, and assert the fresh state_dir itself -- the pre-existing + * parent whose own listing must record the new gateway-recovery/ entry -- is one of + * them. */ +function ensureDirFsyncsThePreexistingParentOfTheTopmostNewAncestor() { + const stateDir = freshDir("ensuredir-cursor"); + const intentKey = recovery.computeIntentKey("conn-cursor-fsync", "toolA", {}); + + const realOpenSync = fs.openSync; + const openedForFsync = []; + fs.openSync = function (p, flags, mode) { + if (flags === "r") openedForFsync.push(p); + return realOpenSync.call(fs, p, flags, mode); + }; + try { + recovery.createIntentIfAbsent(stateDir, intentKey, { connection_id: "conn-cursor-fsync", tool: "toolA", arguments: {}, state: "dispatched", dispatched_at: 1 }); + } finally { + fs.openSync = realOpenSync; + } + + const gatewayRecoveryDir = recovery.recoveryDir(stateDir); + const intentsDirPath = recovery.intentsDir(stateDir); + check("ensuredir-fsyncs-the-fresh-state-dir-itself-not-just-its-new-children", openedForFsync.includes(stateDir), `fsynced dirs: ${JSON.stringify(openedForFsync)}`); + check("ensuredir-still-fsyncs-the-intermediate-newly-created-ancestor", openedForFsync.includes(gatewayRecoveryDir), `fsynced dirs: ${JSON.stringify(openedForFsync)}`); + check("ensuredir-leaf-itself-fsynced-separately-by-the-write-that-follows", openedForFsync.includes(intentsDirPath), `fsynced dirs: ${JSON.stringify(openedForFsync)}`); +} + +/* Codex PR #33 review "retain operator-confirmed results for reconnect replay": when an + * operator resolves a crashed keyed call as EXECUTED, this used to update only the + * connection-scoped intent. Startup recovery then consumes and deletes that intent, so -- + * unlike proxy.js's live completion path -- nothing ever wrote the connection-INDEPENDENT + * completed-signature record, and a reconnecting agent presenting the same idempotency key + * computed a brand-new intentKey, found nothing, and dispatched the already-executed side + * effect a second time. */ +function operatorConfirmedExecutedRetainsTheCrossConnectionSignature() { + const dir = freshDir("operator-confirmed-signature"); + const intentKey = recovery.computeIntentKey("conn-crashed", "charge", { amount: 5 }); + recovery.createIntentIfAbsent(dir, intentKey, { + connection_id: "conn-crashed", + tool: "charge", + arguments: { amount: 5 }, + state: "dispatched", + dispatched_at: 1, + idempotency_key: "customer-key-1", + generation: 1, + }); + recovery.resolveIntentExecuted(dir, intentKey, { value: 7 }); + + const signatureKey = recovery.computeSignatureKey("charge", { amount: 5 }, "customer-key-1"); + const retained = recovery.readCompletedSignature(dir, signatureKey); + check("operator-confirmed-executed-writes-the-completed-signature", retained !== null, "no retained signature was written"); + check( + "operator-confirmed-executed-signature-carries-the-confirmed-result-and-key", + Boolean(retained) && retained.idempotency_key === "customer-key-1" && retained.cached_result && retained.cached_result.value === 7, + JSON.stringify(retained) + ); + + // The operator's own primary resolution is unchanged by the secondary write. + const resolved = recovery.readIntent(dir, intentKey); + check("operator-confirmed-executed-intent-still-terminal-completed", resolved.state === "completed" && resolved.resolution === "operator_confirmed_executed", JSON.stringify(resolved)); +} + +/* Same review item, negative half: a call with NO caller idempotency key can never be + * replayed by proxy.js's own read gate (which requires retained.idempotency_key to match), + * so resolving it must not leave an unreplayable-by-construction file sitting in the + * signature store until its retention window expires. */ +function operatorConfirmedExecutedWithoutAKeyWritesNoSignature() { + const dir = freshDir("operator-confirmed-no-key"); + const intentKey = recovery.computeIntentKey("conn-crashed", "charge", { amount: 5 }); + recovery.createIntentIfAbsent(dir, intentKey, { + connection_id: "conn-crashed", + tool: "charge", + arguments: { amount: 5 }, + state: "dispatched", + dispatched_at: 1, + idempotency_key: null, + generation: 1, + }); + recovery.resolveIntentExecuted(dir, intentKey, { value: 7 }); + const retained = recovery.readCompletedSignature(dir, recovery.computeSignatureKey("charge", { amount: 5 }, null)); + check("operator-confirmed-executed-without-a-key-writes-no-signature", retained === null, JSON.stringify(retained)); +} + +/* Codex PR #33 review "keep the WAL until corrupt-intent cleanup succeeds": + * listIntentsForConnection reads every intent file on disk before filtering and throws on + * any damaged one -- including an unrelated one. Running it AFTER deleteWal meant + * recovery-abandon could irreversibly destroy this connection's last remaining record and + * then abort, leaving the corrupt fence with nothing left to quarantine or replay. */ +function abandonConnectionKeepsTheWalWhenTheIntentScanFails() { + const dir = freshDir("abandon-intent-scan-failure"); + const keys = makeKeys(); + const connectionId = "conn-scan-failure"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + const realListIntentsForConnection = recovery.listIntentsForConnection; + recovery.listIntentsForConnection = () => { + throw Object.assign(new Error("Corrupt intent record \"gs_int_unrelated\": Unexpected end of JSON input"), { code: "GATEWAY_RECOVERY_INTENT_CORRUPT" }); + }; + let threw = null; + try { + abandonConnection(dir, keys, connectionId, silentLog); + } catch (error) { + threw = error; + } finally { + recovery.listIntentsForConnection = realListIntentsForConnection; + } + check("abandon-intent-scan-failure-surfaces-the-error", threw !== null && threw.code === "GATEWAY_RECOVERY_INTENT_CORRUPT", threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check( + "abandon-intent-scan-failure-leaves-the-wal-intact", + recovery.readWalEvents(dir, connectionId).length > 0, + "the WAL was deleted before the intent scan that failed -- nothing left to quarantine or replay" + ); + + // And the whole command stays re-runnable once the corrupt record is dealt with: the + // bundle is already sealed, so the retry takes the content-verified collision path. + let secondThrow = null; + try { + abandonConnection(dir, keys, connectionId, silentLog); + } catch (error) { + secondThrow = error; + } + check("abandon-intent-scan-failure-is-re-runnable-after-the-scan-is-fixed", secondThrow === null, secondThrow && `${secondThrow.code}: ${secondThrow.message}`); + check("abandon-intent-scan-failure-retry-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present after the successful retry"); +} + +function walListActiveConnectionsAndDelete() { + const dir = freshDir("wal-list-delete"); + recovery.appendWalEvent(dir, "conn-a", { type: "SESSION_START" }); + recovery.appendWalEvent(dir, "conn-b", { type: "SESSION_START" }); + const active = recovery.listActiveConnections(dir).sort(); + check("wal-lists-every-active-connection", JSON.stringify(active) === JSON.stringify(["conn-a", "conn-b"]), JSON.stringify(active)); + recovery.deleteWal(dir, "conn-a"); + const afterDelete = recovery.listActiveConnections(dir); + check("wal-delete-removes-only-that-connection", JSON.stringify(afterDelete) === JSON.stringify(["conn-b"]), JSON.stringify(afterDelete)); + // Deleting an already-gone WAL must be a silent no-op (ENOENT swallowed), not a throw + // -- recoverCrashedSessions relies on this for the "events.length === 0" early-out path. + let threw = null; + try { recovery.deleteWal(dir, "conn-a"); } catch (error) { threw = error; } + check("wal-delete-of-already-deleted-is-a-no-op", threw === null, threw && threw.message); +} + +/* Codex PR #33 review "sort active connections before recovery": readdirSync gives no + * portable ordering guarantee -- recoverCrashedSessions consumes listActiveConnections' + * result directly, in order, to decide chain-append sequence/tail-hash assignment, so an + * unsorted order made replay order (and therefore the resulting chain) platform/fs + * dependent. Deliberately does NOT sort the expectation itself, unlike + * walListActiveConnectionsAndDelete above, to actually exercise the function's own + * ordering guarantee rather than the test's. */ +function walListActiveConnectionsReturnsSortedOrder() { + const dir = freshDir("wal-list-sorted"); + // Appended out of lexical order on purpose. + recovery.appendWalEvent(dir, "conn-c", { type: "SESSION_START" }); + recovery.appendWalEvent(dir, "conn-a", { type: "SESSION_START" }); + recovery.appendWalEvent(dir, "conn-b", { type: "SESSION_START" }); + const active = recovery.listActiveConnections(dir); + check("wal-list-active-connections-is-sorted", JSON.stringify(active) === JSON.stringify(["conn-a", "conn-b", "conn-c"]), JSON.stringify(active)); +} + +function walOnEmptyRecoveryDirReturnsNoActiveConnections() { + const dir = freshDir("wal-empty-root"); + const active = recovery.listActiveConnections(dir); + check("wal-list-on-a-state-dir-with-no-recovery-dir-yet-returns-empty", Array.isArray(active) && active.length === 0, JSON.stringify(active)); +} + +// --------------------------------------------------------------------------- +// Intent-store primitives +// --------------------------------------------------------------------------- + +function intentKeyStableRegardlessOfArgumentKeyOrder() { + const k1 = recovery.computeIntentKey("conn-1", "send_email", { to: "a@example.com", subject: "hi" }); + const k2 = recovery.computeIntentKey("conn-1", "send_email", { subject: "hi", to: "a@example.com" }); + check("intent-key-stable-across-argument-key-order", k1 === k2, `${k1} !== ${k2}`); +} + +function intentKeyDiffersOnConnectionToolOrArgs() { + const base = recovery.computeIntentKey("conn-1", "send_email", { to: "a" }); + const diffConn = recovery.computeIntentKey("conn-2", "send_email", { to: "a" }); + const diffTool = recovery.computeIntentKey("conn-1", "send_sms", { to: "a" }); + const diffArgs = recovery.computeIntentKey("conn-1", "send_email", { to: "b" }); + check("intent-key-differs-on-connection", base !== diffConn, base); + check("intent-key-differs-on-tool", base !== diffTool, base); + check("intent-key-differs-on-arguments", base !== diffArgs, base); +} + +function intentKeyTreatsUndefinedArgsAsNull() { + const k1 = recovery.computeIntentKey("conn-1", "ping", undefined); + const k2 = recovery.computeIntentKey("conn-1", "ping", null); + check("intent-key-treats-undefined-arguments-same-as-null", k1 === k2, `${k1} !== ${k2}`); +} + +function createIntentIfAbsentIsRaceSafe() { + const dir = freshDir("intent-create"); + const key = recovery.computeIntentKey("conn-1", "tool", {}); + const first = recovery.createIntentIfAbsent(dir, key, { connection_id: "conn-1", tool: "tool", arguments: {}, state: "dispatched", dispatched_at: 1 }); + check("create-intent-if-absent-returns-record-first-time", Boolean(first) && first.state === "dispatched", JSON.stringify(first)); + const second = recovery.createIntentIfAbsent(dir, key, { connection_id: "conn-1", tool: "tool", arguments: {}, state: "dispatched", dispatched_at: 2 }); + check("create-intent-if-absent-returns-null-when-already-exists", second === null, JSON.stringify(second)); + // And the ORIGINAL record must be untouched by the second, losing attempt. + const onDisk = recovery.readIntent(dir, key); + check("create-intent-if-absent-does-not-let-a-losing-attempt-overwrite", onDisk.dispatched_at === 1, JSON.stringify(onDisk)); +} + +function readIntentOfMissingKeyReturnsNull() { + const dir = freshDir("intent-missing"); + const result = recovery.readIntent(dir, "gs_doesnotexist"); + check("read-intent-of-missing-key-returns-null", result === null, JSON.stringify(result)); +} + +function updateIntentMergesAndRejectsUnknown() { + const dir = freshDir("intent-update"); + const key = recovery.computeIntentKey("conn-1", "tool", {}); + recovery.createIntentIfAbsent(dir, key, { connection_id: "conn-1", tool: "tool", arguments: {}, state: "dispatched", dispatched_at: 1 }); + const updated = recovery.updateIntent(dir, key, { state: "completed", cached_result: { ok: true } }); + check("update-intent-merges-patch", updated.state === "completed" && updated.cached_result.ok === true && updated.dispatched_at === 1, JSON.stringify(updated)); + let threw = null; + try { recovery.updateIntent(dir, "gs_doesnotexist", { state: "completed" }); } catch (error) { threw = error; } + check("update-intent-of-unknown-key-throws-not-found", threw && threw.code === "GATEWAY_RECOVERY_INTENT_NOT_FOUND", threw && threw.code); +} + +function deleteIntentIsIdempotent() { + const dir = freshDir("intent-delete"); + const key = recovery.computeIntentKey("conn-1", "tool", {}); + recovery.createIntentIfAbsent(dir, key, { connection_id: "conn-1", tool: "tool", arguments: {}, state: "dispatched", dispatched_at: 1 }); + recovery.deleteIntent(dir, key); + check("delete-intent-removes-it", recovery.readIntent(dir, key) === null, "still present"); + let threw = null; + try { recovery.deleteIntent(dir, key); } catch (error) { threw = error; } + check("delete-intent-of-already-deleted-is-a-no-op", threw === null, threw && threw.message); +} + +/* Codex PR #33 review "fsync the intent directory after rollback deletion": proxy.js's own + * CALL_START-failure rollback calls deleteIntent to remove a just-created "dispatched" + * intent it now knows never actually dispatched, then tells the caller a retry is safe -- + * but the unlink's own directory-entry removal was never fsynced, so a crash shortly after + * could resurrect the stale record on the next mount. */ +function deleteIntentFsyncsTheIntentsDirectoryAfterRemoval() { + const dir = freshDir("intent-delete-fsync"); + const key = recovery.computeIntentKey("conn-1", "tool", {}); + recovery.createIntentIfAbsent(dir, key, { connection_id: "conn-1", tool: "tool", arguments: {}, state: "dispatched", dispatched_at: 1 }); + const originalFsyncSync = fs.fsyncSync; + let fsyncCount = 0; + fs.fsyncSync = (fd) => { fsyncCount++; return originalFsyncSync(fd); }; + try { + recovery.deleteIntent(dir, key); + } finally { + fs.fsyncSync = originalFsyncSync; + } + check("delete-intent-fsyncs-the-intents-directory", fsyncCount >= 1, `expected at least 1 fsyncSync call, got ${fsyncCount}`); +} + +function listIntentsForConnectionFiltersCorrectly() { + const dir = freshDir("intent-list-conn"); + const keyA1 = recovery.computeIntentKey("conn-a", "tool1", {}); + const keyA2 = recovery.computeIntentKey("conn-a", "tool2", {}); + const keyB1 = recovery.computeIntentKey("conn-b", "tool1", {}); + recovery.createIntentIfAbsent(dir, keyA1, { connection_id: "conn-a", tool: "tool1", arguments: {}, state: "dispatched" }); + recovery.createIntentIfAbsent(dir, keyA2, { connection_id: "conn-a", tool: "tool2", arguments: {}, state: "completed" }); + recovery.createIntentIfAbsent(dir, keyB1, { connection_id: "conn-b", tool: "tool1", arguments: {}, state: "dispatched" }); + const forA = recovery.listIntentsForConnection(dir, "conn-a").map((i) => i.tool).sort(); + check("list-intents-for-connection-only-returns-that-connections-intents", JSON.stringify(forA) === JSON.stringify(["tool1", "tool2"]), JSON.stringify(forA)); +} + +function listAllIntentsSpansEveryConnection() { + const dir = freshDir("intent-list-all"); + recovery.createIntentIfAbsent(dir, recovery.computeIntentKey("conn-a", "t", {}), { connection_id: "conn-a", tool: "t", arguments: {}, state: "ambiguous", ambiguous_reason: "x" }); + recovery.createIntentIfAbsent(dir, recovery.computeIntentKey("conn-b", "t", {}), { connection_id: "conn-b", tool: "t", arguments: {}, state: "dispatched" }); + const all = recovery.listAllIntents(dir); + check("list-all-intents-spans-every-connection", all.length === 2, String(all.length)); + check("list-all-intents-preserves-each-records-own-state", all.some((i) => i.connection_id === "conn-a" && i.state === "ambiguous") && all.some((i) => i.connection_id === "conn-b" && i.state === "dispatched"), JSON.stringify(all)); +} + +function operatorResolutionExecutedAndNotExecuted() { + const dir = freshDir("intent-resolve"); + const keyExecuted = recovery.computeIntentKey("conn-1", "toolA", {}); + const keyNotExecuted = recovery.computeIntentKey("conn-1", "toolB", {}); + recovery.createIntentIfAbsent(dir, keyExecuted, { connection_id: "conn-1", tool: "toolA", arguments: {}, state: "ambiguous" }); + recovery.createIntentIfAbsent(dir, keyNotExecuted, { connection_id: "conn-1", tool: "toolB", arguments: {}, state: "ambiguous" }); + + const resolved = recovery.resolveIntentExecuted(dir, keyExecuted, { charged: true }); + check("resolve-intent-executed-marks-completed-with-cached-result", resolved.state === "completed" && resolved.cached_result.charged === true, JSON.stringify(resolved)); + + // PR #33 round-2 fix (Codex "persist a terminal not-executed resolution"): this used + // to delete the intent immediately. It now persists a terminal `not_executed` state + // instead, so a crashed connection's still-pending WAL replay can resolve to this real + // decision on the next restart rather than looping back to operator review forever -- + // see gateway.js#recoverCrashedSessions's own handling of this state. The intent is + // deleted only once that replay actually consumes it and the connection is sealed. + const resolvedNotExecuted = recovery.resolveIntentNotExecuted(dir, keyNotExecuted); + check("resolve-intent-not-executed-persists-a-terminal-state", resolvedNotExecuted.state === "not_executed", JSON.stringify(resolvedNotExecuted)); + check("resolve-intent-not-executed-is-not-immediately-deleted", recovery.readIntent(dir, keyNotExecuted) !== null, "was deleted immediately"); +} + +/* Codex PR #33 review "reject resolutions that overwrite terminal intents": both resolve* + * functions used to patch unconditionally regardless of the intent's current state -- a + * stale or repeated operator command could silently rewrite an already-`completed` + * intent's own observed result, or flip a `not_executed` intent the other way, discarding + * real evidence. Only dispatched/ambiguous may transition; a repeat of the SAME terminal + * resolution is a no-op; any OTHER terminal state is refused outright. */ +function resolveIntentGuardsAgainstOverwritingATerminalState() { + const dir = freshDir("intent-resolve-terminal-guard"); + const keyCompleted = recovery.computeIntentKey("conn-1", "toolA", {}); + recovery.createIntentIfAbsent(dir, keyCompleted, { connection_id: "conn-1", tool: "toolA", arguments: {}, state: "dispatched", dispatched_at: 1 }); + recovery.resolveIntentExecuted(dir, keyCompleted, { charged: true }); + + let threwOnRepeat = null; + let repeatResult = null; + try { repeatResult = recovery.resolveIntentExecuted(dir, keyCompleted, { charged: true }); } catch (error) { threwOnRepeat = error; } + check("resolve-executed-repeat-is-a-no-op-not-a-throw", threwOnRepeat === null, threwOnRepeat && threwOnRepeat.message); + check("resolve-executed-repeat-does-not-change-the-record", repeatResult && repeatResult.cached_result.charged === true, JSON.stringify(repeatResult)); + + let threwOnConflict = null; + try { recovery.resolveIntentNotExecuted(dir, keyCompleted); } catch (error) { threwOnConflict = error; } + check("resolve-not-executed-refuses-to-overwrite-a-completed-intent", threwOnConflict && threwOnConflict.code === "GATEWAY_RECOVERY_INTENT_TERMINAL", threwOnConflict && threwOnConflict.code); + check("resolve-not-executed-refusal-did-not-mutate-the-record", recovery.readIntent(dir, keyCompleted).state === "completed", JSON.stringify(recovery.readIntent(dir, keyCompleted))); + + const keyNotExecuted = recovery.computeIntentKey("conn-1", "toolB", {}); + recovery.createIntentIfAbsent(dir, keyNotExecuted, { connection_id: "conn-1", tool: "toolB", arguments: {}, state: "ambiguous" }); + recovery.resolveIntentNotExecuted(dir, keyNotExecuted); + let threwOnReverseConflict = null; + try { recovery.resolveIntentExecuted(dir, keyNotExecuted, { forced: true }); } catch (error) { threwOnReverseConflict = error; } + check("resolve-executed-refuses-to-overwrite-a-not-executed-intent", threwOnReverseConflict && threwOnReverseConflict.code === "GATEWAY_RECOVERY_INTENT_TERMINAL", threwOnReverseConflict && threwOnReverseConflict.code); + check("resolve-executed-refusal-did-not-mutate-the-record", recovery.readIntent(dir, keyNotExecuted).state === "not_executed", JSON.stringify(recovery.readIntent(dir, keyNotExecuted))); + + let threwOnRepeatNotExecuted = null; + try { recovery.resolveIntentNotExecuted(dir, keyNotExecuted); } catch (error) { threwOnRepeatNotExecuted = error; } + check("resolve-not-executed-repeat-is-a-no-op-not-a-throw", threwOnRepeatNotExecuted === null, threwOnRepeatNotExecuted && threwOnRepeatNotExecuted.message); +} + +// --------------------------------------------------------------------------- +// gateway.js#recoverCrashedSessions -- WAL replay through session.js's real functions +// --------------------------------------------------------------------------- + +/** Simulates a live connection up through the WAL events proxy.js itself would have + * appended for a fully-clean session (SESSION_START/INITIALIZE/CALL_START/CALL_RESULT), + * WITHOUT ever calling closeConnection -- exactly what a real crash before disconnect + * leaves behind. */ +function seedCleanCallWal(dir, connectionId) { + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 10 }); + const intentKey = recovery.computeIntentKey(connectionId, "echo", { a: 1 }); + // proxy.js's real dispatch guard always creates the intent record BEFORE the CALL_START + // WAL event is appended (see proxy.js's own handleMessage) -- mirror that ordering here + // so these fixtures match what a real crash actually leaves on disk. + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: connectionId, tool: "echo", arguments: { a: 1 }, state: "dispatched", dispatched_at: 9 }); + return intentKey; +} + +/* Round-1 fix-plan commit 4 test fixture: seedCleanCallWal's own bundle_id formula + * (gsa-mcp-shim.js: sha256({init, grantedTools, n: calls.length})) depends only on the + * CALL COUNT, not on any call's actual arguments -- two connections both built from + * seedCleanCallWal always collide on bundle_id regardless of their argument values. This + * variant makes TWO calls (n=2) instead of one, so a session sealed from it lands on a + * genuinely different bundle_id than seedCleanCallWal's own -- needed to advance the + * chain's tail to a second, distinct entry so a test can then re-present the FIRST + * connection's own (now-ancestor, no-longer-tail) chain entry as a bundle-id collision. */ +function seedTwoCallWal(dir, connectionId) { + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 10 }); + const intentKey1 = recovery.computeIntentKey(connectionId, "echo", { a: 1 }); + recovery.createIntentIfAbsent(dir, intentKey1, { connection_id: connectionId, tool: "echo", arguments: { a: 1 }, state: "dispatched", dispatched_at: 9 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 2, tool: "echo", server: "srv", arguments: { a: 2 }, ts: 12 }); + const intentKey2 = recovery.computeIntentKey(connectionId, "echo", { a: 2 }); + recovery.createIntentIfAbsent(dir, intentKey2, { connection_id: connectionId, tool: "echo", arguments: { a: 2 }, state: "dispatched", dispatched_at: 9 }); + return intentKey2; +} + +function recoverAutoSealsASessionThatCrashedAfterCleanDisconnect() { + const dir = freshDir("recover-clean"); + const keys = makeKeys(); + const connectionId = "conn-clean"; + const intentKey = seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + // No CLOSING event -- the crash happened after the response was recorded but before + // the agent ever disconnected. Everything needed to seal is already in the WAL. + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-clean-crash-produces-no-pending-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("recover-clean-crash-deletes-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + const headEntry = chain.readHead(dir); + check("recover-clean-crash-appends-exactly-one-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +function recoverClosesFinding1GapUsingCompletedIntentAsProofOfOutcome() { + const dir = freshDir("recover-finding1-gap"); + const keys = makeKeys(); + const connectionId = "conn-gap"; + const intentKey = seedCleanCallWal(dir, connectionId); + // THE Finding-1 gap: proxy.js's real code marks the intent "completed" BEFORE it + // appends the CALL_RESULT WAL line -- a crash in between leaves exactly this shape. + recovery.updateIntent(dir, intentKey, { state: "completed", completed_at: 12, cached_result: { proven: "from-intent-not-wal" } }); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-closes-finding1-gap-without-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("recover-closes-finding1-gap-deletes-the-now-redundant-completed-intent", recovery.readIntent(dir, intentKey) === null, "intent still present"); + const headEntry = chain.readHead(dir); + check("recover-closes-finding1-gap-still-produces-a-sealed-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +function recoverLeavesUnprovenInFlightCallForOperatorReview() { + const dir = freshDir("recover-unproven"); + const keys = makeKeys(); + const connectionId = "conn-unproven"; + seedCleanCallWal(dir, connectionId); + // The intent is still "dispatched" -- the crash happened with truly no proof either + // way. This must NOT be auto-sealed by guessing. + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-flags-unproven-in-flight-call-for-operator-review", pendingOperatorReview.includes(connectionId), JSON.stringify(pendingOperatorReview)); + check("recover-does-not-touch-the-wal-of-a-flagged-connection", recovery.readWalEvents(dir, connectionId).length > 0, "WAL was deleted"); + check("recover-does-not-seal-anything-for-a-flagged-connection", chain.readHead(dir) === null, "a chain entry was appended despite no proven outcome"); +} + +function recoverIsIdempotentAcrossACrashDuringRecoveryItself() { + const dir = freshDir("recover-repeat"); + const keys = makeKeys(); + const connectionId = "conn-repeat"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + // First recovery pass succeeds and would normally clean up -- but simulate THIS + // process itself crashing between chain.appendSession succeeding and deleteWal + // running, by re-seeding the identical WAL afterward. + recoverCrashedSessions(dir, keys, silentLog); + const eventsAfterFirst = recovery.readWalEvents(dir, connectionId); // [] -- already cleaned up + check("recover-first-pass-cleans-up-normally", eventsAfterFirst.length === 0, JSON.stringify(eventsAfterFirst)); + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-second-pass-on-identical-content-is-not-flagged-for-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("recover-second-pass-still-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + const headEntry = chain.readHead(dir); + check("recover-second-pass-does-not-append-a-second-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +function recoverDeletesEmptyOrFullyTornWalWithoutFlagging() { + const dir = freshDir("recover-empty-wal"); + const keys = makeKeys(); + const connectionId = "conn-empty"; + // A crash mid-write of the very FIRST line leaves nothing parseable at all. + fs.mkdirSync(recovery.activeDir(dir), { recursive: true }); + fs.writeFileSync(recovery.walPath(dir, connectionId), '{"type":"SESSION_ST'); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-does-not-flag-a-connection-with-no-parseable-wal-content", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("recover-deletes-the-unusable-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); +} + +/* Cluster A (generation-aware crash recovery): the intent file's `generation` is + * overwritten IN PLACE on every supersede -- by the time recovery runs it always + * reflects the MOST RECENT generation, not necessarily the one a specific crashed + * CALL_START event belongs to. Reproduces the exact gap this closes: generation 1 + * completes for real but its own CALL_RESULT WAL write never lands (proxy.js's + * documented best-effort append -- a real ENOSPC, not a crash), then the SAME + * connection later dispatches and cleanly completes generation 2 of the identical + * (tool, arguments), whose CALL_RESULT WAL write DOES land. A subsequent real crash + * leaves generation 1's own CALL_START forever unresolved in the WAL while the intent + * file now sits at generation 2. Before this fix, recoverCrashedSessions would have + * bound generation 2's cached_result (a DIFFERENT call's real outcome) to generation 1's + * pending call -- a false attestation. */ +function recoverDoesNotBindALaterGenerationsResultToAnEarlierCrashedGenerationsCall() { + const dir = freshDir("recover-generation-mismatch"); + const keys = makeKeys(); + const connectionId = "conn-gen-mismatch"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + // Generation 1: dispatched, completes for real, but its CALL_RESULT WAL line never lands. + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 10, generation: 1 }); + const intentKey = recovery.computeIntentKey(connectionId, "echo", { a: 1 }); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: connectionId, tool: "echo", arguments: { a: 1 }, state: "dispatched", dispatched_at: 9, generation: 1 }); + recovery.updateIntent(dir, intentKey, { state: "completed", completed_at: 11, cached_result: { from: "generation-1" }, generation: 1 }); + // Generation 2 (same connection, same tool+arguments, no caller idempotency key -- + // proxy.js's supersede path): dispatched, ALSO completes for real, and this time its + // CALL_RESULT WAL line DOES land. + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 2, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 20, generation: 2 }); + recovery.updateIntent(dir, intentKey, { state: "dispatched", dispatched_at: 19, generation: 2, cached_result: undefined, completed_at: undefined }); + recovery.updateIntent(dir, intentKey, { state: "completed", completed_at: 21, cached_result: { from: "generation-2" }, generation: 2 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 2, result: { from: "generation-2" }, isError: false, ts: 21, generation: 2 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check( + "recover-does-not-auto-seal-a-generation-mismatched-crashed-call", + pendingOperatorReview.includes(connectionId), + JSON.stringify(pendingOperatorReview) + ); + check("recover-leaves-the-wal-in-place-for-a-generation-mismatch", recovery.readWalEvents(dir, connectionId).length > 0, "WAL was deleted despite an unresolved generation-1 call"); + check("recover-does-not-seal-a-chain-entry-for-a-generation-mismatch", chain.readHead(dir) === null, "a chain entry was appended despite generation 1's own outcome being unproven"); +} + +/* Cluster B (session-identity distinctness in the audit trail): replaying the identical + * crash-left WAL -- including its own persisted session_id, exactly as proxy.js's real + * openConnection writes it -- in two completely independent recovery runs must produce + * the identical bundle_id both times (replay reuses the persisted session_id; it never + * mints a fresh one -- see session.js#createSession's own doc comment). */ +function walReplayedTwiceInIndependentStateDirsProducesIdenticalBundleId() { + const keys = makeKeys(); + const connectionId = "conn-replay-twice"; + function buildAndRecover() { + const dir = freshDir("replay-twice"); + recovery.appendWalEvent(dir, connectionId, { + type: "SESSION_START", + started_at: 1, + goal: null, + session_id: "fixed-session-id-for-replay-determinism-test", + tools: [{ name: "echo", server: "srv", schema: {} }], + }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 10, generation: 1 }); + const intentKey = recovery.computeIntentKey(connectionId, "echo", { a: 1 }); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: connectionId, tool: "echo", arguments: { a: 1 }, state: "dispatched", dispatched_at: 9, generation: 1 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11, generation: 1 }); + recoverCrashedSessions(dir, keys, silentLog); + return chain.readHead(dir); + } + const headA = buildAndRecover(); + const headB = buildAndRecover(); + check( + "wal-replayed-twice-in-independent-envs-produces-identical-bundle-id", + Boolean(headA && headB && typeof headA.bundle_id === "string" && headA.bundle_id === headB.bundle_id), + JSON.stringify({ headA, headB }) + ); +} + +// --------------------------------------------------------------------------- +// PR #33 round-2 fixes: path-safety, bundle-collision content verification, +// per-connection isolation, not_executed replay, and anomaly-WAL replay. +// --------------------------------------------------------------------------- + +function pathBearingIdentifiersAreRejected() { + const dir = freshDir("path-safety"); + let threwWal = null; + try { recovery.walPath(dir, "../../etc/passwd"); } catch (error) { threwWal = error; } + check("wal-path-rejects-traversal-connection-id", threwWal && threwWal.code === "GATEWAY_RECOVERY_UNSAFE_ID", threwWal && threwWal.message); + let threwIntent = null; + try { recovery.intentPath(dir, "../outside"); } catch (error) { threwIntent = error; } + check("intent-path-rejects-traversal-intent-key", threwIntent && threwIntent.code === "GATEWAY_RECOVERY_UNSAFE_ID", threwIntent && threwIntent.message); + // A normal, real-shaped id (gs_, or a plain connection id) must still work. + let ok = null; + try { ok = recovery.walPath(dir, "conn-abc123_.-"); } catch (error) { ok = error; } + check("wal-path-accepts-a-normal-id", typeof ok === "string", ok && ok.message); +} + +function bundleCollisionWithDifferentContentIsFlaggedNotDiscarded() { + const dir = freshDir("recover-collision"); + const keys = makeKeys(); + // First connection: seals normally, occupying a bundle_id derived only from + // {init, grantedTools, n: calls.length} -- NOT from the actual call content. + seedCleanCallWal(dir, "conn-x"); + recovery.appendWalEvent(dir, "conn-x", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const firstHead = chain.readHead(dir); + + // Second, DIFFERENT connection: same tool, same init shape, same call COUNT (1) -- + // deliberately different arguments/result, which gsa-mcp-shim.js's bundle_id formula + // does not account for, so this collides on bundle_id despite being a genuinely + // different session's content. + const connectionId = "conn-y"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 999 }, ts: 10 }); + const intentKeyY = recovery.computeIntentKey(connectionId, "echo", { a: 999 }); + recovery.createIntentIfAbsent(dir, intentKeyY, { connection_id: connectionId, tool: "echo", arguments: { a: 999 }, state: "dispatched", dispatched_at: 9 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: false, different: true }, isError: false, ts: 11 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("bundle-collision-genuine-conflict-flagged-for-operator", pendingOperatorReview.includes(connectionId), JSON.stringify(pendingOperatorReview)); + check("bundle-collision-genuine-conflict-wal-not-discarded", recovery.readWalEvents(dir, connectionId).length > 0, "WAL was deleted despite unverified collision"); + check("bundle-collision-genuine-conflict-intent-not-discarded", recovery.readIntent(dir, intentKeyY) !== null, "intent was deleted despite unverified collision"); + const headAfter = chain.readHead(dir); + check("bundle-collision-genuine-conflict-no-second-chain-entry-appended", headAfter && headAfter.seq === firstHead.seq, JSON.stringify(headAfter)); +} + +/* Round-1 fix-plan commit 4 regression test ("WAL-survives-failed-cleanup"): reproduces + * the exact scenario the rewritten verifyAndRepairBundleCollision exists for. The OLD + * shipped check compared HEAD directly against the colliding bundle's OWN chain entry -- + * which breaks the instant any OTHER session sealed afterward and moved HEAD on, even + * though the original entry is still a perfectly valid, verified ANCESTOR of the current + * tail. Sequence: connection A seals (chain tail becomes A's entry); a different + * connection B then also seals (chain tail advances to B's entry, A's entry is now an + * ancestor, not the tail); A's own WAL/intent is then re-seeded EXACTLY as it looked + * before recovery ever cleaned it up (its own real "recovery completed but the WAL + * unlink never landed" crash signature) and recovery is run a third time. The rewritten + * check (HEAD equals the chain's own current tail, not equals this bundle's own entry) + * must allow this cleanup rather than refusing it. */ +function bundleCollisionRewriteAllowsAncestorEntryWhenHeadPointsAtANewerTail() { + const dir = freshDir("collision-ancestor-ok"); + const keys = makeKeys(); + const connA = "conn-ancestor-a"; + seedCleanCallWal(dir, connA); + recovery.appendWalEvent(dir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const entryA = chain.readHead(dir); + check("collision-ancestor-fixture-a-sealed", entryA && entryA.seq === 1, JSON.stringify(entryA)); + + const connB = "conn-ancestor-b"; + seedTwoCallWal(dir, connB); + recovery.appendWalEvent(dir, connB, { type: "CALL_RESULT", call_seq: 2, result: { ok: true }, isError: false, ts: 13 }); + recoverCrashedSessions(dir, keys, silentLog); + const entryB = chain.readHead(dir); + check( + "collision-ancestor-fixture-b-sealed-as-new-tail", + entryB && entryB.seq === 2 && entryB.bundle_id !== entryA.bundle_id, + JSON.stringify({ entryA, entryB }) + ); + + // Re-seed connection A's WAL with IDENTICAL content -- the WAL-survives-a-failed- + // cleanup scenario: recovery's own post-seal cleanup for A crashed before deleting its + // WAL/intent, leaving an exact duplicate on disk that a later recovery pass must still + // be able to clean up even though HEAD has since moved on to B's entry. + seedCleanCallWal(dir, connA); + recovery.appendWalEvent(dir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("collision-ancestor-not-flagged-for-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("collision-ancestor-wal-cleaned-up", recovery.readWalEvents(dir, connA).length === 0, "WAL still present after ancestor-collision cleanup"); + const headAfter2 = chain.readHead(dir); + check( + "collision-ancestor-head-still-points-at-b-not-duplicated", + headAfter2 && headAfter2.seq === 2 && headAfter2.bundle_id === entryB.bundle_id, + JSON.stringify(headAfter2) + ); + const chainEntriesAfter = chain.readChain(dir); + check("collision-ancestor-no-third-chain-entry-appended", chainEntriesAfter.length === 2, JSON.stringify(chainEntriesAfter.map((e) => e.seq))); +} + +/* Round-1 fix-plan commit 4 regression test (the negative half of the rewrite above): + * proves the new HEAD-vs-tail check has real teeth, not just permissiveness. Same setup + * (A seals, then B seals as the new tail) but HEAD.json is then corrupted into a genuine + * fork -- same seq/bundle_id as A's own real entry, but a bogus entry_sha256 that does + * not match anything actually in chain.jsonl -- before A's WAL is re-seeded and recovery + * run again. This is neither "HEAD already equals the tail" nor a genuine single-step-lag + * ancestor of it, so the shared classifier must refuse (flag for operator review, leave + * the WAL/intent/chain untouched) rather than silently accept it as another instance of + * the legitimate ancestor case above. */ +function bundleCollisionRewriteRefusesOnAGenuineForkNotJustAnyAncestorMismatch() { + const dir = freshDir("collision-fork-refuses"); + const keys = makeKeys(); + // Round-1 fix-plan commit 6: this test's own fork below is exactly the genuine + // structural failure that now latches chain.js's process-local admission gate + // (getChainIntegrityFailure). Reset before AND after, so this test's fixture never + // depends on latch state left behind by an earlier test, and never leaks its own + // latch into whatever runs next in this same process/file. + chain._resetChainIntegrityFailureForTests(); + const connA = "conn-fork-a"; + seedCleanCallWal(dir, connA); + recovery.appendWalEvent(dir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const entryA = chain.readHead(dir); + + const connB = "conn-fork-b"; + seedTwoCallWal(dir, connB); + recovery.appendWalEvent(dir, connB, { type: "CALL_RESULT", call_seq: 2, result: { ok: true }, isError: false, ts: 13 }); + recoverCrashedSessions(dir, keys, silentLog); + + // Corrupt HEAD.json into a genuine fork: same seq/bundle_id as entry A's real position, + // but a bogus entry_sha256 that matches no real entry in chain.jsonl -- distinct from + // the legitimate single-step-lag ancestor case the classifier auto-repairs. + const forkedHead = Object.assign({}, entryA, { entry_sha256: "f".repeat(64) }); + fs.writeFileSync(chain.headPath(dir), JSON.stringify(forkedHead)); + + seedCleanCallWal(dir, connA); + recovery.appendWalEvent(dir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("collision-fork-flagged-for-operator-review", pendingOperatorReview.includes(connA), JSON.stringify(pendingOperatorReview)); + check("collision-fork-wal-not-discarded", recovery.readWalEvents(dir, connA).length > 0, "WAL deleted despite an unresolved fork"); + const headAfterFork = JSON.parse(fs.readFileSync(chain.headPath(dir), "utf8")); + check("collision-fork-head-left-untouched-not-silently-fixed", headAfterFork.entry_sha256 === forkedHead.entry_sha256, JSON.stringify(headAfterFork)); + chain._resetChainIntegrityFailureForTests(); +} + +/* Codex PR #33 review "verify the chain entry before cleaning a colliding WAL": content + * verification alone proves the BUNDLE FILE on disk really is this connection's own + * durable record -- it does not prove chain.appendSession's own chain.jsonl append (step + * 2 of its own documented write order) ever actually happened. Simulates exactly that + * partial-append crash (bundle file present and content-verified, chain.jsonl/HEAD.json + * still reflecting "no chain yet") by sealing normally once, then stripping only the + * chain.jsonl/HEAD.json side of that append and re-seeding the identical WAL as a second + * crash-left copy would look. Recovery must complete the missing append, not just discard + * the WAL believing nothing more was needed. */ +function recoverRepairsAPartialAppendMissingItsChainEntry() { + const dir = freshDir("recover-partial-append"); + const keys = makeKeys(); + const connectionId = "conn-partial"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const firstHead = chain.readHead(dir); + check("partial-append-fixture-first-pass-sealed", firstHead && firstHead.seq === 1, JSON.stringify(firstHead)); + + // Simulate the crash landing between chain.appendSession's own bundle-file write and its + // chain.jsonl append: the bundle file stays exactly as written; chain.jsonl/HEAD.json are + // rolled back to "nothing appended yet". + fs.unlinkSync(chain.chainPath(dir)); + fs.unlinkSync(chain.headPath(dir)); + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("partial-append-repair-does-not-need-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("partial-append-repair-completes-the-missing-chain-entry", chain.chainHasEntryForBundle(dir, firstHead.bundle_id), "chain.jsonl still has no entry for the content-verified bundle"); + const headAfter = chain.readHead(dir); + check("partial-append-repair-head-points-at-the-repaired-entry", headAfter && headAfter.seq === 1 && headAfter.bundle_id === firstHead.bundle_id, JSON.stringify(headAfter)); + check("partial-append-repair-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); +} + +/* Codex PR #33 review "renew ownership during synchronous recovery": recoverCrashedSessions' + * own per-connection loop is entirely synchronous fs work that can, with enough or large + * enough crash-left WALs, run long enough to starve the writer-claim's async heartbeat past + * its own staleAfterMs -- writer-claim.js's own startHeartbeat() doc comment discloses + * exactly this gap and prescribes a synchronous renew() call at the boundary of any long + * synchronous phase the claim is held across. */ +function recoverCrashedSessionsRenewsTheWriterClaimPerConnection() { + const dir = freshDir("recover-renew"); + const keys = makeKeys(); + seedCleanCallWal(dir, "conn-1"); + recovery.appendWalEvent(dir, "conn-1", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + seedCleanCallWal(dir, "conn-2"); + recovery.appendWalEvent(dir, "conn-2", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + // No maybeRenew() on this fake -- createLeaseGuard's own maybeRenew() wrapper treats + // its absence as a no-op (exactly like an omitted writerClaim), so only this fixture's + // TWO unconditional renew() call sites per connection (the per-connection renew at the + // top of the loop, and the pre-append renew immediately before chain.appendSession) + // are actually counted here. + const fakeWriterClaim = { renew: () => { renewCalls++; } }; + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog, fakeWriterClaim); + check("recover-renews-the-claim-twice-per-connection", renewCalls === 4, String(renewCalls)); + check("recover-with-a-fake-claim-still-seals-normally", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); +} + +function recoverCrashedSessionsWithoutAWriterClaimStillWorks() { + // Every existing direct caller of recoverCrashedSessions (this whole test file) omits + // the writerClaim argument -- confirms it stays fully optional/backward-compatible. + const dir = freshDir("recover-no-claim-arg"); + const keys = makeKeys(); + seedCleanCallWal(dir, "conn-1"); + recovery.appendWalEvent(dir, "conn-1", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + let threw = null; + try { recoverCrashedSessions(dir, keys, silentLog); } catch (error) { threw = error; } + check("recover-without-a-writer-claim-argument-does-not-throw", threw === null, threw && threw.message); +} + +function recoverCrashedSessionsAbortsImmediatelyIfTheClaimIsLost() { + const dir = freshDir("recover-renew-lost"); + const keys = makeKeys(); + seedCleanCallWal(dir, "conn-a"); + recovery.appendWalEvent(dir, "conn-a", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + seedCleanCallWal(dir, "conn-b"); + recovery.appendWalEvent(dir, "conn-b", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + const fakeWriterClaim = { + renew: () => { + renewCalls++; + const error = new Error("Lost the writer-claim: another writer has taken over."); + error.code = "WRITER_CLAIM_LOST"; + throw error; + }, + }; + let threw = null; + try { + recoverCrashedSessions(dir, keys, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("recover-aborts-when-the-claim-is-lost-mid-pass", threw && threw.code === "WRITER_CLAIM_LOST", threw && threw.code); + check("recover-stops-at-the-very-first-renew-failure", renewCalls === 1, String(renewCalls)); + // The claim was lost before EITHER connection's own body ran -- neither WAL may be + // touched, since this process can no longer prove it is still the sole writer. + const remaining = recovery.listActiveConnections(dir); + check("recover-leaves-every-connections-wal-untouched-once-the-claim-is-lost", remaining.length === 2 && remaining.includes("conn-a") && remaining.includes("conn-b"), JSON.stringify(remaining)); + check("recover-appended-no-chain-entry-once-the-claim-is-lost", chain.readHead(dir) === null, JSON.stringify(chain.readHead(dir))); +} + +/* Round-1 fix-plan item 1 (lease keepalive, generalized): renew()'s own failure can be a + * raw fs error (e.g. EIO from writer-claim.js's own openSync) that carries no + * WRITER_CLAIM_LOST code at all -- the required comparison is by reference identity + * (createLeaseGuard's own isLeaseError), never `error.code === "WRITER_CLAIM_LOST"` string + * matching, so a raw fs error must be treated exactly the same as a named claim-loss + * error: fatal to the whole recovery pass, never downgraded to this connection's own + * per-connection "flag for operator review, continue" handling. This exercises the + * UNCONDITIONAL renew() immediately before chain.appendSession specifically (not the + * per-connection renew() at the top of the loop, already covered by + * recoverCrashedSessionsAbortsImmediatelyIfTheClaimIsLost above). */ +function recoverAbortsOnARawFsErrorFromThePreAppendRenewNotJustNamedClaimCodes() { + const dir = freshDir("recover-raw-fs-error-pre-append"); + const keys = makeKeys(); + seedCleanCallWal(dir, "conn-a"); + recovery.appendWalEvent(dir, "conn-a", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + const rawFsError = new Error("EIO: i/o error, open '/state/writer-claim.json'"); + rawFsError.errno = -5; + rawFsError.code = "EIO"; // deliberately NOT "WRITER_CLAIM_LOST" -- a genuine raw fs error + const fakeWriterClaim = { + renew: () => { + renewCalls++; + if (renewCalls === 2) throw rawFsError; // 1: top-of-loop renew, 2: pre-append renew + }, + }; + let threw = null; + try { + recoverCrashedSessions(dir, keys, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("pre-append-renew-raw-fs-error-aborts-with-the-exact-detected-error", threw === rawFsError, threw && threw.message); + check("pre-append-renew-raw-fs-error-never-reaches-chain-append", chain.readHead(dir) === null, JSON.stringify(chain.readHead(dir))); +} + +/* Round-1 fix-plan item 1: the per-event maybeRenew() call inside recoverCrashedSessions' + * own WAL-replay loop must detect a lease taken over by a competing writer mid-replay -- + * not only once per connection at the loop's own top. This also proves the detected loss + * aborts the WHOLE function, not merely the one connection whose replay happened to be + * running when it fired: conn-1 (whose replay/seal finishes before the simulated takeover + * fires) is fully sealed into the chain, but conn-2 (where the takeover is actually + * detected) is left completely untouched -- recoverCrashedSessions never falls through to + * its own per-connection "flag for operator review, continue with the next connection" + * handling for a lease-loss error, unlike every other failure mode that same per-connection + * try/catch already tolerates. */ +function recoverAbortsOnLeaseTakeoverDuringWalReplayNotJustOneConnection() { + const dir = freshDir("recover-takeover-mid-replay"); + const keys = makeKeys(); + seedCleanCallWal(dir, "conn-1"); + recovery.appendWalEvent(dir, "conn-1", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + seedCleanCallWal(dir, "conn-2"); + recovery.appendWalEvent(dir, "conn-2", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let callCount = 0; + const takeoverError = new Error("simulated competing writer took over this state_dir mid-replay"); + const fakeWriterClaim = { + renew: () => {}, + maybeRenew: () => { + callCount++; + // Each connection's WAL replay above has 4 events (SESSION_START, INITIALIZE, + // CALL_START, CALL_RESULT), so calls 1-4 are conn-1's own replay and call 5 is + // conn-2's very first event -- strictly after conn-1's own replay/seal/chain-append + // has already completed in full. + if (callCount === 5) throw takeoverError; + }, + }; + let threw = null; + try { + recoverCrashedSessions(dir, keys, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("takeover-mid-replay-aborts-with-the-exact-detected-error", threw === takeoverError, threw && threw.message); + check("takeover-mid-replay-conn-1-was-already-fully-sealed-first", chain.readHead(dir) !== null, JSON.stringify(chain.readHead(dir))); + check( + "takeover-mid-replay-conn-2-left-completely-untouched-not-flagged-for-operator-review", + recovery.readWalEvents(dir, "conn-2").length === 4, + String(recovery.readWalEvents(dir, "conn-2").length) + ); +} + +/* Round-1 fix-plan item 1: a lease loss detected by verifyAndRepairBundleCollision's own + * leaseGuard.renew() (called immediately before chain.repairMissingChainEntry) must not be + * relabeled as an ordinary "RECOVERY CHAIN-REPAIR FAILURE" -- it must propagate by + * reference identity all the way out of recoverCrashedSessions entirely, exactly like every + * other detected lease loss in this same function. */ +function recoverRepairPathLeaseLossIsNotRelabeledAsAnOrdinaryFailure() { + const dir = freshDir("recover-repair-path-lease-loss"); + const keys = makeKeys(); + const connectionId = "conn-repair-lease"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const firstHead = chain.readHead(dir); + check("repair-path-lease-loss-fixture-first-pass-sealed", firstHead && firstHead.seq === 1, JSON.stringify(firstHead)); + + // Roll chain.jsonl/HEAD.json back to "nothing appended yet" (same partial-append + // simulation recoverRepairsAPartialAppendMissingItsChainEntry above uses), then replay + // the identical WAL again so the second pass hits GATEWAY_BUNDLE_ID_COLLISION and + // attempts the repair path. + fs.unlinkSync(chain.chainPath(dir)); + fs.unlinkSync(chain.headPath(dir)); + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + const rawError = new Error("simulated lease loss during the repair-path renew"); + const fakeWriterClaim = { + renew: () => { + renewCalls++; + if (renewCalls === 3) throw rawError; // 1: top-of-loop, 2: pre-append, 3: repair-path + }, + }; + let threw = null; + try { + recoverCrashedSessions(dir, keys, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("repair-path-lease-loss-propagates-by-reference-not-relabeled", threw === rawError, threw && threw.message); +} + +function unreadableWalForOneConnectionDoesNotBlockAnother() { + const dir = freshDir("recover-isolation"); + const keys = makeKeys(); + // conn-bad's WAL "file" is actually a directory -- fs.readFileSync fails with EISDIR, + // a genuine fs-level read error (GATEWAY_RECOVERY_WAL_UNREADABLE), not a content/JSON + // problem readWalEvents already tolerates. + fs.mkdirSync(recovery.activeDir(dir), { recursive: true }); + fs.mkdirSync(recovery.walPath(dir, "conn-bad")); + // conn-good is a normal, cleanly-recoverable connection. + seedCleanCallWal(dir, "conn-good"); + recovery.appendWalEvent(dir, "conn-good", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("isolation-unreadable-connection-flagged-not-thrown", pendingOperatorReview.includes("conn-bad"), JSON.stringify(pendingOperatorReview)); + check("isolation-good-connection-still-recovered", !pendingOperatorReview.includes("conn-good"), JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("isolation-good-connections-chain-entry-appended-despite-sibling-failure", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +/* Codex PR #33 review "verify collisions in recovery-abandon before cleanup": unlike + * recoverCrashedSessions' own already-hardened GATEWAY_BUNDLE_ID_COLLISION handling + * (bundleCollisionWithDifferentContentIsFlaggedNotDiscarded above), abandonConnection used + * to treat EVERY collision on that error code as the expected repeated-attempt case and + * fall straight through to deleting the connection's WAL/intents -- discarding a genuinely + * DIFFERENT session's only remaining, unrecoverable record. */ +function abandonConnectionRefusesOnUnverifiedBundleCollision() { + const dir = freshDir("abandon-collision"); + const keys = makeKeys(); + // First connection: seals normally via recoverCrashedSessions, occupying a bundle_id + // derived only from {init, grantedTools, n: calls.length} -- not the actual content. + seedCleanCallWal(dir, "conn-x"); + recovery.appendWalEvent(dir, "conn-x", { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(dir, keys, silentLog); + const firstHead = chain.readHead(dir); + + // Second, DIFFERENT connection an operator now runs recovery-abandon on: same tool, + // same init shape, same call count (1) -- deliberately different arguments/result, so + // it collides on bundle_id despite being a genuinely different session's content. + const connectionId = "conn-y"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 999 }, ts: 10 }); + const intentKeyY = recovery.computeIntentKey(connectionId, "echo", { a: 999 }); + recovery.createIntentIfAbsent(dir, intentKeyY, { connection_id: connectionId, tool: "echo", arguments: { a: 999 }, state: "dispatched", dispatched_at: 9 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: false, different: true }, isError: false, ts: 11 }); + + let threw = null; + try { abandonConnection(dir, keys, connectionId, silentLog); } catch (error) { threw = error; } + check("abandon-collision-refuses-rather-than-silently-cleaning-up", threw !== null, "abandonConnection did not throw on an unverified collision"); + check("abandon-collision-wal-not-discarded", recovery.readWalEvents(dir, connectionId).length > 0, "WAL was deleted despite unverified collision"); + check("abandon-collision-intent-not-discarded", recovery.readIntent(dir, intentKeyY) !== null, "intent was deleted despite unverified collision"); + const headAfter = chain.readHead(dir); + check("abandon-collision-no-second-chain-entry-appended", headAfter && headAfter.seq === firstHead.seq, JSON.stringify(headAfter)); +} + +/* Same GATEWAY_BUNDLE_ID_COLLISION content-verification path, but for the ordinary case + * where recovery-abandon really is re-run on the SAME already-sealed connection (matching + * content) -- must still succeed and clean up normally, not regress into refusing every + * collision unconditionally. */ +function abandonConnectionStillSucceedsOnAGenuineRepeatedAttempt() { + const dir = freshDir("abandon-collision-repeat"); + const keys = makeKeys(); + const connectionId = "conn-repeat"; + const intentKey = seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + // First abandon: seals it for real. + abandonConnection(dir, keys, connectionId, silentLog); + const firstHead = chain.readHead(dir); + // Re-seed the IDENTICAL WAL/intent (as a crash-left copy of the same connection's state + // would look on disk) and abandon it again -- same content, same bundle_id, expected. + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + let threw = null; + try { abandonConnection(dir, keys, connectionId, silentLog); } catch (error) { threw = error; } + check("abandon-repeated-genuine-attempt-does-not-throw", threw === null, threw && threw.message); + check("abandon-repeated-genuine-attempt-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + const headAfter = chain.readHead(dir); + check("abandon-repeated-genuine-attempt-no-duplicate-chain-entry", headAfter && headAfter.seq === firstHead.seq, JSON.stringify(headAfter)); +} + +/* Round-1 fix-plan item 1: same unconditional pre-append renew() as recoverCrashedSessions' + * own normal path, applied to abandonConnection's own single-connection chain-append. This + * function has no per-connection try/catch of its own -- a detected lease loss simply + * propagates uncaught out to runRecoveryAbandonCli's caller, exactly as fatal as + * recoverCrashedSessions' own explicit rethrow. */ +function abandonConnectionAbortsWhenThePreAppendRenewFails() { + const dir = freshDir("abandon-pre-append-renew-fails"); + const keys = makeKeys(); + const connectionId = "conn-abandon-renew-fail"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + const rawError = new Error("simulated lease loss immediately before abandonConnection's own chain.appendSession"); + const fakeWriterClaim = { + renew: () => { + renewCalls++; + if (renewCalls === 2) throw rawError; // 1: top-of-function renew, 2: pre-append renew + }, + }; + let threw = null; + try { + abandonConnection(dir, keys, connectionId, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("abandon-pre-append-renew-failure-aborts-with-the-exact-detected-error", threw === rawError, threw && threw.message); + check("abandon-pre-append-renew-failure-never-reaches-chain-append", chain.readHead(dir) === null, JSON.stringify(chain.readHead(dir))); + check("abandon-pre-append-renew-failure-leaves-the-wal-in-place", recovery.readWalEvents(dir, connectionId).length > 0, "WAL was deleted despite the aborted renew"); +} + +/* Round-1 fix-plan item 1: same reference-identity, not-relabeled requirement as + * recoverCrashedSessions' own repair path, for abandonConnection's own + * GATEWAY_BUNDLE_ID_COLLISION handling -- the repair-path lease loss must be rethrown + * unwrapped, not wrapped in fail()'s own GATEWAY_RECOVERY_CHAIN_REPAIR_FAILED. */ +function abandonConnectionRepairPathLeaseLossIsNotRelabeledAsChainRepairFailure() { + const dir = freshDir("abandon-repair-path-lease-loss"); + const keys = makeKeys(); + const connectionId = "conn-abandon-repair-lease"; + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + abandonConnection(dir, keys, connectionId, silentLog); + const firstHead = chain.readHead(dir); + check("abandon-repair-path-lease-loss-fixture-first-pass-sealed", firstHead && firstHead.seq === 1, JSON.stringify(firstHead)); + + fs.unlinkSync(chain.chainPath(dir)); + fs.unlinkSync(chain.headPath(dir)); + seedCleanCallWal(dir, connectionId); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + + let renewCalls = 0; + const rawError = new Error("simulated lease loss during abandonConnection's own repair-path renew"); + const fakeWriterClaim = { + renew: () => { + renewCalls++; + if (renewCalls === 3) throw rawError; // 1: top-of-function, 2: pre-append, 3: repair-path + }, + }; + let threw = null; + try { + abandonConnection(dir, keys, connectionId, silentLog, fakeWriterClaim); + } catch (error) { + threw = error; + } + check("abandon-repair-path-lease-loss-propagates-by-reference-not-wrapped-in-fail", threw === rawError, threw && threw.message); + check("abandon-repair-path-lease-loss-has-no-gateway-recovery-error-code", !(threw && threw.code === "GATEWAY_RECOVERY_CHAIN_REPAIR_FAILED"), threw && threw.code); +} + +function abandonConnectionQuarantinesAnUnreadableWal() { + const dir = freshDir("abandon-quarantine"); + const keys = makeKeys(); + const connectionId = "conn-corrupt"; + fs.mkdirSync(recovery.activeDir(dir), { recursive: true }); + fs.mkdirSync(recovery.walPath(dir, connectionId)); + let threw = null; + try { abandonConnection(dir, keys, connectionId, silentLog); } catch (error) { threw = error; } + check("abandon-quarantine-does-not-throw", threw === null, threw && threw.message); + check("abandon-quarantine-removes-the-active-wal", recovery.listActiveConnections(dir).includes(connectionId) === false, "still listed as active"); + const quarantined = fs.existsSync(recovery.quarantineDir(dir)) ? fs.readdirSync(recovery.quarantineDir(dir)) : []; + check("abandon-quarantine-moves-it-to-the-quarantine-dir", quarantined.some((f) => f.startsWith(connectionId)), JSON.stringify(quarantined)); +} + +function notExecutedIntentReplaysAsAFailedCallAndIsThenCleanedUp() { + const dir = freshDir("recover-not-executed"); + const keys = makeKeys(); + const connectionId = "conn-not-executed"; + const intentKey = seedCleanCallWal(dir, connectionId); + // Simulates the operator having already run + // "recovery-resolve --confirmed not-executed" on this crashed, still-pending call -- + // no CALL_RESULT exists in the WAL (it never got one), but the intent now records the + // operator's decision as a persisted terminal state (see recovery.js#resolveIntentNotExecuted). + recovery.resolveIntentNotExecuted(dir, intentKey); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("not-executed-replay-does-not-need-further-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + check("not-executed-replay-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + check("not-executed-replay-cleans-up-the-intent-once-consumed", recovery.readIntent(dir, intentKey) === null, "intent still present"); + const headEntry = chain.readHead(dir); + check("not-executed-replay-still-produces-a-sealed-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +// --------------------------------------------------------------------------- +// PR #33 round-3 fix: forwardDownstreamRequestToAgent's own WAL persistence for +// downstream-initiated sampling/createMessage forwards (Codex "persist sampling calls +// in the recovery WAL"), and that recovery replay preserves the isModelCall flag and +// does not block auto-seal on an unresolved (unfenced) sampling call. +// --------------------------------------------------------------------------- + +/** Minimal fake GatewayProxy shape forwardDownstreamRequestToAgent actually reads: + * .stateDir, .now(), and .sessions (a Map already containing the one live session). */ +function fakeProxyWithSession(dir, connectionId, s) { + return { stateDir: dir, now: () => 1000, sessions: new Map([[connectionId, s]]) }; +} + +function samplingForwardSuccessIsPersistedToWalWithModelCallFlag() { + const dir = freshDir("sampling-success"); + const connectionId = "conn-sample"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.resolve({ content: [{ type: "text", text: "hi" }] }), connectionId }; + const msg = { method: "sampling/createMessage", id: 5, params: { prompt: "hi" } }; + return forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy).then((resp) => { + check("sampling-forward-returns-the-agents-result", resp.result && resp.result.content[0].text === "hi", JSON.stringify(resp)); + const events = recovery.readWalEvents(dir, connectionId); + const start = events.find((e) => e.type === "CALL_START"); + const result = events.find((e) => e.type === "CALL_RESULT"); + check("sampling-forward-wal-records-call-start-with-model-call-flag", Boolean(start) && start.isModelCall === true && start.tool === "sampling/createMessage", JSON.stringify(events)); + check("sampling-forward-wal-records-call-result-not-an-error", Boolean(result) && result.isError === false, JSON.stringify(events)); + check("sampling-forward-session-records-it-as-a-model-call", s.calls.length === 1 && s.calls[0].model_call === true && s.calls[0].isError === false, JSON.stringify(s.calls)); + }); +} + +function samplingForwardErrorIsPersistedAsFailedResult() { + const dir = freshDir("sampling-error"); + const connectionId = "conn-sample-err"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.reject(new Error("agent unreachable")), connectionId }; + const msg = { method: "sampling/createMessage", id: 6, params: { prompt: "hi" } }; + return forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy).then((resp) => { + check("sampling-forward-error-still-returns-a-jsonrpc-error", Boolean(resp.error), JSON.stringify(resp)); + const events = recovery.readWalEvents(dir, connectionId); + const result = events.find((e) => e.type === "CALL_RESULT"); + check("sampling-forward-error-wal-records-call-result-as-an-error", Boolean(result) && result.isError === true, JSON.stringify(events)); + check("sampling-forward-error-session-records-it-as-an-error", s.calls.length === 1 && s.calls[0].isError === true, JSON.stringify(s.calls)); + }); +} + +/* Codex PR #33 review "refuse sampling when its CALL_START cannot be saved": this WAL + * append happens strictly BEFORE agentPusher.current() is ever invoked -- nothing has been + * sent to the agent yet at the point it fails, so this must refuse to forward (a real, + * retryable JSON-RPC error) rather than dispatch work a subsequent crash could never + * replay (no CALL_START in the WAL at all). */ +function samplingForwardRefusesWhenCallStartCannotBeSaved() { + const dir = freshDir("sampling-wal-failure"); + const connectionId = "conn-sample-wal-fail"; + // Force recovery.appendWalEvent's own ensureDir(activeDir(...)) to fail: pre-create + // "gateway-recovery/active" as a plain FILE so mkdirSync(..., {recursive:true}) throws + // EEXIST instead of creating the directory appendWalEvent needs. + fs.mkdirSync(path.join(dir, "gateway-recovery"), { recursive: true }); + fs.writeFileSync(path.join(dir, "gateway-recovery", "active"), "not a directory"); + + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + let agentWasCalled = false; + const agentPusher = { current: () => { agentWasCalled = true; return Promise.resolve({ content: [] }); }, connectionId }; + const msg = { method: "sampling/createMessage", id: 7, params: { prompt: "hi" } }; + return forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy).then((resp) => { + check("sampling-wal-failure-does-not-forward-to-the-agent", agentWasCalled === false, "agentPusher.current was called despite the WAL append failing"); + check("sampling-wal-failure-returns-a-jsonrpc-error", Boolean(resp.error), JSON.stringify(resp)); + check("sampling-wal-failure-rolls-back-the-pending-call-entry", s.pendingCalls.size === 0, String(s.pendingCalls.size)); + // Per C1 (docs/contracts/wal-append-failure-semantics.md SS2), ANY appendWalEvent + // failure on this connection poisons it, not only a CALL_RESULT failure -- otherwise + // a second sampling request on the same connection could still attempt an append onto + // whatever the first failure may have left behind. + check("sampling-wal-failure-poisons-the-connection-too", Boolean(s.walPoisoned) && typeof s.walPoisoned.reason === "string", JSON.stringify(s.walPoisoned)); + }); +} + +// --------------------------------------------------------------------------- +// Item 5 (2026-09-15 fix-round, C1 contract): fail-closed CALL_RESULT WAL handling for +// forwardDownstreamRequestToAgent's success branch, the mirror-image fail-open error +// branch, and connection-wide WAL poisoning on any append failure on this path. See +// docs/contracts/wal-append-failure-semantics.md for the contract these tests verify. +// --------------------------------------------------------------------------- + +/** Monkeypatches recovery.appendWalEvent (the SAME module-singleton object gateway.js + * itself calls through -- both this test file and gateway.js require the identical + * "./recovery.js"/"scripts/gateway/recovery.js" module instance) so that only the ONE + * next append whose event.type === failType runs through `injectFault` instead of going + * straight to the real implementation; every other append (including a later one for the + * same connection, once the poison check should have refused it) runs unpatched. Always + * restores the original in `finally`, whether `fn` resolves, rejects, or throws + * synchronously. `injectFault` receives the REAL appendWalEvent (bound with its original + * arguments) so it can inject a raw fs-level failure around an append that actually still + * writes to the real filesystem for real (this is fault injection at the syscall layer, + * not a fake that skips the disk -- see its callers below). */ +function withOneAppendWalEventFaultForType(failType, injectFault, fn) { + const original = recovery.appendWalEvent; + let armed = true; + recovery.appendWalEvent = function (stateDir, connectionId, event) { + if (!armed || event.type !== failType) return original.apply(recovery, arguments); + armed = false; + const args = arguments; + return injectFault(() => original.apply(recovery, args)); + }; + const restore = () => { recovery.appendWalEvent = original; }; + let result; + try { + result = fn(); + } catch (error) { + restore(); + throw error; + } + if (result && typeof result.then === "function") { + return result.then( + (value) => { restore(); return value; }, + (error) => { restore(); throw error; } + ); + } + restore(); + return result; +} + +function samplingCallResultWalFailurePoisonsTheConnectionAndFailsClosed() { + const dir = freshDir("sampling-result-poison"); + const connectionId = "conn-sample-poison"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + let agentCallCount = 0; + const agentPusher = { + current: () => { agentCallCount++; return Promise.resolve({ content: [{ type: "text", text: "hi" }] }); }, + connectionId, + }; + const msg = { method: "sampling/createMessage", id: 8, params: { prompt: "hi" } }; + return withOneAppendWalEventFaultForType( + "CALL_RESULT", + () => { throw new Error("ENOSPC: simulated CALL_RESULT append failure"); }, + () => forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy) + ).then((resp) => { + check("sampling-result-wal-failure-returns-a-jsonrpc-error-not-the-real-result", Boolean(resp.error), JSON.stringify(resp)); + check("sampling-result-wal-failure-does-not-claim-safe-to-retry", !/safe to retry/i.test(resp.error.message), resp.error.message); + check( + "sampling-result-wal-failure-states-the-honest-risk-not-a-false-safety-claim", + /did not durably record/i.test(resp.error.message) && /may already have executed/i.test(resp.error.message) && /retrying/i.test(resp.error.message), + resp.error.message + ); + check("sampling-result-wal-failure-records-the-call-as-an-error", s.calls.length === 1 && s.calls[0].isError === true, JSON.stringify(s.calls)); + check( + "sampling-result-wal-failure-does-not-attach-a-fabricated-digest", + s.calls[0].result && s.calls[0].result.discarded_result_sha256 === undefined, + JSON.stringify(s.calls[0].result) + ); + const anomaly = s.anomalies.find((a) => a.kind === "GATEWAY_RECOVERY_WAL_APPEND_FAILED" && a.tool === "sampling/createMessage"); + check("sampling-result-wal-failure-reuses-the-existing-anomaly-kind-not-a-new-one", Boolean(anomaly), JSON.stringify(s.anomalies)); + check("sampling-result-wal-failure-poisons-the-connection", Boolean(s.walPoisoned) && typeof s.walPoisoned.reason === "string", JSON.stringify(s.walPoisoned)); + check("sampling-result-wal-failure-still-called-the-agent-once", agentCallCount === 1, String(agentCallCount)); + + // A second, brand-new sampling request on the SAME (now-poisoned) connection must be + // refused before it ever reaches agentPusher.current or attempts another append -- + // C1 SS3's "no best-effort second append," generalized to the whole connection. + const msg2 = { method: "sampling/createMessage", id: 9, params: { prompt: "again" } }; + return forwardDownstreamRequestToAgent(msg2, agentPusher, () => {}, proxy).then((resp2) => { + check("sampling-result-wal-poisoned-refuses-a-second-call-without-touching-the-agent", agentCallCount === 1, String(agentCallCount)); + check("sampling-result-wal-poisoned-second-call-returns-an-error", Boolean(resp2.error), JSON.stringify(resp2)); + check("sampling-result-wal-poisoned-second-call-names-the-poison", /poisoned/i.test(resp2.error.message), resp2.error.message); + check("sampling-result-wal-poisoned-does-not-grow-pending-calls", s.pendingCalls.size === 0, String(s.pendingCalls.size)); + }); + }); +} + +/* Mirror-image of the success-branch failure above (C1): the downstream already got a + * real JSON-RPC error from the agent's own rejection, so a WAL append failure recording + * THAT error result must stay fail-open (nothing left to withhold) -- but per C1 SS2 it + * still poisons the connection, and per this item's own requirement it must be attested + * via session.recordAnomaly, not merely logged. */ +function samplingCallResultErrorBranchWalFailureStaysFailOpenButIsAttestedAndPoisons() { + const dir = freshDir("sampling-error-result-wal-fail"); + const connectionId = "conn-sample-error-wal-fail"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.reject(new Error("agent unreachable")), connectionId }; + const msg = { method: "sampling/createMessage", id: 10, params: { prompt: "hi" } }; + return withOneAppendWalEventFaultForType( + "CALL_RESULT", + () => { throw new Error("EROFS: simulated CALL_RESULT append failure on the error branch"); }, + () => forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy) + ).then((resp) => { + check( + "sampling-error-branch-wal-failure-still-fails-open-and-returns-the-agents-real-error", + Boolean(resp.error) && resp.error.message === "agent unreachable", + JSON.stringify(resp) + ); + check("sampling-error-branch-wal-failure-still-records-the-error-result", s.calls.length === 1 && s.calls[0].isError === true, JSON.stringify(s.calls)); + const anomaly = s.anomalies.find((a) => a.kind === "GATEWAY_RECOVERY_WAL_APPEND_FAILED" && a.tool === "sampling/createMessage"); + check("sampling-error-branch-wal-failure-is-attested-not-just-logged", Boolean(anomaly), JSON.stringify(s.anomalies)); + check("sampling-error-branch-wal-failure-still-poisons-the-connection", Boolean(s.walPoisoned), JSON.stringify(s.walPoisoned)); + }); +} + +/* Real fault injection at the syscall layer (not a fake that skips the disk), followed by + * an actual crash-recovery/replay pass over whatever the injected failure genuinely left + * on disk -- per C1 SS1, a caught append failure does not tell the caller which of the + * three outcomes actually happened, so these three tests drive each one directly and + * confirm replay behaves correctly for each. */ + +/* C1 outcome 3, first half: "the line was written completely, and a later step failed" -- + * writeFullySync AND appendDurableLine's own file-level fsyncSync both genuinely succeed + * (every byte is written and confirmed durable), and only the subsequent fs.closeSync of + * that same fd throws. (Note: fsyncDir, recovery.js's OTHER post-write step, is + * deliberately best-effort/non-throwing by its own design -- see its doc comment -- so it + * cannot be the source of a propagating append failure; closeSync is the real one here.) + * The content is genuinely, completely, durably on disk despite the throw. */ +function samplingCallResultFailureAfterACompleteWriteIsWalAuthoritativeOnReplay() { + const dir = freshDir("sampling-result-after-complete-write"); + const connectionId = "conn-sample-after-complete-write"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.resolve({ content: [{ type: "text", text: "real-result-after-complete-write" }] }), connectionId }; + const msg = { method: "sampling/createMessage", id: 11, params: { prompt: "hi" } }; + + return withOneAppendWalEventFaultForType( + "CALL_RESULT", + (runRealAppend) => { + const originalCloseSync = fs.closeSync; + let closeCalls = 0; + fs.closeSync = function (fd) { + closeCalls++; + // The first closeSync call within this one append is appendDurableLine's own + // file-fd close, reached only after writeFullySync + fsyncSync[file] both + // already returned successfully for real. + if (closeCalls === 1) throw new Error("EIO: simulated close failure after a complete, fsynced write"); + return originalCloseSync.apply(fs, arguments); + }; + try { + return runRealAppend(); + } finally { + fs.closeSync = originalCloseSync; + } + }, + () => forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy) + ).then((resp) => { + check("sampling-after-complete-write-still-fails-closed-to-the-live-caller", Boolean(resp.error), JSON.stringify(resp)); + check("sampling-after-complete-write-poisons-the-connection", Boolean(s.walPoisoned), JSON.stringify(s.walPoisoned)); + + // Confirm the bytes actually made it to disk complete, despite the throw. + const events = recovery.readWalEvents(dir, connectionId); + const resultEvent = events.find((e) => e.type === "CALL_RESULT"); + check( + "sampling-after-complete-write-wal-line-is-genuinely-complete-on-disk", + Boolean(resultEvent) && resultEvent.isError === false, + JSON.stringify(events) + ); + + // An actual crash-recovery/replay pass over this exact on-disk state, as if this + // process had crashed right after the failed append (which the live gateway, above, + // could not distinguish from total loss -- it told the caller "error" regardless). + // Per C1, replay is WAL-authoritative: it must recover the REAL successful result the + // WAL actually holds, not re-derive the live process's own "not durably recorded" + // error. + const keys = makeKeys(); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("sampling-after-complete-write-replay-needs-no-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("sampling-after-complete-write-replay-auto-seals", Boolean(headEntry) && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + const expectedHash = expectedResultSha256({ content: [{ type: "text", text: "real-result-after-complete-write" }] }); + check( + "sampling-after-complete-write-replay-recovers-the-real-successful-result", + /"is_error":false/.test(trace) && trace.includes(expectedHash), + trace + ); + }); +} + +/* C1 outcome 3b: "fsyncSync (data) ... can throw after writeFullySync has already returned + * successfully" -- the FILE-level fsync itself is the one that fails this time, before the + * directory fsync is ever reached. The complete line is still genuinely present in the + * file (writeSync already delivered every byte); only durability-confirmation failed. */ +function samplingCallResultFailureDuringFsyncIsWalAuthoritativeOnReplay() { + const dir = freshDir("sampling-result-during-fsync"); + const connectionId = "conn-sample-during-fsync"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.resolve({ content: [{ type: "text", text: "real-result-during-fsync" }] }), connectionId }; + const msg = { method: "sampling/createMessage", id: 12, params: { prompt: "hi" } }; + + return withOneAppendWalEventFaultForType( + "CALL_RESULT", + (runRealAppend) => { + const originalFsyncSync = fs.fsyncSync; + fs.fsyncSync = function () { + // The very first fsyncSync call for this append IS the file-level one -- + // fail it immediately, before fsyncDir is ever reached. + fs.fsyncSync = originalFsyncSync; + throw new Error("EIO: simulated file-level fsync failure right after a complete write"); + }; + try { + return runRealAppend(); + } finally { + fs.fsyncSync = originalFsyncSync; + } + }, + () => forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy) + ).then((resp) => { + check("sampling-during-fsync-still-fails-closed-to-the-live-caller", Boolean(resp.error), JSON.stringify(resp)); + check("sampling-during-fsync-poisons-the-connection", Boolean(s.walPoisoned), JSON.stringify(s.walPoisoned)); + + const events = recovery.readWalEvents(dir, connectionId); + const resultEvent = events.find((e) => e.type === "CALL_RESULT"); + check( + "sampling-during-fsync-wal-line-is-genuinely-complete-on-disk", + Boolean(resultEvent) && resultEvent.isError === false, + JSON.stringify(events) + ); + + const keys = makeKeys(); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("sampling-during-fsync-replay-needs-no-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("sampling-during-fsync-replay-auto-seals", Boolean(headEntry) && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + const expectedHash = expectedResultSha256({ content: [{ type: "text", text: "real-result-during-fsync" }] }); + check( + "sampling-during-fsync-replay-recovers-the-real-successful-result", + /"is_error":false/.test(trace) && trace.includes(expectedHash), + trace + ); + }); +} + +/* C1 outcome 2: a torn/partial line. fs.writeSync makes SOME progress and then a LATER + * call in writeFullySync's own loop THROWS outright (not merely returns 0) -- unlike + * appendWalEventFailsClosedWhenWriteSyncMakesNoProgress above (a "no progress" return, + * which writeFullySync's own stall handling cleanly truncates back to the pre-append + * size), a raw throw from mid-loop bypasses that truncate-on-stall path entirely and + * leaves a genuinely torn, unparseable line physically on disk. */ +function samplingCallResultFailureAfterAPartialWriteReplaysTheOrphanedCallStartAsUnresolved() { + const dir = freshDir("sampling-result-partial-write"); + const connectionId = "conn-sample-partial-write"; + const s = session.createSession(connectionId, { now: () => 1000 }); + session.recordToolsList(s, []); + const proxy = fakeProxyWithSession(dir, connectionId, s); + const agentPusher = { current: () => Promise.resolve({ content: [{ type: "text", text: "never-durably-recorded" }] }), connectionId }; + const msg = { method: "sampling/createMessage", id: 13, params: { prompt: "hi" } }; + + return withOneAppendWalEventFaultForType( + "CALL_RESULT", + (runRealAppend) => { + const originalWriteSync = fs.writeSync; + let calls = 0; + fs.writeSync = function (fd, buffer, offset, length, position) { + calls++; + if (calls === 1) { + // Genuine partial progress: only half the requested bytes actually land. + const partial = Math.max(1, Math.floor(length / 2)); + return originalWriteSync(fd, buffer, offset, partial, position); + } + // The NEXT call throws outright -- writeFullySync's own stall-truncate path + // never runs, so the partial bytes from call 1 are left behind, torn. + throw new Error("EIO: simulated mid-write failure leaving a torn line"); + }; + try { + return runRealAppend(); + } finally { + fs.writeSync = originalWriteSync; + } + }, + () => forwardDownstreamRequestToAgent(msg, agentPusher, () => {}, proxy) + ).then((resp) => { + check("sampling-partial-write-still-fails-closed-to-the-live-caller", Boolean(resp.error), JSON.stringify(resp)); + check("sampling-partial-write-poisons-the-connection", Boolean(s.walPoisoned), JSON.stringify(s.walPoisoned)); + + // The torn line is not a parseable CALL_RESULT at all -- readWalEvents' own + // stop-at-first-bad-line contract drops it entirely. + const events = recovery.readWalEvents(dir, connectionId); + check("sampling-partial-write-torn-line-produces-no-parseable-call-result", !events.some((e) => e.type === "CALL_RESULT"), JSON.stringify(events)); + check("sampling-partial-write-call-start-before-the-tear-is-still-intact", events.some((e) => e.type === "CALL_START"), JSON.stringify(events)); + + // An actual crash-recovery/replay pass: only the CALL_START survives, so this is + // exactly the existing "orphaned sampling CALL_START" replay path -- recorded as a + // real, disconnected/unproven error and auto-sealed, never guessed as a success and + // never left stuck pending operator review (there is no intent fence for a sampling + // call to review in the first place). + const keys = makeKeys(); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("sampling-partial-write-replay-needs-no-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("sampling-partial-write-replay-auto-seals", Boolean(headEntry) && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + check( + "sampling-partial-write-replay-records-an-error-not-a-guessed-success", + /"is_error":true/.test(trace) && !/never-durably-recorded/.test(trace), + trace + ); + }); +} + +/* Not a fault-injection scenario at all: a genuinely SUCCESSFUL append, with the process + * (hypothetically) crashing in the gap between that successful append returning and + * forwardDownstreamRequestToAgent's own session.recordCallResult/logCompletion ever + * running -- there is no in-memory session left to consult, only the WAL. Per C1, replay + * must be WAL-authoritative (recover the real result from disk, not need any live state) + * and CALL_RESULT replay must be idempotent (a second recovery pass over the identical + * WAL content, e.g. this same recovery pass itself crashing before cleanup, must not + * throw, must not re-seal, and must not append a second chain entry). */ +function samplingCallResultCrashBetweenSuccessfulAppendAndInMemoryMutationReplaysIdempotently() { + const dir = freshDir("sampling-crash-between-append-and-memory"); + const keys = makeKeys(); + const connectionId = "conn-sampling-crash-gap"; + const seed = () => { + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "sampling/createMessage", server: "sampling", arguments: { prompt: "hi" }, isModelCall: true, ts: 10 }); + // This append itself succeeds for real -- "crash between a successful append and the + // in-memory mutation" means the live process got exactly this far and then died + // before session.recordCallResult/logCompletion ever ran. + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { content: [{ type: "text", text: "hi back" }] }, isError: false, ts: 11 }); + }; + seed(); + const first = recoverCrashedSessions(dir, keys, silentLog); + check("sampling-crash-gap-replay-needs-no-operator-review", first.pendingOperatorReview.length === 0, JSON.stringify(first.pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("sampling-crash-gap-replay-is-wal-authoritative-and-auto-seals", Boolean(headEntry) && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + const expectedHash = expectedResultSha256({ content: [{ type: "text", text: "hi back" }] }); + check("sampling-crash-gap-replay-recovers-the-real-successful-result", /"is_error":false/.test(trace) && trace.includes(expectedHash), trace); + check("sampling-crash-gap-replay-cleans-up-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + + // Idempotency: simulate THIS recovery pass itself crashing between sealing and cleanup + // by re-seeding the identical WAL and running recovery again (same idiom as + // recoverIsIdempotentAcrossACrashDuringRecoveryItself above) -- must not throw, must not + // append a second chain entry, and must not need operator review the second time either. + seed(); + const second = recoverCrashedSessions(dir, keys, silentLog); + check("sampling-crash-gap-second-replay-pass-is-not-flagged-for-operator-review", second.pendingOperatorReview.length === 0, JSON.stringify(second.pendingOperatorReview)); + check("sampling-crash-gap-second-replay-pass-does-not-append-a-second-chain-entry", chain.readHead(dir).seq === 1, JSON.stringify(chain.readHead(dir))); + check("sampling-crash-gap-call-result-replay-is-idempotent", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present after second pass"); +} + +function recoverPreservesModelCallFlagOnSamplingReplay() { + const dir = freshDir("recover-sampling-replay"); + const keys = makeKeys(); + const connectionId = "conn-sampling-replay"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "sampling/createMessage", server: "sampling", arguments: { prompt: "hi" }, isModelCall: true, ts: 10 }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_RESULT", call_seq: 1, result: { content: [{ type: "text", text: "hi back" }] }, isError: false, ts: 11 }); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-sampling-replay-no-operator-review-needed", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("recover-sampling-replay-produces-a-sealed-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + check("recover-sampling-replay-preserves-model-call-flag-not-hardcoded-false", /"model_call":true/.test(trace), trace); +} + +function recoverDoesNotBlockAutoSealOnAnUnresolvedSamplingCall() { + const dir = freshDir("recover-sampling-unresolved"); + const keys = makeKeys(); + const connectionId = "conn-sampling-crash"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + // No CALL_RESULT: the gateway crashed while the agent was still computing its model + // response. There is no intent/fence for a sampling call (unlike a fenced tools/call), + // so this must NOT be treated the same as an unproven fenced call -- it should auto-seal + // as a real, disconnected/failed result rather than block on operator review forever. + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "sampling/createMessage", server: "sampling", arguments: { prompt: "hi" }, isModelCall: true, ts: 10 }); + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("recover-unresolved-sampling-call-does-not-need-operator-review", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const headEntry = chain.readHead(dir); + check("recover-unresolved-sampling-call-still-auto-seals", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const trace = bundle.contents["execution_trace.jsonl"]; + check("recover-unresolved-sampling-call-recorded-as-an-error-not-a-guessed-success", /"is_error":true/.test(trace), trace); +} + +// --------------------------------------------------------------------------- +// gateway.js#abandonConnection -- explicit operator override +// --------------------------------------------------------------------------- + +function abandonConnectionSealsAndReleasesTheFenceForAnUnprovenCall() { + const dir = freshDir("abandon"); + const keys = makeKeys(); + const connectionId = "conn-abandon"; + const intentKey = seedCleanCallWal(dir, connectionId); + // Confirm this connection would otherwise be stuck pending operator review. + const before = recoverCrashedSessions(dir, keys, silentLog); + check("abandon-precondition-connection-was-pending-operator-review", before.pendingOperatorReview.includes(connectionId), JSON.stringify(before)); + + abandonConnection(dir, keys, connectionId, silentLog); + check("abandon-deletes-the-wal", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); + check("abandon-releases-the-intent-fence", recovery.readIntent(dir, intentKey) === null, "intent still present"); + const headEntry = chain.readHead(dir); + check("abandon-still-produces-a-sealed-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); +} + +function abandonConnectionOnAlreadyCleanConnectionIsANoOp() { + const dir = freshDir("abandon-noop"); + const keys = makeKeys(); + let threw = null; + try { abandonConnection(dir, keys, "never-existed", silentLog); } catch (error) { threw = error; } + check("abandon-of-a-connection-with-no-wal-does-not-throw", threw === null, threw && threw.message); +} + +/** Round-N fix (frontier-panel finding #5, Paul's explicit choice of Fable's "make the gap + * observable" proposal): mirrors proxy.js's own cached-replay branch -- the CALL_START for + * a completed-signature cache hit IS durably written with replayed:true, but its matching + * CALL_RESULT never makes it into the WAL before the crash. The intent stays "dispatched" + * (never proven), so this is exactly the orphaned-replayed-step gap finding #5 named. */ +function seedOrphanedReplayedCallWal(dir, connectionId) { + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 7, tool: "echo", server: "srv", arguments: { a: 99 }, ts: 10, replayed: true }); + const intentKey = recovery.computeIntentKey(connectionId, "echo", { a: 99 }); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: connectionId, tool: "echo", arguments: { a: 99 }, state: "dispatched", dispatched_at: 9 }); + return intentKey; +} + +/** Proves finding #5 Part B end-to-end, not just "no regression": recoverCrashedSessions' + * own per-pending-call loop performs the identical startedFrom.replayed check but can + * never seal a needsOperator connection (see that function's own doc comment) -- an + * earlier version of this fix added the anomaly there, which was confirmed dead code + * (never reaches a sealed bundle). abandonConnection is the actual sealing path, so this + * test constructs the orphaned-replayed-step WAL, confirms recoverCrashedSessions flags it + * for operator review WITHOUT sealing anything, then confirms recovery-abandon actually + * produces a sealed bundle whose decision_record.md contains the explicit "INCOMPLETE: + * step 7" marker Paul asked for. */ +function abandonConnectionRecordsIncompleteMarkerForOrphanedReplayedStep() { + const dir = freshDir("abandon-replayed-incomplete"); + const keys = makeKeys(); + const connectionId = "conn-replayed-incomplete"; + const intentKey = seedOrphanedReplayedCallWal(dir, connectionId); + + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, silentLog); + check("replayed-incomplete-precondition-flagged-for-operator-review", pendingOperatorReview.includes(connectionId), JSON.stringify(pendingOperatorReview)); + check("replayed-incomplete-precondition-nothing-sealed-yet", chain.readHead(dir) === null, JSON.stringify(chain.readHead(dir))); + check("replayed-incomplete-precondition-intent-still-dispatched", recovery.readIntent(dir, intentKey).state === "dispatched", JSON.stringify(recovery.readIntent(dir, intentKey))); + + abandonConnection(dir, keys, connectionId, silentLog); + + const headEntry = chain.readHead(dir); + check("replayed-incomplete-abandon-produces-a-sealed-chain-entry", headEntry && headEntry.seq === 1, JSON.stringify(headEntry)); + const bundle = JSON.parse(fs.readFileSync(chain.bundlePath(dir, headEntry.bundle_id), "utf8")); + const decisionRecord = bundle.contents["decision_record.md"]; + check( + "replayed-incomplete-sealed-bundle-contains-the-incomplete-marker", + /GATEWAY_RECOVERY_INCOMPLETE_REPLAYED_STEP/.test(decisionRecord) && /INCOMPLETE: step 7 was a cached replay/.test(decisionRecord), + decisionRecord + ); + check("replayed-incomplete-generic-abandon-anomaly-also-present", /OPERATOR_ABANDONED_RECOVERY/.test(decisionRecord), decisionRecord); + check("replayed-incomplete-fence-still-released", recovery.readIntent(dir, intentKey) === null, "intent still present"); + check("replayed-incomplete-wal-still-cleaned-up", recovery.readWalEvents(dir, connectionId).length === 0, "WAL still present"); +} + +// --------------------------------------------------------------------------- +// PR #33 round-2 fixes that need a real WriterClaim / real startGateway plumbing. +// --------------------------------------------------------------------------- + + +/* Codex PR #33 review "write the entire intent record before dispatching": state-store.js's + * atomicCreateExclusive/atomicOverwriteFile each made exactly ONE fs.writeSync call and + * then fsync'd + linked/renamed the result into place. fs.writeSync is permitted to write + * FEWER bytes than requested, so a short write PUBLISHED a truncated file while returning + * success -- for createIntentIfAbsent that means proxy.js dispatches a real downstream side + * effect against an intent fence that is invalid JSON on disk. */ +function atomicWritesLoopUntilTheWholePayloadIsWritten() { + const dir = freshDir("short-write"); + const intentKey = recovery.computeIntentKey("conn-short-write", "email", { to: "a@b.c" }); + + // Force a pathological short write: one byte per fs.writeSync call, exactly what Node's + // own docs permit and what a real pipe/quota-pressured filesystem actually does. + const realWriteSync = fs.writeSync; + /* One byte per call, for BOTH fs.writeSync call shapes this codebase uses: the + * (fd, buffer, offset, length) form recovery.js#writeFullySync already loops on, and the + * (fd, string) form state-store.js's atomic primitives use -- a stub that only shortened + * the former would silently stop exercising the fix the moment the code under test + * passed a string, which is exactly what it does. */ + const shortWriteSync = function (fd, buffer, offset, length, ...rest) { + if (Buffer.isBuffer(buffer) && typeof offset === "number" && typeof length === "number" && length > 1) { + return realWriteSync.call(fs, fd, buffer, offset, 1); + } + if (offset === undefined && length === undefined && rest.length === 0) { + const encoded = Buffer.isBuffer(buffer) ? buffer : Buffer.from(String(buffer), "utf8"); + if (encoded.length > 1) return realWriteSync.call(fs, fd, encoded, 0, 1); + } + return realWriteSync.apply(fs, [fd, buffer, offset, length, ...rest]); + }; + fs.writeSync = shortWriteSync; + let created = null; + let thrown = null; + try { + created = recovery.createIntentIfAbsent(dir, intentKey, { + connection_id: "conn-short-write", + tool: "email", + arguments: { to: "a@b.c" }, + state: "dispatched", + dispatched_at: 1, + idempotency_key: "short-write-key", + generation: 1, + }); + } catch (error) { + thrown = error; + } finally { + fs.writeSync = realWriteSync; + } + check("short-write-intent-create-did-not-throw", thrown === null, thrown && thrown.message); + check("short-write-intent-create-returned-a-record", Boolean(created) && created.intent_key === intentKey, JSON.stringify(created)); + + // The decisive assertion: what is actually ON DISK must be the complete record, not a + // truncated prefix that only LOOKS like a successful create to the caller. + const rawOnDisk = fs.readFileSync(recovery.intentPath(dir, intentKey), "utf8"); + let parsed = null; + let parseError = null; + try { parsed = JSON.parse(rawOnDisk); } catch (error) { parseError = error; } + check( + "short-write-intent-is-complete-valid-json-on-disk", + parseError === null && parsed && parsed.intent_key === intentKey && parsed.state === "dispatched" && parsed.idempotency_key === "short-write-key", + `parseError=${parseError && parseError.message}; raw=${JSON.stringify(rawOnDisk)}` + ); + check( + "short-write-intent-round-trips-through-readIntent", + (() => { try { const r = recovery.readIntent(dir, intentKey); return Boolean(r) && r.arguments && r.arguments.to === "a@b.c"; } catch (error) { return false; } })(), + rawOnDisk + ); + + // Same guarantee for the overwrite primitive (updateIntent / recordCompletedSignature). + fs.writeSync = shortWriteSync; + try { + recovery.updateIntent(dir, intentKey, { state: "completed", completed_at: 2, cached_result: { ok: true, padding: "x".repeat(200) } }); + } finally { + fs.writeSync = realWriteSync; + } + const rawAfterUpdate = fs.readFileSync(recovery.intentPath(dir, intentKey), "utf8"); + let updated = null; + try { updated = JSON.parse(rawAfterUpdate); } catch (error) { updated = null; } + check( + "short-write-intent-overwrite-is-complete-valid-json-on-disk", + Boolean(updated) && updated.state === "completed" && updated.cached_result && updated.cached_result.padding.length === 200, + JSON.stringify(rawAfterUpdate).slice(0, 300) + ); +} + +/* Codex PR #33 review "fsync both directories after quarantining a WAL": recovery-abandon's + * active/ -> quarantine/ rename is the durable save point it reports success on, but + * rename(2) only mutates two DIRECTORY ENTRIES -- neither is durable until its own parent + * directory's fd is fsynced. */ +function quarantineWalFsyncsBothSourceAndDestinationDirectories() { + const dir = freshDir("quarantine-fsync"); + const connectionId = "conn-quarantine-fsync"; + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", session_id: "sid-1", tools: [] }); + + // Record which PATHS actually get an fsync during quarantineWal, by correlating each + // fsync'd fd back to the path it was opened from. + const fdToPath = new Map(); + const realOpenSync = fs.openSync; + const realFsyncSync = fs.fsyncSync; + const fsyncedPaths = []; + fs.openSync = function (p, ...rest) { + const fd = realOpenSync.call(fs, p, ...rest); + fdToPath.set(fd, String(p)); + return fd; + }; + fs.fsyncSync = function (fd) { + if (fdToPath.has(fd)) fsyncedPaths.push(fdToPath.get(fd)); + return realFsyncSync.call(fs, fd); + }; + let to = null; + try { + to = recovery.quarantineWal(dir, connectionId); + } finally { + fs.openSync = realOpenSync; + fs.fsyncSync = realFsyncSync; + } + check("quarantine-wal-moved-the-file", Boolean(to) && fs.existsSync(to) && !fs.existsSync(recovery.walPath(dir, connectionId)), String(to)); + check( + "quarantine-wal-fsyncs-the-source-active-directory", + fsyncedPaths.includes(recovery.activeDir(dir)), + JSON.stringify(fsyncedPaths) + ); + check( + "quarantine-wal-fsyncs-the-destination-quarantine-directory", + fsyncedPaths.includes(recovery.quarantineDir(dir)), + JSON.stringify(fsyncedPaths) + ); +} + +/* Codex PR #33 review "sweep expired completed-signature records" + CodeRabbit PR #33 + * review "sweep expired completed signatures" (the same finding from both reviewers): the + * advertised 24-hour retention was only ever enforced per-key on read, so a signature whose + * exact (tool, arguments, idempotency key) is never presented again was never deleted by + * anything and accumulated without bound. */ +function sweepExpiredSignaturesRemovesOnlyExpiredRecordsAndIsBounded() { + const dir = freshDir("signature-sweep"); + const ttl = recovery.DEFAULT_RETAINED_SIGNATURE_TTL_MS; + const now = Date.now(); + + const expiredKeys = []; + for (let i = 0; i < 4; i++) { + const key = recovery.computeSignatureKey("email", { n: i }, `expired-${i}`); + expiredKeys.push(key); + recovery.recordCompletedSignature(dir, key, { completed_at: now - ttl - 60000, idempotency_key: `expired-${i}`, cached_result: { ok: true } }); + } + const liveKey = recovery.computeSignatureKey("email", { n: "live" }, "live-key"); + recovery.recordCompletedSignature(dir, liveKey, { completed_at: now - 1000, idempotency_key: "live-key", cached_result: { ok: true } }); + // A record with no usable completed_at: readCompletedSignature already treats this as + // expired, so the sweep must agree rather than keep it forever. + const malformedKey = recovery.computeSignatureKey("email", { n: "malformed" }, "malformed-key"); + recovery.recordCompletedSignature(dir, malformedKey, { idempotency_key: "malformed-key", cached_result: { ok: true } }); + + const result = recovery.sweepExpiredSignatures(dir); + check( + "sweep-removes-every-expired-signature-record", + expiredKeys.every((k) => !fs.existsSync(recovery.signaturePath(dir, k))), + JSON.stringify(result) + ); + check("sweep-keeps-a-still-live-signature-record", fs.existsSync(recovery.signaturePath(dir, liveKey)), JSON.stringify(result)); + check("sweep-removes-a-record-with-no-usable-completed-at", !fs.existsSync(recovery.signaturePath(dir, malformedKey)), JSON.stringify(result)); + check("sweep-reports-what-it-reclaimed", result.removed === 5 && result.scanned === 6, JSON.stringify(result)); + check( + "sweep-leaves-a-still-live-record-readable-through-the-normal-path", + (() => { const r = recovery.readCompletedSignature(dir, liveKey); return Boolean(r) && r.idempotency_key === "live-key"; })(), + "live record unreadable after the sweep" + ); + + // Bounded: one sweep never examines more than maxEntries directory entries, so a huge + // signatures/ directory cannot make a single startup arbitrarily long. + const bounded = recovery.sweepExpiredSignatures(dir, { maxEntries: 1 }); + check("sweep-honors-its-maxEntries-bound", bounded.scanned <= 1, JSON.stringify(bounded)); +} + +/* The sweep above is only useful if something actually CALLS it: startup recovery is the + * bounded, once-per-process pass both reviewers named as the right place. */ +function startupRecoverySweepsExpiredSignatureRecords() { + const dir = freshDir("startup-signature-sweep"); + const ttl = recovery.DEFAULT_RETAINED_SIGNATURE_TTL_MS; + const expiredKey = recovery.computeSignatureKey("email", { n: 1 }, "startup-expired"); + recovery.recordCompletedSignature(dir, expiredKey, { completed_at: Date.now() - ttl - 60000, idempotency_key: "startup-expired", cached_result: { ok: true } }); + const liveKey = recovery.computeSignatureKey("email", { n: 2 }, "startup-live"); + recovery.recordCompletedSignature(dir, liveKey, { completed_at: Date.now(), idempotency_key: "startup-live", cached_result: { ok: true } }); + + recoverCrashedSessions(dir, makeKeys(), silentLog); + check("startup-recovery-sweeps-an-expired-signature-record", !fs.existsSync(recovery.signaturePath(dir, expiredKey)), "expired record survived startup recovery"); + check("startup-recovery-keeps-a-live-signature-record", fs.existsSync(recovery.signaturePath(dir, liveKey)), "live record was wrongly swept"); +} + +function writerClaimIsReleasedWhenStartupRecoveryThrows() { + const root = freshDir("release-on-recovery-failure"); + const stateDir = path.join(root, "state"); + fs.mkdirSync(path.join(root, ".graphsmith", "state"), { recursive: true }); + const { writeConfirmedMode } = require(path.join(ROOT, "tests", "gateway", "_fixtures", "mode-file.js")); + writeConfirmedMode(root, "standalone"); + const kp = crypto.generateKeyPairSync("ed25519"); + const keyPath = path.join(root, "signing-key.pem"); + fs.writeFileSync(keyPath, kp.privateKey.export({ type: "pkcs8", format: "pem" }).toString()); + const configPath = path.join(root, "gateway-config.json"); + fs.writeFileSync(configPath, JSON.stringify({ + schema_version: "1.0", + state_dir: stateDir, + // Never actually reached -- recovery throws before connectAllDownstreams runs. + downstream_servers: [{ name: "unused", transport: "stdio", endpoint: "node -e process.exit(1)" }], + signing_key_ref: keyPath, + })); + // Make gateway-recovery/active a FILE, not a directory: recovery.listActiveConnections' + // own fs.readdirSync then fails with ENOTDIR, a genuine non-ENOENT error this function + // does not (and should not) swallow -- an unlistable recovery directory is a real, + // whole-pass failure distinct from any single connection's own unreadable state. + fs.mkdirSync(recovery.recoveryDir(stateDir), { recursive: true }); + fs.writeFileSync(recovery.activeDir(stateDir), "not a directory"); + + let rejection = null; + return startGateway({ configPath, root, log: () => {} }).then( + () => { rejection = null; }, + (error) => { rejection = error; } + ).then(() => { + check("recovery-listActiveConnections-failure-rejects-startGateway", rejection !== null, "startGateway resolved instead of rejecting"); + // The writer-claim file must be gone -- proof that writerClaim.release() actually ran + // rather than leaking a claim an immediate restart would then be refused for. + const { WriterClaim: WC } = require(path.join(ROOT, "scripts", "writer-claim.js")); + const fresh = new WC(stateDir, { hostId: "test-second-instance" }); + let acquireError = null; + try { fresh.acquire(); fresh.release(); } catch (error) { acquireError = error; } + check("writer-claim-released-so-a-fresh-instance-can-immediately-acquire-it", acquireError === null, acquireError && acquireError.message); + }); +} + +/* Round-1 fix-plan commit 4, Paul's 2026-09-15 startup-posture decision: a genuine + * chain-integrity failure found by chain.reconcileHead at startup must hard-fail -- + * refuse to start, write a diagnostic artifact naming the manual-recovery runbook, and + * release the writer-claim lease -- rather than starting in some degraded/silent state. + * Builds a real state_dir with two genuinely sealed sessions (via recoverCrashedSessions, + * the real public API -- not by disabling the safety check), then corrupts HEAD.json into + * a genuine fork the same way the bundle-collision regression test above does, so + * startGateway's own reconcileHead call hits exactly the "worse than single-step lag" + * case this whole commit exists to hard-fail on. */ +/* Round-1 fix-plan commit 6: if an earlier connection's own repair path latches a + * genuine chain-integrity failure partway through this loop, every remaining + * connection is guaranteed to hit the identical failure -- this function must abort + * immediately rather than spend more fsyncs parsing/replaying WALs guaranteed to be + * quarantined anyway. Builds the latch through the real public detector + * (chain.reconcileHead on a genuinely forked chain), exactly as commit 4's own + * startup-hard-fail test above does, rather than poking chain.js's private state. */ +function recoverCrashedSessionsAbortsImmediatelyWhenAlreadyLatched() { + const dir = freshDir("recover-already-latched"); + const keys = makeKeys(); + + const connA = "conn-already-latched-a"; + seedCleanCallWal(dir, connA); + recovery.appendWalEvent(dir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 1 }); + recoverCrashedSessions(dir, keys, silentLog); + const entryA = chain.readHead(dir); + + // Fork HEAD against the real, already-sealed tail -- the exact "worse than + // single-step lag" case commit 4's own reconcileHead refuses and latches on. + const forkedHead = Object.assign({}, entryA, { entry_sha256: "9".repeat(64) }); + fs.writeFileSync(chain.headPath(dir), JSON.stringify(forkedHead)); + chain._resetChainIntegrityFailureForTests(); + try { + chain.reconcileHead(dir); + } catch (error) { + // expected: this is exactly how a real process would come to have this latched. + } + check("recover-abort-fixture-is-latched", Boolean(chain.getChainIntegrityFailure()), JSON.stringify(chain.getChainIntegrityFailure())); + + // Two brand-new crashed connections, each with a fully clean, resolvable WAL -- if + // recoverCrashedSessions did not abort early, both would ordinarily seal without issue. + const connB = "conn-already-latched-b"; + seedCleanCallWal(dir, connB); + recovery.appendWalEvent(dir, connB, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 2 }); + const connC = "conn-already-latched-c"; + seedCleanCallWal(dir, connC); + recovery.appendWalEvent(dir, connC, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 3 }); + + const logs = []; + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, (l) => logs.push(l)); + check("recover-abort-processes-nothing", pendingOperatorReview.length === 0, JSON.stringify(pendingOperatorReview)); + const stillActive = recovery.listActiveConnections(dir); + check("recover-abort-leaves-conn-b-wal-in-place", stillActive.includes(connB), JSON.stringify(stillActive)); + check("recover-abort-leaves-conn-c-wal-in-place", stillActive.includes(connC), JSON.stringify(stillActive)); + check("recover-abort-logs-a-recovery-aborted-message", logs.some((l) => /RECOVERY ABORTED/.test(l)), JSON.stringify(logs)); + chain._resetChainIntegrityFailureForTests(); +} + +/* Round-1 fix-plan commit 6: documented explicitly, in both code comments and the + * WRITTEN status output, that the periodic status walk never latches (docs/contracts/ + * chain-validity.md SS5/SS6) and that its cadence is not what stops admission on any + * finding -- see writeStatusFile's own doc comment for the full rationale. */ +function writeStatusFileDocumentsTheCadenceLimitationInTheWrittenOutput() { + const dir = freshDir("status-cadence-note"); + fs.mkdirSync(dir, { recursive: true }); + const ctx = { config: { state_dir: dir }, writerClaim: { status: () => ({}) }, connections: new Map(), proxy: { openSessionCount: () => 0 } }; + writeStatusFile(ctx, silentLog); + const written = JSON.parse(fs.readFileSync(path.join(dir, "gateway-status.json"), "utf8")); + check( + "status-file-documents-the-cadence-mid-chain-limitation", + typeof written.session_chain_integrity_cadence_note === "string" && + /mid-chain/i.test(written.session_chain_integrity_cadence_note) && + /HEAD\/tail/i.test(written.session_chain_integrity_cadence_note) && + /evidence only/i.test(written.session_chain_integrity_cadence_note), + JSON.stringify(written.session_chain_integrity_cadence_note) + ); +} + +/* Round-1 fix-plan commit 6: a genuine content-level structural finding (a tampered + * interior entry) that the periodic status walk's own evidence surfaces must NOT, by + * itself, latch admission -- per docs/contracts/chain-validity.md SS5, this walk is + * evidence-only and mixing its own hand-rolled classification into the admission-latch + * path would violate the "one authoritative structural validator" contract. Real + * on-disk chain/bundle files (not chain.validateChain's in-memory fixtures) so this + * exercises the exact path buildHealthStatus/writeStatusFile actually runs. */ +function writeStatusFileNeverLatchesEvenOnAGenuineTamperedEntryItFinds() { + const dir = freshDir("status-never-latches-tampered"); + const e1 = { schema_version: "1.0", seq: 1, bundle_id: "gsa-status-tampered-0000001", prev_entry_sha256: null }; + e1.entry_sha256 = chain.computeEntrySha256(e1); + const e2 = { schema_version: "1.0", seq: 2, bundle_id: "gsa-status-tampered-0000002", prev_entry_sha256: e1.entry_sha256 }; + e2.entry_sha256 = chain.computeEntrySha256(e2); + const tamperedE2 = Object.assign({}, e2, { entry_sha256: "7".repeat(64) }); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + fs.writeFileSync(chain.chainPath(dir), JSON.stringify(e1) + "\n" + JSON.stringify(tamperedE2) + "\n"); + fs.writeFileSync(chain.headPath(dir), JSON.stringify({ schema_version: "1.0", seq: 2, bundle_id: tamperedE2.bundle_id, entry_sha256: tamperedE2.entry_sha256 })); + fs.writeFileSync(chain.bundlePath(dir, e1.bundle_id), "{}"); + fs.writeFileSync(chain.bundlePath(dir, tamperedE2.bundle_id), "{}"); + + chain._resetChainIntegrityFailureForTests(); + const ctx = { config: { state_dir: dir }, writerClaim: { status: () => ({}) }, connections: new Map(), proxy: { openSessionCount: () => 0 } }; + const health = buildHealthStatus(ctx); + check( + "status-walk-fixture-actually-finds-the-tampered-entry", + health.session_chain_integrity.status === "failed" && /TAMPERED/.test(health.session_chain_integrity.evidence.join(" ")), + JSON.stringify(health.session_chain_integrity) + ); + writeStatusFile(ctx, silentLog); + check("status-walk-does-not-latch-even-on-a-genuine-finding", chain.getChainIntegrityFailure() === null, JSON.stringify(chain.getChainIntegrityFailure())); +} + +function startupHardFailsWritesDiagnosticAndReleasesTheClaimOnAGenuineChainFork() { + const root = freshDir("startup-chain-fork"); + const stateDir = path.join(root, "state"); + fs.mkdirSync(path.join(root, ".graphsmith", "state"), { recursive: true }); + const { writeConfirmedMode } = require(path.join(ROOT, "tests", "gateway", "_fixtures", "mode-file.js")); + writeConfirmedMode(root, "standalone"); + const kp = crypto.generateKeyPairSync("ed25519"); + const keyPath = path.join(root, "signing-key.pem"); + fs.writeFileSync(keyPath, kp.privateKey.export({ type: "pkcs8", format: "pem" }).toString()); + const configPath = path.join(root, "gateway-config.json"); + fs.writeFileSync(configPath, JSON.stringify({ + schema_version: "1.0", + state_dir: stateDir, + // Never actually reached -- reconcileHead throws before connectAllDownstreams runs. + downstream_servers: [{ name: "unused", transport: "stdio", endpoint: "node -e process.exit(1)" }], + signing_key_ref: keyPath, + })); + + const keys = makeKeys(); + const connA = "conn-startup-fork-a"; + seedCleanCallWal(stateDir, connA); + recovery.appendWalEvent(stateDir, connA, { type: "CALL_RESULT", call_seq: 1, result: { ok: true }, isError: false, ts: 11 }); + recoverCrashedSessions(stateDir, keys, silentLog); + const entryA = chain.readHead(stateDir); + + const connB = "conn-startup-fork-b"; + seedTwoCallWal(stateDir, connB); + recovery.appendWalEvent(stateDir, connB, { type: "CALL_RESULT", call_seq: 2, result: { ok: true }, isError: false, ts: 13 }); + recoverCrashedSessions(stateDir, keys, silentLog); + + const forkedHead = Object.assign({}, entryA, { entry_sha256: "e".repeat(64) }); + fs.writeFileSync(chain.headPath(stateDir), JSON.stringify(forkedHead)); + + let rejection = null; + return startGateway({ configPath, root, log: () => {} }).then( + () => { rejection = null; }, + (error) => { rejection = error; } + ).then(() => { + check( + "startup-hard-fail-rejects-with-the-structural-failure-code", + rejection !== null && rejection.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", + rejection ? `${rejection.code}: ${rejection.message}` : "startGateway resolved instead of rejecting" + ); + + const diagnosticPath = path.join(stateDir, "gateway-startup-failure.json"); + let diagnostic = null; + let readError = null; + try { + diagnostic = JSON.parse(fs.readFileSync(diagnosticPath, "utf8")); + } catch (error) { + readError = error; + } + check("startup-hard-fail-writes-a-diagnostic-artifact", diagnostic !== null, readError && readError.message); + check( + "startup-hard-fail-diagnostic-names-the-structural-failure-code", + Boolean(diagnostic) && diagnostic.code === "GATEWAY_CHAIN_STRUCTURAL_FAILURE", + JSON.stringify(diagnostic) + ); + check( + "startup-hard-fail-diagnostic-references-the-manual-recovery-runbook", + Boolean(diagnostic) && typeof diagnostic.runbook === "string" && diagnostic.runbook.includes("docs/runbooks/gateway-chain-corruption-recovery.md") && String(diagnostic.action_required || "").includes(diagnostic.runbook), + JSON.stringify(diagnostic) + ); + + // The writer-claim file must be gone -- proof that writerClaim.release() actually ran + // rather than leaking a claim an immediate restart would then be refused for. + const { WriterClaim: WC } = require(path.join(ROOT, "scripts", "writer-claim.js")); + const fresh = new WC(stateDir, { hostId: "test-second-instance-chain-fork" }); + let acquireError = null; + try { fresh.acquire(); fresh.release(); } catch (error) { acquireError = error; } + check("startup-hard-fail-releases-the-writer-claim", acquireError === null, acquireError && acquireError.message); + }); +} + +/* Codex PR #33 review "report each unresolved intent key in recovery output": a bare + * `in_flight` COUNT cannot distinguish an ordinary live in-progress call from a crashed + * connection's own unresolved one left in the same "dispatched" state -- an operator needs + * the actual keys (recovery-resolve/recovery-abandon both require the exact key) to act. */ +function healthStatusItemizesDispatchedIntentsAlongsideTheCount() { + const dir = freshDir("health-dispatched"); + const intentKeyA = recovery.computeIntentKey("conn-a", "toolA", { x: 1 }); + recovery.createIntentIfAbsent(dir, intentKeyA, { connection_id: "conn-a", tool: "toolA", arguments: { x: 1 }, state: "dispatched", dispatched_at: 123 }); + const intentKeyB = recovery.computeIntentKey("conn-b", "toolB", {}); + recovery.createIntentIfAbsent(dir, intentKeyB, { connection_id: "conn-b", tool: "toolB", arguments: {}, state: "ambiguous", ambiguous_at: 456, ambiguous_reason: "test" }); + + const ctx = { + config: { state_dir: dir }, + writerClaim: { status: () => ({}) }, + connections: new Map(), + proxy: { openSessionCount: () => 0 }, + }; + const status = buildHealthStatus(ctx); + check("health-in-flight-count-still-reported", status.recovery.in_flight === 1, JSON.stringify(status.recovery)); + check( + "health-dispatched-list-itemizes-the-actual-intent", + Array.isArray(status.recovery.dispatched) && status.recovery.dispatched.length === 1 && status.recovery.dispatched[0].intent_key === intentKeyA && status.recovery.dispatched[0].connection_id === "conn-a" && status.recovery.dispatched[0].tool === "toolA", + JSON.stringify(status.recovery) + ); + check( + "health-pending-operator-review-unaffected-by-this-change", + status.recovery.pending_operator_review.length === 1 && status.recovery.pending_operator_review[0].intent_key === intentKeyB, + JSON.stringify(status.recovery) + ); +} + +/* Codex PR #33 review "include the intent key in the advertised resolution command": the + * RECOVERY_AMBIGUOUS_INTENT log used to print a literal "" placeholder in its example + * remediation command regardless of which (or how many) calls were actually unresolved -- + * unusable without first hand-parsing raw recovery files to find the real key. */ +function recoverAmbiguousIntentLogNamesTheRealKeyNotAPlaceholder() { + const dir = freshDir("recover-log-real-key"); + const keys = makeKeys(); + const connectionId = "conn-unresolved"; + // A crashed call with NO intent record at all (never even reached createIntentIfAbsent, + // or its file was lost) -- recoverCrashedSessions must still flag it for operator review + // (the existing "needsOperator" fallback) and now names its real, computable key. + recovery.appendWalEvent(dir, connectionId, { type: "SESSION_START", started_at: 1, goal: null, tools: [{ name: "echo", server: "srv", schema: {} }] }); + recovery.appendWalEvent(dir, connectionId, { type: "INITIALIZE", clientInfo: { name: "agent", version: "1" }, serverInfo: { name: "srv", version: "1" } }); + recovery.appendWalEvent(dir, connectionId, { type: "CALL_START", call_seq: 1, tool: "echo", server: "srv", arguments: { a: 1 }, ts: 10 }); + const expectedIntentKey = recovery.computeIntentKey(connectionId, "echo", { a: 1 }); + + const logLines = []; + const { pendingOperatorReview } = recoverCrashedSessions(dir, keys, (line) => logLines.push(line)); + check("recover-log-real-key-flags-for-operator-review", pendingOperatorReview.includes(connectionId), JSON.stringify(pendingOperatorReview)); + const ambiguousLog = logLines.find((l) => l.includes("RECOVERY_AMBIGUOUS_INTENT")); + check("recover-log-real-key-line-exists", Boolean(ambiguousLog), JSON.stringify(logLines)); + check("recover-log-real-key-names-the-actual-computed-key", Boolean(ambiguousLog) && ambiguousLog.includes(expectedIntentKey), ambiguousLog || "no log line"); + check("recover-log-real-key-does-not-print-the-old-placeholder", Boolean(ambiguousLog) && !ambiguousLog.includes("--intent "), ambiguousLog || "no log line"); +} + +function recoveryResolveRequiresResultFileForExecuted() { + const dir = freshDir("cli-result-file"); + const root = freshDir("cli-result-file-root"); + const kp = crypto.generateKeyPairSync("ed25519"); + const keyPath = path.join(root, "signing-key.pem"); + fs.writeFileSync(keyPath, kp.privateKey.export({ type: "pkcs8", format: "pem" }).toString()); + const configPath = path.join(root, "gateway-config.json"); + fs.writeFileSync(configPath, JSON.stringify({ + schema_version: "1.0", + state_dir: dir, + downstream_servers: [{ name: "unused", transport: "stdio", endpoint: "node -e process.exit(1)" }], + signing_key_ref: keyPath, + })); + const intentKey = recovery.computeIntentKey("conn-cli", "toolX", { x: 1 }); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-cli", tool: "toolX", arguments: { x: 1 }, state: "ambiguous" }); + + let threw = null; + try { + runRecoveryResolveCli(["--connection", "conn-cli", "--intent", intentKey, "--confirmed", "executed", "--config", configPath], () => {}); + } catch (error) { + threw = error; + } + check("recovery-resolve-executed-without-result-file-is-rejected", threw !== null && threw.code === "GATEWAY_RECOVERY_CLI_USAGE", threw ? threw.message : "did not throw"); + check("recovery-resolve-rejected-attempt-does-not-mutate-the-intent", recovery.readIntent(dir, intentKey).state === "ambiguous", JSON.stringify(recovery.readIntent(dir, intentKey))); + + // Providing --result-file must still work. + const resultFile = path.join(root, "result.json"); + fs.writeFileSync(resultFile, JSON.stringify({ charged: true })); + runRecoveryResolveCli(["--connection", "conn-cli", "--intent", intentKey, "--confirmed", "executed", "--result-file", resultFile, "--config", configPath], () => {}); + const resolved = recovery.readIntent(dir, intentKey); + check("recovery-resolve-executed-with-result-file-succeeds", resolved.state === "completed" && resolved.cached_result.charged === true, JSON.stringify(resolved)); + + // The writer-claim taken internally by runRecoveryResolveCli must be released + // afterward -- a second call against a different intent must not be refused. + const intentKey2 = recovery.computeIntentKey("conn-cli", "toolY", {}); + recovery.createIntentIfAbsent(dir, intentKey2, { connection_id: "conn-cli", tool: "toolY", arguments: {}, state: "ambiguous" }); + let secondThrew = null; + try { + runRecoveryResolveCli(["--connection", "conn-cli", "--intent", intentKey2, "--confirmed", "not-executed", "--config", configPath], () => {}); + } catch (error) { + secondThrew = error; + } + check("recovery-resolve-releases-its-writer-claim-after-each-run", secondThrew === null, secondThrew && secondThrew.message); +} + +function recoveryResolveRefusesWhileAnotherWriterHoldsTheClaim() { + const dir = freshDir("cli-writer-claim"); + const root = freshDir("cli-writer-claim-root"); + const kp = crypto.generateKeyPairSync("ed25519"); + const keyPath = path.join(root, "signing-key.pem"); + fs.writeFileSync(keyPath, kp.privateKey.export({ type: "pkcs8", format: "pem" }).toString()); + const configPath = path.join(root, "gateway-config.json"); + fs.writeFileSync(configPath, JSON.stringify({ + schema_version: "1.0", + state_dir: dir, + downstream_servers: [{ name: "unused", transport: "stdio", endpoint: "node -e process.exit(1)" }], + signing_key_ref: keyPath, + host_id: "owning-host", + })); + const intentKey = recovery.computeIntentKey("conn-locked", "toolZ", {}); + recovery.createIntentIfAbsent(dir, intentKey, { connection_id: "conn-locked", tool: "toolZ", arguments: {}, state: "ambiguous" }); + + const owner = new WriterClaim(dir, { hostId: "owning-host-2" }); + owner.acquire(); + let threw = null; + try { + runRecoveryResolveCli(["--connection", "conn-locked", "--intent", intentKey, "--confirmed", "not-executed", "--config", configPath], () => {}); + } catch (error) { + threw = error; + } finally { + owner.release(); + } + check("recovery-resolve-refuses-while-another-writer-holds-the-claim", threw !== null && String(threw.code || "").startsWith("WRITER_CLAIM"), threw ? `${threw.code}: ${threw.message}` : "did not throw"); + check("recovery-resolve-refused-attempt-did-not-mutate-the-intent", recovery.readIntent(dir, intentKey).state === "ambiguous", JSON.stringify(recovery.readIntent(dir, intentKey))); +} + +function main() { + // Round-1 fix-plan commit 6: chain.js's admission latch is process-local module + // state, not per-test -- start this whole suite from a known-clean slate regardless + // of require-time or import-order side effects in any dependency. + chain._resetChainIntegrityFailureForTests(); + walAppendAndReadRoundTrip(); + walReadOfMissingConnectionReturnsEmpty(); + walTornTailLineToleratedRestKept(); + walListActiveConnectionsAndDelete(); + walListActiveConnectionsReturnsSortedOrder(); + recoveryFilesAndDirsUseTheSharedAccessModel(); + + // Codex PR #33 review round 2 (2026-09-17): + directoryFsyncRealIoFailurePropagates(); + directoryFsyncUnsupportedAndMissingAreStillTolerated(); + // PR #33/#29 merge-verification correction to frontier-panel finding #4 (2026-09-20): + ensureDirFsyncsThePreexistingParentOfTheTopmostNewAncestor(); + operatorConfirmedExecutedRetainsTheCrossConnectionSignature(); + operatorConfirmedExecutedWithoutAKeyWritesNoSignature(); + abandonConnectionKeepsTheWalWhenTheIntentScanFails(); + walOnEmptyRecoveryDirReturnsNoActiveConnections(); + appendWalEventRetriesOnAShortWrite(); + appendWalEventFailsClosedWhenWriteSyncMakesNoProgress(); + + intentKeyStableRegardlessOfArgumentKeyOrder(); + intentKeyDiffersOnConnectionToolOrArgs(); + intentKeyTreatsUndefinedArgsAsNull(); + createIntentIfAbsentIsRaceSafe(); + readIntentOfMissingKeyReturnsNull(); + updateIntentMergesAndRejectsUnknown(); + deleteIntentIsIdempotent(); + deleteIntentFsyncsTheIntentsDirectoryAfterRemoval(); + listIntentsForConnectionFiltersCorrectly(); + listAllIntentsSpansEveryConnection(); + operatorResolutionExecutedAndNotExecuted(); + resolveIntentGuardsAgainstOverwritingATerminalState(); + + recoverAutoSealsASessionThatCrashedAfterCleanDisconnect(); + recoverClosesFinding1GapUsingCompletedIntentAsProofOfOutcome(); + recoverLeavesUnprovenInFlightCallForOperatorReview(); + recoverIsIdempotentAcrossACrashDuringRecoveryItself(); + recoverDeletesEmptyOrFullyTornWalWithoutFlagging(); + recoverDoesNotBindALaterGenerationsResultToAnEarlierCrashedGenerationsCall(); + walReplayedTwiceInIndependentStateDirsProducesIdenticalBundleId(); + + abandonConnectionSealsAndReleasesTheFenceForAnUnprovenCall(); + abandonConnectionOnAlreadyCleanConnectionIsANoOp(); + abandonConnectionRecordsIncompleteMarkerForOrphanedReplayedStep(); + + pathBearingIdentifiersAreRejected(); + bundleCollisionWithDifferentContentIsFlaggedNotDiscarded(); + bundleCollisionRewriteAllowsAncestorEntryWhenHeadPointsAtANewerTail(); + bundleCollisionRewriteRefusesOnAGenuineForkNotJustAnyAncestorMismatch(); + recoverRepairsAPartialAppendMissingItsChainEntry(); + recoverCrashedSessionsRenewsTheWriterClaimPerConnection(); + recoverCrashedSessionsWithoutAWriterClaimStillWorks(); + recoverCrashedSessionsAbortsImmediatelyIfTheClaimIsLost(); + recoverAbortsOnARawFsErrorFromThePreAppendRenewNotJustNamedClaimCodes(); + recoverAbortsOnLeaseTakeoverDuringWalReplayNotJustOneConnection(); + recoverRepairPathLeaseLossIsNotRelabeledAsAnOrdinaryFailure(); + unreadableWalForOneConnectionDoesNotBlockAnother(); + abandonConnectionQuarantinesAnUnreadableWal(); + abandonConnectionRefusesOnUnverifiedBundleCollision(); + abandonConnectionStillSucceedsOnAGenuineRepeatedAttempt(); + abandonConnectionAbortsWhenThePreAppendRenewFails(); + abandonConnectionRepairPathLeaseLossIsNotRelabeledAsChainRepairFailure(); + notExecutedIntentReplaysAsAFailedCallAndIsThenCleanedUp(); + healthStatusItemizesDispatchedIntentsAlongsideTheCount(); + recoverAmbiguousIntentLogNamesTheRealKeyNotAPlaceholder(); + + recoveryResolveRequiresResultFileForExecuted(); + recoveryResolveRefusesWhileAnotherWriterHoldsTheClaim(); + + return samplingForwardSuccessIsPersistedToWalWithModelCallFlag() + .then(() => samplingForwardErrorIsPersistedAsFailedResult()) + .then(() => samplingForwardRefusesWhenCallStartCannotBeSaved()) + .then(() => samplingCallResultWalFailurePoisonsTheConnectionAndFailsClosed()) + .then(() => samplingCallResultErrorBranchWalFailureStaysFailOpenButIsAttestedAndPoisons()) + .then(() => samplingCallResultFailureAfterACompleteWriteIsWalAuthoritativeOnReplay()) + .then(() => samplingCallResultFailureDuringFsyncIsWalAuthoritativeOnReplay()) + .then(() => samplingCallResultFailureAfterAPartialWriteReplaysTheOrphanedCallStartAsUnresolved()) + .then(() => samplingCallResultCrashBetweenSuccessfulAppendAndInMemoryMutationReplaysIdempotently()) + .then(() => { + recoverPreservesModelCallFlagOnSamplingReplay(); + recoverDoesNotBlockAutoSealOnAnUnresolvedSamplingCall(); + }) + .then(() => { + atomicWritesLoopUntilTheWholePayloadIsWritten(); + quarantineWalFsyncsBothSourceAndDestinationDirectories(); + sweepExpiredSignaturesRemovesOnlyExpiredRecordsAndIsBounded(); + startupRecoverySweepsExpiredSignatureRecords(); + }) + .then(() => writerClaimIsReleasedWhenStartupRecoveryThrows()) + .then(() => { + recoverCrashedSessionsAbortsImmediatelyWhenAlreadyLatched(); + writeStatusFileDocumentsTheCadenceLimitationInTheWrittenOutput(); + writeStatusFileNeverLatchesEvenOnAGenuineTamperedEntryItFinds(); + }) + .then(() => startupHardFailsWritesDiagnosticAndReleasesTheClaimOnAGenuineChainFork()) + .then(() => { + const passed = results.filter((r) => r.status === "PASS").length; + const failed = results.filter((r) => r.status === "FAIL").length; + const skipped = results.filter((r) => r.status === "SKIP").length; + console.log(`SUMMARY passed=${passed} failed=${failed} skipped=${skipped}`); + process.exit(failures ? 1 : 0); + }); +} + +main().catch((error) => { + console.error("FATAL:", error); + process.exit(1); +}); diff --git a/tests/gateway/session/run-tests.js b/tests/gateway/session/run-tests.js index dad7007d..3736f405 100644 --- a/tests/gateway/session/run-tests.js +++ b/tests/gateway/session/run-tests.js @@ -120,6 +120,119 @@ function anomaliesCappedWithTerminalMarker() { ); } +/* Round-1 fix-round follow-up (2026-09-15, "anomaly cap tests and the WAL/disk-growth + * follow-up ticket"): the existing anomaliesCappedWithTerminalMarker test above only ever + * drives the cap through recordUnmatchedResponseAnomaly. recordAnomaly is a SEPARATE + * external entry point (used by proxy.js's blocked-retry path, not by unmatched-response + * handling) that happens to share the same pushCappedAnomaly implementation -- prove the + * cap boundary holds for THIS entry point directly rather than assuming the shared + * implementation makes that redundant. Also settles, with a test rather than an + * assumption (per Paul's 2026-09-15 instruction -- prior review rounds disagreed on this), + * that recordAnomaly's return value flips from true to false exactly at the cap boundary: + * true for every one of the first MAX_ANOMALIES_PER_SESSION calls (including the one that + * also appends the ANOMALY_CAP_REACHED terminal marker), false for every call after. */ +function recordAnomalyRespectsCapWithTerminalMarker() { + const { MAX_ANOMALIES_PER_SESSION } = session; + const s = session.createSession("conn-record-anomaly-cap"); + const returns = []; + for (let i = 0; i < MAX_ANOMALIES_PER_SESSION; i++) { + returns.push(session.recordAnomaly(s, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: `attempt ${i}` })); + } + check( + "recordAnomaly-returns-true-for-every-call-up-to-and-including-the-cap", + returns.every((r) => r === true), + JSON.stringify(returns.filter((r) => r !== true)) + ); + check( + "recordAnomaly-cap-plus-terminal-marker-present-at-MAX", + s.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1 && s.anomalies[s.anomalies.length - 1].kind === "ANOMALY_CAP_REACHED", + `length=${s.anomalies.length}` + ); + // Past the cap: every further call is dropped (returns false) and anomalies stops + // growing entirely -- not even a second terminal marker. + const post1 = session.recordAnomaly(s, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: "past cap #1" }); + const post2 = session.recordAnomaly(s, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: "past cap #2" }); + check("recordAnomaly-returns-false-once-cap-is-reached", post1 === false && post2 === false, JSON.stringify([post1, post2])); + check( + "recordAnomaly-past-cap-does-not-grow-anomalies-further", + s.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1, + `length=${s.anomalies.length}` + ); +} + +/* Frontier-panel decision (Paul, 2026-09-10, cluster E2, see proxy.js's own comment on + * "a reconnecting agent gets a brand-new connectionId"): a reconnect is, at the + * session-capture layer, simply a fresh session.createSession call with its own empty + * anomalies array -- the cap is a PER-SESSION budget, not a per-agent or per-connectionId- + * string lifetime budget. Prove that a session exhausted to its cap does not poison a + * later session: the later one starts at zero and can independently record up to its own + * full budget. */ +function reconnectStartsWithAFreshAnomalyBudget() { + const { MAX_ANOMALIES_PER_SESSION } = session; + const exhausted = session.createSession("conn-reconnect-before-crash"); + for (let i = 0; i < MAX_ANOMALIES_PER_SESSION + 3; i++) { + session.recordAnomaly(exhausted, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: `attempt ${i}` }); + } + check( + "pre-reconnect-session-is-fully-exhausted", + exhausted.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1, + `length=${exhausted.anomalies.length}` + ); + + // The reconnecting agent's new connection -- a DIFFERENT connectionId, matching + // proxy.js's own documented behavior that a reconnect never reuses the old one. + const reconnected = session.createSession("conn-reconnect-after-crash"); + check("reconnected-session-starts-with-zero-anomalies", reconnected.anomalies.length === 0, `length=${reconnected.anomalies.length}`); + const recorded = session.recordAnomaly(reconnected, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: "first attempt after reconnect" }); + check("reconnected-session-records-normally-despite-the-old-session-being-exhausted", recorded === true && reconnected.anomalies.length === 1, JSON.stringify(reconnected.anomalies)); + + // And the reconnected session has its OWN full budget, unaffected by the old one's cap + // having been hit -- fill it independently and confirm it caps at the same boundary. + for (let i = 1; i < MAX_ANOMALIES_PER_SESSION + 3; i++) { + session.recordAnomaly(reconnected, { kind: "GATEWAY_AMBIGUOUS_RETRY", tool: "t", detail: `attempt ${i}` }); + } + check( + "reconnected-session-caps-independently-at-its-own-full-budget", + reconnected.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1, + `length=${reconnected.anomalies.length}` + ); +} + +/* LIMITATION, confirmed here rather than silently accepted (per Paul's 2026-09-15 + * instruction): MAX_ANOMALIES_PER_SESSION is a plain event-COUNT cap with no notion of + * severity or evidentiary value -- it drops the (count+1)-th anomaly regardless of + * whether it is one more low-value duplicate of noise already recorded or the single + * highest-value entry in the whole session (e.g. the one GATEWAY_AMBIGUOUS_RETRY that + * proves a duplicate dispatch was actually prevented). A sufficiently large flood of + * low-value anomalies (e.g. UNMATCHED_RESPONSE from a chatty misbehaving downstream) can + * therefore silently crowd out and suppress a later high-value one from ever reaching the + * sealed bundle. This test proves the suppression happens (it is not hypothetical) and + * exists to keep this limitation visible rather than let it regress into an unexamined + * assumption; see KNOWN-LIMITATIONS.md ("Anomaly cap is an event count, not a value or + * byte budget") for the tracked follow-up. */ +function cappedAnomaliesCanSuppressAHigherValueLaterEvent() { + const { MAX_ANOMALIES_PER_SESSION } = session; + const s = session.createSession("conn-suppression-risk"); + for (let i = 0; i < MAX_ANOMALIES_PER_SESSION; i++) { + session.recordAnomaly(s, { kind: "UNMATCHED_RESPONSE", detail: `low-value noise #${i}` }); + } + check("suppression-setup-cap-exactly-reached", s.anomalies.length === MAX_ANOMALIES_PER_SESSION + 1, `length=${s.anomalies.length}`); + + // A genuinely high-value event arrives immediately after: a blocked duplicate dispatch, + // the exact kind of anomaly Option C's design depends on the sealed bundle attesting. + const recorded = session.recordAnomaly(s, { + kind: "GATEWAY_AMBIGUOUS_RETRY", + tool: "transfer-funds", + detail: "a prior attempt of this exact operation was still in flight -- dispatch halted", + }); + check("high-value-anomaly-after-cap-is-silently-dropped-not-recorded", recorded === false, String(recorded)); + check( + "high-value-anomaly-kind-does-not-appear-anywhere-in-the-sealed-anomalies", + !s.anomalies.some((a) => a.kind === "GATEWAY_AMBIGUOUS_RETRY"), + JSON.stringify(s.anomalies.filter((a) => a.kind === "GATEWAY_AMBIGUOUS_RETRY")) + ); +} + function disconnectMarksPendingAsError() { const s = session.createSession("conn-4"); session.recordCallStart(s, 1, { tool: "hangs", server: "srv", arguments: {}, ts: 1 }); @@ -240,6 +353,46 @@ function malformedResultFlagStoredAndPropagatedToSealedTrace() { check("sealed-trace-omits-malformed-result-key-for-well-formed-call-byte-parity", !traceClean.includes("malformed_result"), traceClean); } +/* Cluster B (session-identity distinctness in the audit trail): two sessions that + * record byte-identical {init, tools, calls, results} must NOT be treated as "the same + * session" merely because gsa-mcp-shim.js's own bundle_id formula cannot otherwise tell + * them apart -- see that module's own doc comment on why session_id was folded into the + * hash. Also asserts createSession mints a session_id automatically when the caller + * (the live proxy.js#openConnection path) does not supply one. */ +function identicalContentDifferentSessionIdsProduceDifferentBundleIds() { + const keys = makeKeys(); + function sealWithSessionId(sessionId) { + const s = session.createSession("conn-identical", { sessionId }); + session.recordInitialize(s, { clientInfo: { name: "same-agent", version: "1" } }); + session.recordToolsList(s, [{ name: "t", server: "srv", schema: {} }]); + session.recordCallStart(s, 1, { tool: "t", server: "srv", arguments: { a: 1 }, ts: 1 }); + session.recordCallResult(s, 1, { result: { ok: true }, ts: 2 }); + return session.finalizeSession(s, keys); + } + const sealedA = sealWithSessionId("session-id-A"); + const sealedB = sealWithSessionId("session-id-B"); + check( + "identical-content-different-session-ids-produce-different-bundle-ids", + sealedA.bundle.manifest.bundle_id !== sealedB.bundle.manifest.bundle_id, + JSON.stringify({ a: sealedA.bundle.manifest.bundle_id, b: sealedB.bundle.manifest.bundle_id }) + ); + // Same session_id + identical content must still be deterministic (this is what makes + // "replay the same session twice" produce the same bundle_id in the first place). + const sealedA2 = sealWithSessionId("session-id-A"); + check( + "identical-content-and-session-id-produce-the-same-bundle-id", + sealedA.bundle.manifest.bundle_id === sealedA2.bundle.manifest.bundle_id, + JSON.stringify({ a: sealedA.bundle.manifest.bundle_id, a2: sealedA2.bundle.manifest.bundle_id }) + ); +} + +function sessionIdAutoGeneratedWhenNotProvided() { + const s1 = session.createSession("conn-auto-1"); + const s2 = session.createSession("conn-auto-2"); + check("session-id-auto-generated-when-omitted", typeof s1.sessionId === "string" && s1.sessionId.length > 0, String(s1.sessionId)); + check("session-id-auto-generated-differs-across-sessions", s1.sessionId !== s2.sessionId, JSON.stringify([s1.sessionId, s2.sessionId])); +} + function main() { createSessionRejectsBadConnectionId(); toolServerAttribution(); @@ -247,6 +400,9 @@ function main() { unmatchedResponseIsAnomalyNotCrash(); unmatchedResponseAnomalyDoesNotTouchPendingCalls(); anomaliesCappedWithTerminalMarker(); + recordAnomalyRespectsCapWithTerminalMarker(); + reconnectStartsWithAFreshAnomalyBudget(); + cappedAnomaliesCanSuppressAHigherValueLaterEvent(); disconnectMarksPendingAsError(); disconnectCallbackFailureDoesNotAbortLoopOrPropagate(); modelCallFlagPreserved(); @@ -254,6 +410,8 @@ function main() { duplicateJsonRpcIdRejected(); finalizeRefusesWithPendingCalls(); finalizeSealsAndVerifies(); + sessionIdAutoGeneratedWhenNotProvided(); + identicalContentDifferentSessionIdsProduceDifferentBundleIds(); const passed = results.filter((r) => r.status === "PASS").length; const failed = results.filter((r) => r.status === "FAIL").length; diff --git a/tests/gateway/startup-permissions/run-tests.js b/tests/gateway/startup-permissions/run-tests.js new file mode 100644 index 00000000..cbb9b346 --- /dev/null +++ b/tests/gateway/startup-permissions/run-tests.js @@ -0,0 +1,404 @@ +#!/usr/bin/env node +"use strict"; + +/* Regression suite for scripts/gateway/startup-permissions.js -- the gateway's startup + * permission-tightening pass (Paul's decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model": group-readable + * state directories, 0750 dirs / 0640 files, NOT single-user 0700/0600). + * + * Covers: + * - the exact, explicit set of directories/files this pass targets (a drift/typo test: + * accidentally adding gateway-status.json, or dropping one of the real sensitive + * paths, should fail a test here, not be discovered in production). + * - real files produced by the REAL writer-claim.js/chain.js/recovery.js write paths + * (not hand-crafted fixtures) end up re-chmoded correctly, proving this module's + * path-builder calls actually line up with those modules' own. + * - a missing file/directory (a fresh deployment) is skipped silently, never an error. + * - the operator-facing gateway-status.json health file is NEVER touched by this pass, + * even though it lives directly inside the same state directory this pass tightens. + * - a real permission fault (EPERM/EIO, simulated) is surfaced, not swallowed. + */ + +const crypto = require("crypto"); +const fs = require("fs"); +const os = require("os"); +const path = require("path"); + +const ROOT = path.resolve(__dirname, "../../.."); +const startupPermissions = require(path.join(ROOT, "scripts", "gateway", "startup-permissions.js")); +const recovery = require(path.join(ROOT, "scripts", "gateway", "recovery.js")); +const chain = require(path.join(ROOT, "scripts", "gateway", "chain.js")); +const { WriterClaim } = require(path.join(ROOT, "scripts", "writer-claim.js")); +const stateStore = require(path.join(ROOT, "scripts", "state-store.js")); +const session = require(path.join(ROOT, "scripts", "gateway", "session.js")); +const { GatewayProxy } = require(path.join(ROOT, "scripts", "gateway", "proxy.js")); + +/** A genuinely sealed bundle produced by session.js's own real finalizeSession -- used by + * the runtime-creation-mode tests below, which must drive the REAL chain.js/proxy.js write + * paths rather than hand-crafting files whose modes would prove nothing. */ +function makeSealedBundle(keys, connectionId) { + const s = session.createSession(connectionId, { now: () => Date.now() }); + session.recordInitialize(s, { clientInfo: {}, serverInfo: {} }); + session.recordToolsList(s, []); + return session.finalizeSession(s, keys); +} + +let failures = 0; +const results = []; +function record(name, status, reason) { + console.log(status === "PASS" ? `PASS ${name}` : `FAIL ${name}+${reason || "unknown"}`); + results.push({ name, status, reason: reason || "" }); + if (status === "FAIL") failures++; +} +function check(name, cond, reason) { + record(name, cond ? "PASS" : "FAIL", reason); +} + +function freshDir(prefix) { + return fs.mkdtempSync(path.join(os.tmpdir(), `gs-startup-permissions-${prefix}-`)); +} +function mode(p) { + return fs.statSync(p).mode & 0o777; +} +function withPatched(obj, key, replacement, fn) { + const original = obj[key]; + obj[key] = replacement; + try { return fn(); } finally { obj[key] = original; } +} +function withPlatform(value, fn) { + const original = Object.getOwnPropertyDescriptor(process, "platform"); + Object.defineProperty(process, "platform", { value, configurable: true }); + try { return fn(); } finally { Object.defineProperty(process, "platform", original); } +} +function codeError(code, message) { + return Object.assign(new Error(message || code), { code }); +} +function makeClaimInstanceId() { + return crypto.randomBytes(16).toString("hex"); +} + +/* ---- exact target-list tests: catches drift/typos in the sensitive-path lists ---- */ + +function sensitiveDirectoriesListIsExactlyTheDocumentedSet() { + const dir = freshDir("dirs-list"); + const expected = [ + dir, + chain.sessionsDir(dir), + startupPermissions.sessionQuarantineDir(dir), + recovery.recoveryDir(dir), + recovery.activeDir(dir), + recovery.intentsDir(dir), + recovery.signaturesDir(dir), + ].sort(); + const actual = startupPermissions.sensitiveDirectories(dir).slice().sort(); + check("sensitive-directories-list-matches-documented-set", + JSON.stringify(actual) === JSON.stringify(expected), + `expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}`); +} + +function sensitiveFilesListNamesTheFixedFilesAndNeverHeadJsonTwice() { + const dir = freshDir("files-list-fixed"); + fs.mkdirSync(chain.sessionsDir(dir), { recursive: true }); + // No HEAD.json/chain.jsonl on disk yet -- the fixed-name entries are still LISTED + // (chmodPathOrFail's own ENOENT handling is what makes a missing one harmless), and + // sessionsDir's own glob must never separately re-list HEAD.json as if it were a + // sealed bundle. + fs.writeFileSync(chain.headPath(dir), "{}"); + const files = startupPermissions.sensitiveFiles(dir); + const headOccurrences = files.filter((f) => f === chain.headPath(dir)).length; + check("sensitive-files-includes-writer-claim-path", + files.includes(require(path.join(ROOT, "scripts", "writer-claim.js")).claimPath(dir)), JSON.stringify(files)); + check("sensitive-files-includes-chain-jsonl-path", files.includes(chain.chainPath(dir)), JSON.stringify(files)); + check("sensitive-files-includes-head-json-path", files.includes(chain.headPath(dir)), JSON.stringify(files)); + check("sensitive-files-lists-head-json-exactly-once-not-duplicated-via-sessions-glob", + headOccurrences === 1, `expected exactly 1 occurrence of HEAD.json, got ${headOccurrences}`); +} + +function sensitiveFilesNeverListsTheOperatorStatusFile() { + const dir = freshDir("files-list-status-excluded"); + fs.mkdirSync(dir, { recursive: true }); + const statusPath = path.join(dir, "gateway-status.json"); + fs.writeFileSync(statusPath, "{}"); + const files = startupPermissions.sensitiveFiles(dir); + check("sensitive-files-never-includes-gateway-status-json", + !files.includes(statusPath), JSON.stringify(files)); +} + +/* ---- integration: real files produced by the real write paths ---- */ + +function tightenStateDirPermissionsRetrofitsEveryRealSensitivePathToTheAccessModel() { + const dir = freshDir("real-files"); + + // Real writer-claim. + const claim = new WriterClaim(dir, { instanceId: makeClaimInstanceId() }); + claim.acquire(); + const claimPath = claim.path; + fs.chmodSync(claimPath, 0o644); // simulate a claim written before this fix existed + + // Real chain: one sealed bundle -> chain.jsonl append -> HEAD.json. + const sealed = { bundle: { manifest: { bundle_id: "bundle-real-1" }, contents: {} } }; + chain.appendSession(dir, sealed); + fs.chmodSync(chain.bundlePath(dir, "bundle-real-1"), 0o644); + fs.chmodSync(chain.chainPath(dir), 0o644); + fs.chmodSync(chain.headPath(dir), 0o644); + + // A real quarantined sealed bundle, mirroring proxy.js#quarantineSealedBundle's own + // write shape (this test writes it directly rather than importing proxy.js, keeping + // this suite's dependency graph the same one-way shape as startup-permissions.js's + // own -- see that module's header comment on why). + const quarantineDir = startupPermissions.sessionQuarantineDir(dir); + fs.mkdirSync(quarantineDir, { recursive: true }); + const quarantinedBundlePath = path.join(quarantineDir, "conn-1-123.json"); + fs.writeFileSync(quarantinedBundlePath, JSON.stringify({ sealed }), { flag: "wx" }); + fs.chmodSync(quarantinedBundlePath, 0o644); + + // Real WAL, intent, and signature records. + recovery.appendWalEvent(dir, "conn-2", { type: "SESSION_START" }); + const walPath = recovery.walPath(dir, "conn-2"); + fs.chmodSync(walPath, 0o644); + fs.chmodSync(recovery.activeDir(dir), 0o755); + + recovery.createIntentIfAbsent(dir, "gs_intent_1", { state: "dispatched" }); + const intentPath = recovery.intentPath(dir, "gs_intent_1"); + fs.chmodSync(intentPath, 0o644); + fs.chmodSync(recovery.intentsDir(dir), 0o755); + + recovery.recordCompletedSignature(dir, "gs_sig_1", { completed_at: Date.now() }); + const signaturePath = recovery.signaturePath(dir, "gs_sig_1"); + fs.chmodSync(signaturePath, 0o644); + fs.chmodSync(recovery.signaturesDir(dir), 0o755); + + // The one named exception: the operator health file, deliberately left at a normal, + // pre-existing operator-readable mode. + const statusPath = path.join(dir, "gateway-status.json"); + fs.writeFileSync(statusPath, "{}"); + fs.chmodSync(statusPath, 0o644); + + fs.chmodSync(chain.sessionsDir(dir), 0o755); + fs.chmodSync(quarantineDir, 0o755); + fs.chmodSync(recovery.recoveryDir(dir), 0o755); + fs.chmodSync(dir, 0o755); + + const changed = startupPermissions.tightenStateDirPermissions(dir); + + check("real-writer-claim-file-tightened-to-0640", mode(claimPath) === 0o640, mode(claimPath).toString(8)); + check("real-bundle-file-tightened-to-0640", mode(chain.bundlePath(dir, "bundle-real-1")) === 0o640, ""); + check("real-chain-jsonl-tightened-to-0640", mode(chain.chainPath(dir)) === 0o640, ""); + check("real-head-json-tightened-to-0640", mode(chain.headPath(dir)) === 0o640, ""); + check("real-quarantined-bundle-tightened-to-0640", mode(quarantinedBundlePath) === 0o640, ""); + check("real-wal-file-tightened-to-0640", mode(walPath) === 0o640, ""); + check("real-intent-file-tightened-to-0640", mode(intentPath) === 0o640, ""); + check("real-signature-file-tightened-to-0640", mode(signaturePath) === 0o640, ""); + + check("real-state-dir-tightened-to-0750", mode(dir) === 0o750, ""); + check("real-sessions-dir-tightened-to-0750", mode(chain.sessionsDir(dir)) === 0o750, ""); + check("real-quarantine-dir-tightened-to-0750", mode(quarantineDir) === 0o750, ""); + check("real-recovery-dir-tightened-to-0750", mode(recovery.recoveryDir(dir)) === 0o750, ""); + check("real-active-dir-tightened-to-0750", mode(recovery.activeDir(dir)) === 0o750, ""); + check("real-intents-dir-tightened-to-0750", mode(recovery.intentsDir(dir)) === 0o750, ""); + check("real-signatures-dir-tightened-to-0750", mode(recovery.signaturesDir(dir)) === 0o750, ""); + + check("gateway-status-json-mode-untouched-by-this-pass", mode(statusPath) === 0o644, + `expected the operator status file to be left at 0644, got ${mode(statusPath).toString(8)}`); + check("gateway-status-json-not-reported-as-changed", + !changed.files.includes(statusPath), JSON.stringify(changed.files)); + + const uniqueFiles = new Set(changed.files); + check("no-file-reported-as-changed-twice", + uniqueFiles.size === changed.files.length, + `duplicates in ${JSON.stringify(changed.files)}`); +} + +/* ---- missing paths are ordinary, not a fault ---- */ + +function freshDeploymentWithNothingOnDiskYetDoesNotThrow() { + const dir = freshDir("empty"); + fs.mkdirSync(dir, { recursive: true }); + let threw = null; + let changed = null; + try { changed = startupPermissions.tightenStateDirPermissions(dir); } + catch (error) { threw = error; } + check("empty-state-dir-does-not-throw", threw === null, threw && threw.message); + check("empty-state-dir-still-tightens-the-state-dir-itself", + changed && changed.dirs.includes(dir), JSON.stringify(changed)); + check("empty-state-dir-reports-no-files-changed", + changed && changed.files.length === 0, JSON.stringify(changed)); + check("empty-state-dir-mode-is-0750", mode(dir) === 0o750, mode(dir).toString(8)); +} + +function partiallyPopulatedDeploymentSkipsWhatIsMissingAndTightensWhatExists() { + const dir = freshDir("partial"); + // Only the writer-claim exists; nothing gateway-sessions/gateway-recovery related + // has ever been created (no crash has ever happened on this deployment yet). + const claim = new WriterClaim(dir, { instanceId: makeClaimInstanceId() }); + claim.acquire(); + fs.chmodSync(claim.path, 0o644); + + let threw = null; + let changed = null; + try { changed = startupPermissions.tightenStateDirPermissions(dir); } + catch (error) { threw = error; } + check("partial-deployment-does-not-throw", threw === null, threw && threw.message); + check("partial-deployment-tightens-the-existing-claim-file", + mode(claim.path) === 0o640, mode(claim.path).toString(8)); + check("partial-deployment-reports-only-the-existing-file", + changed.files.length === 1 && changed.files[0] === claim.path, JSON.stringify(changed.files)); +} + +/* ---- a real permission fault is surfaced, not swallowed ---- */ + +function aRealChmodFailureAbortsTheWholePassRatherThanBeingSwallowed() { + const dir = freshDir("real-fault"); + const claim = new WriterClaim(dir, { instanceId: makeClaimInstanceId() }); + claim.acquire(); + + let threw = null; + withPlatform("linux", () => { + withPatched(fs, "chmodSync", (target, requestedMode) => { + throw codeError("EACCES", "permission denied"); + }, () => { + try { startupPermissions.tightenStateDirPermissions(dir); } + catch (error) { threw = error; } + }); + }); + check("real-chmod-failure-propagates-out-of-tightenStateDirPermissions", + threw && threw.code === "STATE_STORE_CHMOD_FAILED", + `expected STATE_STORE_CHMOD_FAILED, got ${threw ? threw.code : "no error"}`); +} + + +/* Codex PR #33 review "enforce modes when creating post-startup session files": this + * permission pass runs ONCE, at startup, and skips paths that do not exist yet. On a fresh + * state directory gateway-sessions/ and chain.jsonl are created LATER -- by chain.js's own + * mkdirSync/openSync -- and the sealed-bundle quarantine directory/file are created later + * still, only when an append fails. Without explicit modes at those runtime creation sites + * they are born 0755/0644 (world-readable) under the usual 022 umask and stay that way + * until the next restart re-runs this pass. Deliberately runs with a PERMISSIVE umask, so + * a regression cannot be masked by whatever umask the CI runner happens to have. */ +function runtimeCreatedSessionPathsAreBornWithTheConfiguredModes() { + if (process.platform === "win32") { + console.log("SKIP runtime-created-session-paths-get-the-configured-modes (POSIX modes are not meaningful on win32)"); + results.push({ name: "runtime-created-session-paths-get-the-configured-modes", status: "SKIP", reason: "win32" }); + return; + } + const dir = freshDir("runtime-modes"); + const keys = (() => { const kp = crypto.generateKeyPairSync("ed25519"); return { privateKey: kp.privateKey, signer: "test-key", algo: "ed25519" }; })(); + const sealed = makeSealedBundle(keys, "runtime-modes-session"); + + const previousUmask = process.umask(0o022); + try { + // NOTE: the startup pass is deliberately NOT run here -- these paths do not exist at + // startup on a fresh state directory, which is precisely the gap being closed. + chain.appendSession(dir, sealed); + } finally { + process.umask(previousUmask); + } + check( + "runtime-created-gateway-sessions-dir-gets-the-configured-dir-mode", + mode(chain.sessionsDir(dir)) === stateStore.DEFAULT_DIR_MODE, + `${chain.sessionsDir(dir)} is 0${mode(chain.sessionsDir(dir)).toString(8)}, expected 0${stateStore.DEFAULT_DIR_MODE.toString(8)}` + ); + check( + "runtime-created-chain-jsonl-gets-the-configured-file-mode", + mode(chain.chainPath(dir)) === stateStore.DEFAULT_FILE_MODE, + `${chain.chainPath(dir)} is 0${mode(chain.chainPath(dir)).toString(8)}, expected 0${stateStore.DEFAULT_FILE_MODE.toString(8)}` + ); + check( + "runtime-created-chain-jsonl-is-never-world-readable", + (mode(chain.chainPath(dir)) & 0o007) === 0, + `0${mode(chain.chainPath(dir)).toString(8)}` + ); +} + +/* Same Codex finding, for the worst of the three paths: proxy.js#quarantineSealedBundle + * writes the FULL raw sealed bundle, and both its directory and its file are created only + * at the moment an append fails -- long after this module's one-time startup pass. */ +function runtimeCreatedSealedBundleQuarantineIsBornWithTheConfiguredModes() { + if (process.platform === "win32") { + console.log("SKIP runtime-created-sealed-bundle-quarantine-gets-the-configured-modes (POSIX modes are not meaningful on win32)"); + results.push({ name: "runtime-created-sealed-bundle-quarantine-gets-the-configured-modes", status: "SKIP", reason: "win32" }); + return; + } + const dir = freshDir("runtime-quarantine-modes"); + const keys = (() => { const kp = crypto.generateKeyPairSync("ed25519"); return { privateKey: kp.privateKey, signer: "test-key", algo: "ed25519" }; })(); + const sealed = makeSealedBundle(keys, "runtime-quarantine-session"); + + // Drive the REAL production path: an append that fails after sealing quarantines the + // bundle. isWriterClaimValid()===false is exactly the branch closeConnection takes to + // quarantine rather than append (see proxy.js's writer-claim revalidation). + const conn = { + transport: "fake", + call: async () => ({ ok: true }), + close: () => {}, + isClosed: () => false, + whenClosed: () => new Promise(() => {}), + }; + const proxy = new GatewayProxy({ + connections: new Map([["srv", conn]]), + mergedTools: [{ name: "echo", server: "srv", schema: {} }], + toolOwners: new Map([["echo", "srv"]]), + serverInfos: {}, + keys, + stateDir: dir, + log: () => {}, + isWriterClaimValid: () => false, + onSealFailure: () => {}, + }); + + const previousUmask = process.umask(0o022); + let quarantineDir = null; + return (async () => { + try { + proxy.openConnection("conn-q"); + await proxy.handleMessage("conn-q", { jsonrpc: "2.0", id: 0, method: "initialize", params: {} }); + await proxy.handleMessage("conn-q", { jsonrpc: "2.0", method: "notifications/initialized", params: {} }); + await proxy.closeConnection("conn-q", "forcing a quarantine"); + } finally { + process.umask(previousUmask); + } + quarantineDir = startupPermissions.sessionQuarantineDir(dir); + const files = fs.existsSync(quarantineDir) ? fs.readdirSync(quarantineDir).filter((n) => n.endsWith(".json")) : []; + check("a-sealed-bundle-was-actually-quarantined", files.length === 1, `quarantine dir contents: ${JSON.stringify(files)}`); + if (files.length !== 1) return; + check( + "runtime-created-sealed-bundle-quarantine-dir-gets-the-configured-dir-mode", + mode(quarantineDir) === stateStore.DEFAULT_DIR_MODE, + `0${mode(quarantineDir).toString(8)}, expected 0${stateStore.DEFAULT_DIR_MODE.toString(8)}` + ); + const quarantinedFile = path.join(quarantineDir, files[0]); + check( + "runtime-created-sealed-bundle-quarantine-file-gets-the-configured-file-mode", + mode(quarantinedFile) === stateStore.DEFAULT_FILE_MODE, + `0${mode(quarantinedFile).toString(8)}, expected 0${stateStore.DEFAULT_FILE_MODE.toString(8)}` + ); + check( + "the-raw-quarantined-sealed-bundle-is-never-world-readable", + (mode(quarantinedFile) & 0o007) === 0, + `0${mode(quarantinedFile).toString(8)}` + ); + })(); +} + +function main() { + sensitiveDirectoriesListIsExactlyTheDocumentedSet(); + sensitiveFilesListNamesTheFixedFilesAndNeverHeadJsonTwice(); + sensitiveFilesNeverListsTheOperatorStatusFile(); + tightenStateDirPermissionsRetrofitsEveryRealSensitivePathToTheAccessModel(); + freshDeploymentWithNothingOnDiskYetDoesNotThrow(); + partiallyPopulatedDeploymentSkipsWhatIsMissingAndTightensWhatExists(); + aRealChmodFailureAbortsTheWholePassRatherThanBeingSwallowed(); + runtimeCreatedSessionPathsAreBornWithTheConfiguredModes(); + + return Promise.resolve(runtimeCreatedSealedBundleQuarantineIsBornWithTheConfiguredModes()).then(() => { + const passed = results.filter((r) => r.status === "PASS").length; + const failed = results.filter((r) => r.status === "FAIL").length; + const skipped = results.filter((r) => r.status === "SKIP").length; + console.log(`SUMMARY passed=${passed} failed=${failed} skipped=${skipped}`); + process.exit(failures ? 1 : 0); + }); +} + +main().catch((error) => { + console.error("FATAL:", error); + process.exit(1); +}); diff --git a/tests/state-store/atomic-primitives/run-tests.js b/tests/state-store/atomic-primitives/run-tests.js index bcd769f3..9da94ec2 100644 --- a/tests/state-store/atomic-primitives/run-tests.js +++ b/tests/state-store/atomic-primitives/run-tests.js @@ -62,6 +62,17 @@ function codeError(code, message) { return Object.assign(new Error(message || code), { code }); } +/* process.umask() is PROCESS-GLOBAL, not per-call -- every test below that uses this + * sets it, runs the assertion synchronously (no `await`, no callback, nothing that could + * yield the event loop between the set and the restore), and restores the ORIGINAL value + * in `finally` before returning. This file's own `main()` calls each test function to + * completion, one at a time, in a single process -- there is no concurrent test that + * could observe or race the umask while it is temporarily changed here. */ +function withUmask(mask, fn) { + const original = process.umask(mask); + try { return fn(); } finally { process.umask(original); } +} + /* ---- atomicCreateExclusive ---- */ function atomicCreateExclusiveContentIsActuallyWritten() { @@ -259,6 +270,179 @@ function atomicOverwriteFileUnlistedCodeSwallowedOnWin32() { "the rename must have already happened before the swallowed dir-fsync failure"); } +/* ---- access-model defaults (Paul's decision, 2026-09-15 fix round -- see + * claude/graphsmith-fix-round-plan-round1-2026-09-15.md, "Access model"): group-readable + * (0750/0640), NOT single-user (0700/0600). These two functions are the ONLY ones that + * create genuinely new files going forward; the retrofit of files that already exist on + * disk is a separate startup pass (scripts/gateway/startup-permissions.js, tested in its + * own suite). ---- */ + +function atomicCreateExclusiveDefaultModeIsGroupReadable() { + const dir = freshDir("create-default-mode"); + const target = path.join(dir, "record.json"); + stateStore.atomicCreateExclusive(target, "payload"); + const mode = fs.statSync(target).mode & 0o777; + check("atomicCreateExclusive-default-mode-is-0640", mode === 0o640, + `expected the new access-model default 0640, got ${mode.toString(8)}`); +} + +function atomicOverwriteFileDefaultModeIsGroupReadable() { + const dir = freshDir("overwrite-default-mode"); + const target = path.join(dir, "record.json"); + fs.writeFileSync(target, "seed"); + stateStore.atomicOverwriteFile(target, "new-content", dir); + const mode = fs.statSync(target).mode & 0o777; + check("atomicOverwriteFile-default-mode-is-0640", mode === 0o640, + `expected the new access-model default 0640, got ${mode.toString(8)}`); +} + +/* THE REQUIRED umask(0) CASE: proves fchmodSync is doing real, load-bearing work, not + * merely riding along with whatever the ambient umask would have produced anyway. Under + * umask 0, fs.openSync's own default mode (0666, unmasked) would leave the file WORLD + * WRITABLE if this function only relied on the mode passed to open()/the process umask -- + * exactly the defect this test exists to catch. The access-model default (0640) can only + * appear here because atomicCreateExclusive explicitly fchmodSync's the temp file + * afterward, irrespective of umask. */ +function atomicCreateExclusiveUmaskZeroStillProducesConfiguredMode() { + const dir = freshDir("create-umask-zero"); + const target = path.join(dir, "record.json"); + withUmask(0, () => { + stateStore.atomicCreateExclusive(target, "payload"); + }); + const mode = fs.statSync(target).mode & 0o777; + check("atomicCreateExclusive-umask-0-still-yields-0640-not-0666", mode === 0o640, + `expected 0640 regardless of umask 0 -- a result of 0666 here would mean fchmodSync ` + + `is not actually running, only the ambient umask is -- got ${mode.toString(8)}`); +} + +/* THE REQUIRED umask(022) / pre-existing-0644 CASE: proves atomicOverwriteFile's rename + * delivers the NEW temp file's mode, not whatever mode the OLD target file already had. + * Under umask 022, a temp file created with NO explicit mode would land at 0644 -- THE + * SAME as the pre-existing target -- which would make this assertion pass even if the + * rename silently preserved the old target's mode instead of replacing it. Choosing a + * DIFFERENT default (0640) for the temp file is what makes "did the final mode come from + * the rename's new inode, or survive from the old target" an observable, falsifiable + * question rather than a coincidence of two paths agreeing on 0644. */ +function atomicOverwriteFileRenameCarriesNewInodeModeOverPreExistingFile() { + const dir = freshDir("overwrite-umask-022-preexisting"); + const target = path.join(dir, "record.json"); + fs.writeFileSync(target, "seed"); + fs.chmodSync(target, 0o644); + const preExistingMode = fs.statSync(target).mode & 0o777; + withUmask(0o022, () => { + stateStore.atomicOverwriteFile(target, "new-content", dir); + }); + const mode = fs.statSync(target).mode & 0o777; + check("atomicOverwriteFile-pre-existing-target-really-was-0644", preExistingMode === 0o644, + `test setup invariant broken: expected the pre-existing target to be 0644, got ${preExistingMode.toString(8)}`); + check("atomicOverwriteFile-rename-carries-new-inode-mode-not-preexisting-0644", mode === 0o640, + `expected the post-rename target to carry the NEW temp file's mode (0640), not the ` + + `pre-existing target's 0644 -- got ${mode.toString(8)}`); + check("atomicOverwriteFile-rename-carries-new-inode-mode-content-still-correct", + fs.readFileSync(target, "utf8") === "new-content", + "content should still be correctly written despite the mode assertion"); +} + +function atomicCreateExclusiveModeNullOptsOutOfEnforcement() { + const dir = freshDir("create-mode-null"); + const target = path.join(dir, "record.json"); + withUmask(0, () => { + stateStore.atomicCreateExclusive(target, "payload", { mode: null }); + }); + const mode = fs.statSync(target).mode & 0o777; + check("atomicCreateExclusive-mode-null-opts-out-of-enforcement", mode === 0o666, + `expected { mode: null } to leave the raw, umask-governed default (0666 under umask ` + + `0) untouched rather than enforce 0640, got ${mode.toString(8)}`); +} + +function atomicOverwriteFileModeNullSkipsFchmodEnforcement() { + const dir = freshDir("overwrite-mode-null"); + const target = path.join(dir, "record.json"); + fs.writeFileSync(target, "seed"); + fs.chmodSync(target, 0o644); + withUmask(0, () => { + stateStore.atomicOverwriteFile(target, "new-content", dir, { mode: null }); + }); + const mode = fs.statSync(target).mode & 0o777; + check("atomicOverwriteFile-mode-null-skips-fchmod-enforcement", mode === 0o666, + `expected { mode: null } under umask 0 to leave the temp file's raw default (0666) ` + + `rather than enforce 0640 -- got ${mode.toString(8)}`); +} + +function atomicCreateExclusiveSurfacesRealChmodFailures() { + const dir = freshDir("create-chmod-fails"); + const target = path.join(dir, "record.json"); + let threw = null; + withPlatform("linux", () => { + withPatched(fs, "fchmodSync", () => { throw codeError("EACCES", "no chmod for you"); }, () => { + try { stateStore.atomicCreateExclusive(target, "payload"); } + catch (error) { threw = error; } + }); + }); + check("atomicCreateExclusive-real-chmod-failure-surfaces", + threw && threw.code === "STATE_STORE_CHMOD_FAILED", + `expected STATE_STORE_CHMOD_FAILED, got ${threw ? threw.code : "no error"}`); +} + +function atomicCreateExclusiveSwallowsChmodFailureOnWin32() { + const dir = freshDir("create-chmod-win32"); + const target = path.join(dir, "record.json"); + let threw = null; + withPlatform("win32", () => { + withPatched(fs, "fchmodSync", () => { throw codeError("EACCES", "no posix modes here"); }, () => { + try { stateStore.atomicCreateExclusive(target, "payload"); } + catch (error) { threw = error; } + }); + }); + check("atomicCreateExclusive-chmod-failure-swallowed-on-win32", threw === null, + `expected a win32 chmod failure to be swallowed, but it threw: ${threw && threw.code}`); + check("atomicCreateExclusive-chmod-failure-swallowed-on-win32-content-still-written", + fs.readFileSync(target, "utf8") === "payload", + "content should still land despite the swallowed chmod failure"); +} + +function chmodPathOrFailSkipsMissingPathSilently() { + const dir = freshDir("chmod-path-missing"); + const missing = path.join(dir, "does-not-exist.json"); + const result = stateStore.chmodPathOrFail(missing, 0o640); + check("chmodPathOrFail-missing-path-returns-false-not-throw", result === false, + `expected false for a missing path, got ${result}`); +} + +function chmodPathOrFailSurfacesRealErrorOnPosix() { + const dir = freshDir("chmod-path-real-error"); + const target = path.join(dir, "record.json"); + fs.writeFileSync(target, "seed"); + let threw = null; + withPlatform("linux", () => { + withPatched(fs, "chmodSync", () => { throw codeError("EACCES", "denied"); }, () => { + try { stateStore.chmodPathOrFail(target, 0o640); } + catch (error) { threw = error; } + }); + }); + check("chmodPathOrFail-real-error-surfaces", + threw && threw.code === "STATE_STORE_CHMOD_FAILED", + `expected STATE_STORE_CHMOD_FAILED, got ${threw ? threw.code : "no error"}`); +} + +function chmodPathOrFailSwallowsOnWin32() { + const dir = freshDir("chmod-path-win32"); + const target = path.join(dir, "record.json"); + fs.writeFileSync(target, "seed"); + let threw = null; + let result; + withPlatform("win32", () => { + withPatched(fs, "chmodSync", () => { throw codeError("EACCES", "denied"); }, () => { + try { result = stateStore.chmodPathOrFail(target, 0o640); } + catch (error) { threw = error; } + }); + }); + check("chmodPathOrFail-win32-swallowed-not-thrown", threw === null, + `expected a win32 chmod failure to be swallowed, got ${threw && threw.code}`); + check("chmodPathOrFail-win32-swallowed-returns-false", result === false, + `expected false for a swallowed win32 chmod failure, got ${result}`); +} + /* ---- validateNamedRecord ---- */ function validateNamedRecordRejectsUnknownDefName() { @@ -296,6 +480,17 @@ function main() { atomicOverwriteFileRenameFailureCleansUpTempFile(); atomicOverwriteFileUnlistedCodePropagatesOnPosixOnly(); atomicOverwriteFileUnlistedCodeSwallowedOnWin32(); + atomicCreateExclusiveDefaultModeIsGroupReadable(); + atomicOverwriteFileDefaultModeIsGroupReadable(); + atomicCreateExclusiveUmaskZeroStillProducesConfiguredMode(); + atomicOverwriteFileRenameCarriesNewInodeModeOverPreExistingFile(); + atomicCreateExclusiveModeNullOptsOutOfEnforcement(); + atomicOverwriteFileModeNullSkipsFchmodEnforcement(); + atomicCreateExclusiveSurfacesRealChmodFailures(); + atomicCreateExclusiveSwallowsChmodFailureOnWin32(); + chmodPathOrFailSkipsMissingPathSilently(); + chmodPathOrFailSurfacesRealErrorOnPosix(); + chmodPathOrFailSwallowsOnWin32(); validateNamedRecordRejectsUnknownDefName(); validateNamedRecordInvalidRecordMessageNamesDefAndContext(); diff --git a/tests/state-store/writer-claim/run-tests.js b/tests/state-store/writer-claim/run-tests.js index 068c1d38..dd3ea10f 100644 --- a/tests/state-store/writer-claim/run-tests.js +++ b/tests/state-store/writer-claim/run-tests.js @@ -592,6 +592,94 @@ async function crCodex_heartbeatSurfacesClaimLoss() { `expected an uncaught WRITER_CLAIM_LOST when no onClaimLost handler was given, got ${uncaught && uncaught.code}`); } +/* Round-1 fix-plan item 1 (lease keepalive, generalized): maybeRenew() is the shared, + * time-gated renewal helper gateway.js's WAL replay loops (and, in later commits, + * chain.reconcileHead's per-entry walk / the startup integrity walk) call at loop + * boundaries. Pinned directly against this class with a manual clock -- never Date.now() + * -- both to prove the gating math itself and, indirectly, that it reads elapsed time via + * this.clock.now(): if it read Date.now() instead, the huge (real, wall-clock) gap + * between this manual clock's fixed epoch and the actual current time would make it + * renew unconditionally, and the "skip within the window" assertion below would fail. */ +function grFix1_maybeRenewIsTimeGatedByTheClaimsOwnClock() { + const root = freshRoot("maybe-renew-gate"); + const clock = createManualClock(); + const w = newClaim(root, { instanceId: "7".repeat(32), clock }); + w.acquire(); + const afterAcquire = w.status(); + + // Advancing LESS than heartbeatMs: maybeRenew() must be a no-op. + clock.advance(HEARTBEAT_MS - 1); + w.maybeRenew(); + const stillUnrenewed = w.status(); + check("maybe-renew-skips-within-the-heartbeat-window", + stillUnrenewed.renewed_at === afterAcquire.renewed_at, + `expected renewed_at to stay ${afterAcquire.renewed_at}, got ${stillUnrenewed.renewed_at}`); + + // Advancing past heartbeatMs: maybeRenew() must actually renew, using the claim's own + // clock for the new renewed_at value. + clock.advance(2); + w.maybeRenew(); + const renewed = w.status(); + check("maybe-renew-renews-once-the-heartbeat-window-has-elapsed", + renewed.renewed_at === clock.now(), + `expected renewed_at to be bumped to ${clock.now()}, got ${renewed.renewed_at}`); + + // Immediately calling it again must be a no-op again -- the SAME instance-level + // _lastRenewAt is shared across every call site, not a fresh window per call. + const beforeSecond = w.status().renewed_at; + w.maybeRenew(); + check("maybe-renew-does-not-renew-twice-within-the-same-window", + w.status().renewed_at === beforeSecond, + `expected renewed_at to stay ${beforeSecond}, got ${w.status().renewed_at}`); +} + +function grFix1_maybeRenewPropagatesALostClaimJustLikeRenew() { + const root = freshRoot("maybe-renew-lost"); + const clock = createManualClock(); + const w = newClaim(root, { instanceId: "9".repeat(32), clock }); + w.acquire(); + clock.advance(HEARTBEAT_MS + 1); + fs.unlinkSync(w.path); // simulate an out-of-band loss + let thrown = null; + try { w.maybeRenew(); } catch (error) { thrown = error; } + check("maybe-renew-propagates-writer-claim-lost-once-the-gate-is-due", + Boolean(thrown) && thrown.code === "WRITER_CLAIM_LOST", thrown && thrown.code); +} + +/* Round-1 fix-plan item 1: previously only `error.code === "WRITER_CLAIM_LOST"` halted the + * background heartbeat -- ANY other renew() failure (a raw fs error, or CORRUPT_CLAIM) + * fell through silently, leaving the timer running and only `_lastHeartbeatError` set. + * Lease ownership is process-global: a renewal failure for ANY reason means this process + * can no longer be certain it is the sole attesting writer, so every failure must halt. */ +async function grFix1_heartbeatHaltsOnAnyRenewFailureNotOnlyClaimLost() { + const root = freshRoot("cr-heartbeat-non-lost-failure"); + const clock = createManualClock(); + let lostError = null; + const w = newClaim(root, { + instanceId: "a".repeat(32), clock, heartbeatMs: 20, + onClaimLost: (error) => { lostError = error; }, + }); + w.acquire(); + const realRenew = w.renew.bind(w); + w.renew = () => { + const error = new Error("simulated disk I/O error"); + error.code = "EIO"; + throw error; + }; + w.startHeartbeat(); + await waitUntil(() => lostError !== null, 2000); + w.stopHeartbeat(); + w.renew = realRenew; + + check("heartbeat-halts-on-a-non-writer-claim-lost-renew-failure", + Boolean(lostError) && lostError.code === "EIO", + `expected the heartbeat to halt and report the raw EIO error, got ${lostError && lostError.code}`); + check("heartbeat-clears-token-on-any-renew-failure", w._claimToken === null, + "the in-memory claim token must be cleared on ANY renew failure, not only WRITER_CLAIM_LOST"); + check("heartbeat-stops-timer-on-any-renew-failure", w._timer === null, + "the heartbeat timer must stop itself on ANY renew failure"); +} + function crCodex_renewToctouDoesNotClobberContender() { /* [inferring] SKIPPED on win32: this technique unlinks owner.path while renew()'s own * fd (fs.openSync(this.path, "r+"), held open across the whole check-then-write per @@ -744,6 +832,9 @@ async function main() { crCodex_clockInstantValidated(); crCodex_releaseKeepsTokenOnThrow(); await crCodex_heartbeatSurfacesClaimLoss(); + grFix1_maybeRenewIsTimeGatedByTheClaimsOwnClock(); + grFix1_maybeRenewPropagatesALostClaimJustLikeRenew(); + await grFix1_heartbeatHaltsOnAnyRenewFailureNotOnlyClaimLost(); crCodex_renewToctouDoesNotClobberContender(); crCodex_unlinkIfTokenDoesNotClobberSwappedFile();