Gateway: crash-recovery WAL + idempotency fencing (Option C, Codex PR #29 Findings 1+2)

.gitattributes

@@ -48,3 +48,18 @@
 # entire value is that one specific byte survives.
 scripts/ci-check-pr-separation.js binary
 tests/watcher/sonnet/run-tests.js binary
+
+# scripts/gateway/chain.js also carries a literal NUL byte -- the hash-chain's own
+# field-separator sentinel inside a .join("\0") call, so no legitimate field value
+# can collide across two different joins. Unlike the two fixtures above, this
+# file's whole point is that its CODE gets reviewed, so it must stay diffable.
+# `binary` (the directive used for those two files) is the wrong tool here: it is
+# shorthand for `-diff -merge -text`, so copying it verbatim would leave chain.js
+# permanently in the exact same "Binary files ... differ" state the plain
+# NUL-byte auto-detection already produces today -- it would not fix the
+# reviewability problem, it would lock it in. `diff` (bare, positive) is narrower:
+# it does not touch -text/-merge, it only tells git's diff engine to treat this
+# one path as text for diff purposes despite the byte that trips the binary
+# heuristic, so a real line-based diff renders on GitHub. Verified locally via
+# `git check-attr` and `git diff` before this attribute was added and after.
+scripts/gateway/chain.js diff

docs/contracts/wal-append-failure-semantics.md

@@ -0,0 +1,147 @@
+# C1 — WAL Append-Failure Semantics
+
+> **Status:** binding design contract for PR #33's crash-recovery/idempotency follow-up
+> work. Referenced by commit 1 (lease keepalive), commit 5 (sampling `CALL_RESULT`
+> fail-closed), and any future code that calls `recovery.appendWalEvent` or reads a
+> connection WAL. Locked by the 2026-09-15 fix-round panel (see
+> `claude/graphsmith-fix-round-plan-round1-2026-09-15.md` in the GraphSmith project);
+> do not reopen items 1-5 below without a new decision record.
+
+This contract governs `scripts/gateway/recovery.js#appendWalEvent` (and its private
+helpers `appendDurableLine`/`writeFullySync`) and every caller of it. It exists
+because the original PR #33 follow-up plan modeled a WAL append failure as "nothing
+was written" and built retry/recovery logic on that assumption; three independent
+reviewers (Fable, GPT-6 Astra, Grok-4.6) found the assumption false and cross-agreed
+on why. This document is the corrected model.
+
+## 1. A throw from `appendWalEvent` means UNKNOWN persistence, not absence
+
+`appendWalEvent` is `ensureDir` → `openSync(path, "a", 0o600)` → `writeFullySync`
+(loops `writeSync` until every byte is written) → `fsyncSync` → `closeSync` →
+`fsyncDir`. A caller that catches a throw from this call **must not** assume the
+event was not durably recorded. Exactly three outcomes are possible, and they are
+indistinguishable from the caller's stack frame alone:
+
+1. **Nothing was written.** The `openSync` or the first `writeSync` failed before
+   any bytes landed (e.g. `ENOSPC`, `EROFS` at open time).
+2. **A torn/partial line was written.** `writeFullySync`'s own truncate-on-stall
+   path (see its header comment in `recovery.js`) tries to undo a partial write by
+   `ftruncateSync`-ing back to the pre-append size, but that truncate is itself a
+   filesystem call and can fail (the exact failure mode this contract exists to
+   cover). A torn line can therefore remain on disk after the throw.
+3. **The line was written completely, and a later step failed.** `fsyncSync` (data)
+   or the caller's own subsequent `fsyncDir` can throw *after* `writeFullySync` has
+   already returned successfully and the complete line is sitting in the file
+   (possibly not yet durable across a real crash, but present and syntactically
+   valid to any reader that opens the file right now).
+
+Because outcome 3 is possible, **a "failed append" and "the event was not recorded"
+are not the same fact.** Any code that decides what to tell a downstream caller, or
+what to write into an in-memory session record, must say "not confirmed durable,"
+never "not recorded" or "discarded."
+
+## 2. On any append failure, the connection's WAL is poisoned in memory
+
+The moment `appendWalEvent` throws for a given `connectionId`, that connection's
+in-memory session state must be marked **poisoned** (a boolean or reason string on
+the session record is sufficient) and **every subsequent call to
+`appendWalEvent` for that same connection must be refused before it touches the
+filesystem** — the caller should treat the refusal exactly like a second append
+failure (see §3), without re-attempting the write.
+
+Rationale: per §1 outcome 2, the file may already end in an unterminated, torn JSON
+line. `readWalEvents`' replay contract (`recovery.js`, see its header comment) is
+"stop at the first line that fails to `JSON.parse`, keep everything before it" —
+which exists precisely because a crash can only tear the *last* line in flight.
+That contract silently assumes nothing is ever appended to the file again after a
+tear. If a second, unrelated append is allowed to proceed on a torn WAL, its bytes
+concatenate directly onto the torn line, and the result is a single malformed line
+that swallows both the original partial content and the new event — the new event
+is now unparseable too, and is lost to replay along with everything the caller
+believed it just durably recorded. Poisoning the connection converts "silently lose
+a real event" into "loudly refuse further writes on this connection," which is the
+correct fail-closed direction.
+
+The poison flag is in-memory only and connection-scoped. It is not written to disk,
+does not affect any other connection's WAL, and clears when the connection is
+closed/reopened (a new connection gets a fresh WAL file and a fresh poison state).
+
+## 3. No best-effort second append after a possible tear
+
+Do not attempt a second, best-effort `appendWalEvent` call to record an error
+result (or anything else) after the first append for that event already failed —
+neither immediately, nor later on the same poisoned connection. This reverses an
+earlier draft of the plan, which proposed a best-effort second append "for the
+error CALL_RESULT" so that in-memory state and on-disk replay state would agree.
+
+Two independent, cross-confirmed failure modes make that unsafe:
+
+- **Concatenation onto a torn line** (§2): the second append can land directly
+  after an unterminated line from the first attempt, corrupting both.
+- **Two conflicting terminal records for the same correlation key.** If the first
+  append actually completed (§1 outcome 3) and only a later fsync step threw, a
+  second append for the *same logical event* — even an "error" record instead of
+  the original "success" record — creates two `CALL_RESULT` lines on disk for one
+  correlation key: one recording success, one recording failure. Replay then has
+  two candidate terminal records for the same call, and which one "wins" was left
+  unspecified by the earlier draft. §4 defines that resolution rule now, but the
+  conflict itself is avoidable simply by never attempting the second write.
+
+The correct sequence when an append fails is: mark the connection poisoned (§2),
+record the caller-facing outcome (e.g. a JSON-RPC error to the downstream caller)
+and the in-memory session state, and stop. Do not touch the WAL again for that
+connection.
+
+## 4. Replay conflict rule for terminal records
+
+`readWalEvents` can, despite §2-3, still encounter two `CALL_RESULT` events for the
+same correlation key on legitimate historical WALs (e.g. a build that predates this
+contract, or any other path that ever produced a duplicate before this rule
+existed). Replay must resolve that deterministically:
+
+> **First complete `CALL_RESULT` per correlation key wins.** Every later
+> `CALL_RESULT` for the same correlation key is treated as a duplicate: it is not
+> applied to the replayed session state, and it is recorded as an anomaly (not
+> silently dropped and not silently preferred over the first one).
+
+"Complete" here means the record parsed successfully as JSON and passed its own
+shape check — i.e. it is one of the events `readWalEvents` returns at all (a torn
+line never reaches this rule because it stops the read before ever producing an
+event for that line, per its own stop-at-first-bad-line contract).
+
+This rule must be implemented once, in whatever function replays a connection's WAL
+into session state, and covered by a test that constructs exactly this state (two
+complete `CALL_RESULT` events for one correlation key) and asserts: the replayed
+session reflects the first one, the second is present in the anomaly list, and
+replay does not throw.
+
+## 5. `discarded_result_sha256`: exact attempted bytes, or omit the field
+
+When code records that a result could not be durably written (e.g. the sampling
+`CALL_RESULT` fail-closed path), it may attach a `discarded_result_sha256` field so
+a later auditor can correlate "the agent produced X" with "the gateway withheld X."
+That field is only trustworthy under one rule:
+
+> **Hash the exact serialized buffer that was attempted for the WAL append** (the
+> same `Buffer`/string passed to `writeFullySync`/`appendDurableLine`), **or omit
+> the field entirely.** Never hash a fresh `JSON.stringify(resultObject)` call made
+> after the fact for the sole purpose of populating this field.
+
+Rationale: `JSON.stringify` output depends on object key insertion order, which is
+not guaranteed to be stable across two separate serializations of "the same"
+logical object (a field added, reordered, or populated by a different code path on
+a retry can silently change the byte string while the object's meaning is
+unchanged). A digest computed from a *second*, independent serialization is
+therefore not a reliable commitment to what was actually attempted on disk — it can
+disagree with the real attempted bytes for reasons that have nothing to do with the
+result's content. If the exact attempted buffer is not available in scope where the
+digest is computed, the field must be left off the record rather than populated
+with a digest of something else. A missing field is honest; a digest of the wrong
+bytes is not.
+
+## Summary (for code comments)
+
+> Per `docs/contracts/wal-append-failure-semantics.md` (C1): a WAL append throw
+> means unknown persistence, not absence. Poison the connection on failure; never
+> retry. Replay keeps the first complete terminal record per correlation key.
+> `discarded_result_sha256` hashes the exact attempted buffer or is omitted.

docs/runbooks/gateway-chain-corruption-recovery.md

@@ -0,0 +1,145 @@
+# Runbook: gateway-session chain corruption recovery
+
+> Referenced from every `GATEWAY_CHAIN_STRUCTURAL_FAILURE` diagnostic this gateway
+> writes (`<state_dir>/gateway-startup-failure.json`) and from every
+> `GATEWAY_CHAIN_HEAD_DIVERGED` refusal it logs mid-run. Ships alongside round-1
+> fix-plan commit 4 (`chain.reconcileHead` + the classified append-time HEAD/tail
+> check) per Paul's 2026-09-15 startup-posture decision
+> (`claude/graphsmith-fix-round-plan-round1-2026-09-15.md`,
+> `docs/contracts/chain-validity.md` SS4): a hard-failed startup must always end in a
+> concrete next step, never just a stopped process. Read this top to bottom before
+> touching `chain.jsonl` or `HEAD.json` by hand -- every option below is destructive in
+> a different way, and picking the wrong one can destroy attested history that was
+> otherwise recoverable.
+
+## 1. Is this actually corruption, or just a lagging pointer?
+
+Not every divergence between `HEAD.json` and `chain.jsonl` needs this runbook. Two
+classes of divergence are **not** corruption and are already handled automatically,
+under the writer-claim, by this gateway itself:
+
+- **A single-step lagging `HEAD`** (the chain's own tail is exactly one entry ahead of
+  `HEAD`, and the link hash matches) is auto-repaired the moment the next session tries
+  to append, or at the next startup by `chain.reconcileHead`. This is the ordinary
+  signature of a crash between `chain.jsonl`'s own append and `HEAD.json`'s update. You
+  do not need this runbook for it -- just restart the gateway (or let the next session
+  close normally) and check `gateway.js status` afterward.
+- **Any size of lagging `HEAD`, found at startup**, where the chain independently
+  verifies end-to-end and `HEAD` names a genuine, hash-verified entry somewhere earlier
+  in that same chain, is repaired by `chain.reconcileHead` before the gateway ever
+  starts accepting sessions.
+
+If the gateway is refusing to start with a `GATEWAY_CHAIN_STRUCTURAL_FAILURE`
+diagnostic (check `<state_dir>/gateway-startup-failure.json`), or a running gateway is
+refusing every session close with `GATEWAY_CHAIN_HEAD_DIVERGED` /
+`GATEWAY_CHAIN_INTEGRITY_FAILED`, you are past the automatic cases. Read the
+diagnostic's `message`/`reason` field first -- it names exactly which of the cases below
+you are in (a fork, a broken hash link, a sequence gap, an invalid genesis, HEAD ahead
+of the chain's own tail, or an unreadable/torn tail) before you pick a recovery option.
+
+## 2. Before doing anything else
+
+1. **Stop the gateway process** if it is somehow still running (it should already have
+   refused to start, or be about to exit after a runtime refusal -- confirm it is not
+   silently retrying).
+2. **Back up the entire `<state_dir>/gateway-sessions/` directory** (`chain.jsonl`,
+   `HEAD.json`, every `<bundle_id>.json`, and the `quarantine/` subdirectory if present)
+   to a separate location before touching anything. Every option below assumes you can
+   get back to exactly this state if you make a mistake.
+3. **Do not delete or hand-edit `chain.jsonl`/`HEAD.json` directly** until you have
+   picked one of the three options below and understand its consequences. A single
+   `sed`/manual JSON edit that produces a shape-valid-looking record is exactly how a
+   silent, undetected fork gets introduced.
+
+## 3. Option A -- restore from backup
+
+**Use this when** you have an independent, trusted backup of `<state_dir>/gateway-
+sessions/` taken before the corruption occurred (a nightly snapshot, a filesystem
+snapshot, etc.), and the corrupted window since that backup is small enough that losing
+the sessions recorded in it is acceptable.
+
+1. Confirm the backup itself verifies: run the chain-integrity check
+   (`checks/register-gateway-sessions.js`, or `gateway.js status` once the gateway can
+   start against a copy of the backup in a scratch `state_dir`) against the backup
+   **before** restoring it over the live directory.
+2. Stop the gateway (if not already stopped), replace the live `gateway-sessions/`
+   directory with the verified backup, and restart.
+3. Any bundles/WALs newer than the backup are now orphaned. Move them to a separate
+   `manual-review/` directory outside `state_dir` for a human to inspect -- do not
+   delete them; they may still be useful evidence even though they can no longer be
+   chain-appended automatically.
+
+**Cost:** every session sealed after the backup was taken is gone from the attested
+chain, even if it was otherwise legitimate.
+
+## 4. Option B -- truncate at the last known-good entry
+
+**Use this when** the failure is localized to the chain's own **tail** (the last one or
+two records) -- e.g. a torn/unparseable final record from a crash mid-write, or a
+tampered/malformed final entry -- and every entry before it verifies cleanly. This is
+the right choice when you do **not** have a recent backup but most of the chain's
+history is still trustworthy.
+
+1. Run the structural validator against `chain.jsonl` up to (but not including) the
+   suspect final record(s) to confirm everything before that point is genuinely intact
+   (hashes, links, sequence, genesis). Do not assume -- verify.
+2. Truncate `chain.jsonl` to end exactly after the last entry that verified. Use a tool
+   that truncates at a byte offset you have confirmed corresponds to a complete,
+   newline-terminated record -- never a text editor's own line-ending guess.
+3. Rebuild `HEAD.json` from that new, truncated tail (see the exact record shape in
+   Option C step 2 below).
+4. Move the bundle file(s) for any entries you truncated away into `manual-review/`
+   (outside `state_dir`) rather than deleting them.
+5. Restart the gateway. `chain.reconcileHead` will confirm the truncated chain verifies
+   and that `HEAD.json` matches its new tail before accepting any session.
+
+**Cost:** every entry from the first bad record onward is removed from the attested
+chain, whether or not each individual one was itself legitimate (a sequence gap is not
+repairable in place -- an entry that comes after a bad one can never be re-attached
+without also being re-verified from scratch).
+
+## 5. Option C -- rebuild `HEAD` from the validated chain tail
+
+**Use this when** `chain.jsonl` itself is fully intact and verifies end-to-end, and the
+**only** thing wrong is `HEAD.json` -- but the divergence is worse than the single-step
+or verified-ancestor cases this gateway already repairs automatically (for example,
+`HEAD.json` was manually edited to a bogus value, or points at a `seq` beyond the
+chain's own real tail, or was lost/replaced by something that does not correspond to any
+real entry in the chain at all).
+
+1. Run the structural validator against the **entire** `chain.jsonl` with `head`
+   omitted/ignored to confirm it independently verifies (correct genesis, contiguous
+   `seq`, every link hash and entry hash correct) end-to-end. If it does **not**
+   verify, stop -- you are actually in Option B or Option A's territory, not this one.
+2. Take the chain's own last entry (the highest `seq`, confirmed correct by step 1) and
+   write `HEAD.json` as exactly:
+   ```json
+   {
+     "schema_version": "1.0",
+     "seq": <that entry's seq>,
+     "bundle_id": "<that entry's bundle_id>",
+     "entry_sha256": "<that entry's entry_sha256>"
+   }
+   ```
+   Do not hand-construct any field other than by copying it verbatim from the verified
+   tail entry.
+3. Restart the gateway. `chain.reconcileHead` re-verifies the same thing you just did by
+   hand before accepting any session, so a mistake here fails startup again rather than
+   silently taking effect.
+
+**Cost:** none to the chain's own history -- this option only ever repairs the pointer,
+never the chain -- but it requires you to have correctly confirmed step 1's "fully
+intact, end-to-end" precondition; skipping that verification and rebuilding `HEAD` over
+a chain that does **not** actually verify silently converts a detectable corruption into
+an falsely-"healthy"-looking gateway.
+
+## 6. After recovery
+
+- Confirm `gateway.js status` reports `session_chain_integrity: { status: "verified" }`
+  before considering the incident closed.
+- Investigate *why* the corruption happened (disk full, a manual edit, a filesystem
+  fault, a bug) before it recurs -- none of the three options above address root cause.
+- If you used Option A or Option B, the sessions you removed from the attested chain are
+  gone from `SG-FR-7`'s guarantee even though their raw bundle/WAL files may still exist
+  in `manual-review/`; do not re-append them later by hand -- that would itself be
+  exactly the kind of undetectable, unattested mutation this chain exists to prevent.