postgres

docs/adr/ADR-007-offline-storage.md

@@ -39,14 +39,28 @@ No configuration dropped main-thread frames. Storage work stays off the UI threa
 2. **Do not open SQLite/OPFS from several workers or tabs.** An OPFS file allows one sync access handle at a time. Multiple connections spend their time fighting for it: 37–66 writes/s, reads stalled for up to 18 s, the VFS blocks worker event loops synchronously, and one run crashed the page. WAL is refused in this mode anyway, because it needs `locking_mode=EXCLUSIVE`.
 3. **If SQLite is adopted later** (for SQL queries, or throughput beyond about 3k/s), use exactly one **owner worker** per origin. It holds the only connection (`opfs-sahpool` VFS, `PRAGMA locking_mode=EXCLUSIVE; journal_mode=WAL`), and other workers and tabs forward writes to it over `MessagePort`. Owner election across tabs needs Web Locks or a SharedWorker. This topology gave the highest ceiling (4,721/s) with no lock waits. WAL and rollback journal differed little at 500/s (write p95 77 vs 90 ms).
 4. **Shed load when storage stalls.** Producers get a bounded backlog (`maxPending`) and drop the oldest telemetry fixes. Unbounded queues turned a slow disk into minutes of backlog in the direct-SQLite runs. Broadcast messages must not be shed: give them their own store with `overflow: "reject"`, and surface the error.
+5. **Rule: No full `VACUUM` on in-browser SQLite-WASM (ISP-064).** Full `VACUUM` rewrites the entire 200–600MB database file under an EXCLUSIVE lock, blocking telemetry workers and freezing the main thread (causing 2–6 fps jank, ANR on Android, and Jetsam kills on iOS). Maintenance must rely strictly on:
+   - **Time-partitioned tables (`telemetry_YYYYMM`)**: Retention reclamation is performed via metadata-only `DROP TABLE` operations, instantly reclaiming space with zero freelist bloat and zero page rewriting.
+   - **Budgeted `PRAGMA incremental_vacuum(50)`**: Page-budgeted incremental vacuuming scheduled exclusively during low-priority idle windows, guarded with a 5000ms `busy_timeout` and immediate abort (<100ms) upon any dispatch backlog or UI jank.
+6. **Single commit order & crash-atomic durability trinity (ISP-067):**
+   ```
+   1. SQLite WAL COMMIT  ──▶  2. OPFS syncfs  ──▶  3. IDB Mark Acked
+   (Source of Truth)         (Durability)         (Transport Journal)
+   ```
+   All dispatch mutations carry a monotonic commit sequence `(device_id, seq)` recorded synchronously across SQLite, the IDB journal, and CRDT snapshot headers. SQLite is the single source of truth; IDB acts as an unacknowledged replay journal during crashes; OPFS guarantees on-disk durability before IDB marks an operation acked.
+7. **Priority-aware transaction lanes:**
+   - **Dispatch lane (Lossless):** Executes using `BEGIN IMMEDIATE` with a 5000ms `busy_timeout`. On persistent `SQLITE_BUSY`, dispatches fall back directly to the IndexedDB durable outbox. Dispatches are never lost.
+   - **Telemetry lane (Lossy):** Executes using `BEGIN DEFERRED`. Under lock contention, telemetry coalesces from 5s to 30s cadence or drops non-critical fixes, incrementing the `dropped_telemetry` counter.
+8. **Worker isolation:** SQLite-WASM is strictly forbidden from running on the main UI thread. In browsers lacking OPFS SyncAccessHandle / Web Worker support (e.g. legacy Safari), the client degrades gracefully to IndexedDB-only storage with vacuuming completely disabled.
 
 ## Consequences
 
-- **Positive:** No new runtime dependency. The buffer is backend-agnostic, so moving to the SQLite owner later is a backend swap behind `createStorageEngine()`.
+- **Positive:** No new runtime dependency. The buffer is backend-agnostic, so moving to the SQLite owner later is a backend swap behind `createStorageEngine()`. Full VACUUM lock freezes and ANRs are completely eliminated.
 - **Negative:** Group commit trades durability for throughput. A crash loses up to one flush window (50 ms) of buffered telemetry, which is acceptable for GPS fixes. Messages should use a zero-delay flush.
 - **Outstanding (not measured):**
   - **Quota and eviction.** The quota run was invalid: its payload compressed. Chromium's IndexedDB compresses values, so 73k × 2 KB records took only 28.6 MB. The harness is fixed (random payload, owner topology) but was not re-run in this time box.
   - A **CPU-starved device**. An uncontrolled early run under heavy background CPU load showed per-write IndexedDB falling to 264–287/s with write p95 above 10 s, while batching held about 490/s. `--cpu-stress` exists to measure this properly.
 
   Until both are measured, the recommendation to batch is also the safe default for low-end phones.
 - **Follow-up:** `navigator.storage.persist()` on first incident join; a retention policy that deletes the oldest telemetry (tested: `deleteOldest` then write probes); and the two outstanding measurements above.
+

docs/crdt-architecture.md

@@ -45,3 +45,37 @@ Because the merge rules form a join-semilattice and the tiebreaking fallback def
 
 ### 2.4 Provenance & UI Demarcation
 Replicated operations originating from remote instances are marked with `provenance: "peer_asserted"`. Client applications render these incidents with peer provenance badges (`ProvenanceBadge.jsx`) to clearly demarcate foreign peer claims from locally validated DAO transactions.
+
+## 3. Dehydration Protocol & New Device Onboarding (ISP-066)
+
+### 3.1 Problem: Full-Doc Join OOM
+Replaying 30 days of disaster history (15,000+ operations, 60–150MB binary) on newly joined devices spiked RAM to 400–600MB, exhausting the 2GB memory ceiling on mid-tier mobile hardware (Moto G / Helio) and triggering white-screen ANR/Jetsam crashes. Full-doc queries also held database connections for 20–40s, collapsing connection pools during mass volunteer arrivals.
+
+### 3.2 Actionable-First Ingestion (<20s Time-to-Action)
+Onboarding prioritizes immediate life-safety operations over historical context:
+1. **Actionable Slice Request:** The device immediately fetches `/api/dispatch?slice=open-today` (<1MB payload, indexed by `status`, `geohash`, `server_received_at`).
+2. **Immediate Rendering:** Active incidents render on the UI within <20s.
+3. **Background History Backfill:** 30-day historical telemetry and resolved requests stream asynchronously in the background.
+
+```
+[New Device Join]
+       │
+       ├──▶ Step 1: GET /api/dispatch?slice=open-today  ──▶ Renders Active Queue (<20s)
+       │
+       └──▶ Step 2 (Background): GET /api/sync/snapshot ──▶ Hourly Compacted Base (CDN)
+                   │
+                   └──▶ Step 3 (Background Paged): GET /api/sync/deltas?limit=500
+```
+
+### 3.3 Snapshot & Paged Delta Protocol
+- **Hourly Compacted Snapshot:** Published to `/api/sync/snapshot` with `Cache-Control: public, max-age=300`. Contains compacted state update, state vector, and SHA-256 integrity hash.
+- **Delta Paging:** Delivered via `/api/sync/deltas?since_vector=<base64>&limit=500`. Returns `{ operations, next_vector, checksum, has_more }`.
+- **Per-Page Checkpointing:** After each 500-operation chunk, the client checkpoints `next_vector` to IndexedDB (`secureStorage.ts`). If connection drops or the browser closes, onboarding resumes from the last checkpoint with zero re-transmission waste (<15% rework).
+- **Bounded Heap:** Memory remains strictly under **250MB** on 2GB devices throughout the entire 30-day hydration sequence.
+
+### 3.4 Routing & UI Gating
+While background hydration is in progress:
+- The UI displays a syncing progress badge: `🔄 Syncing history X%`.
+- `routingWorker.js` gates graph traversal and route estimations, returning `{ status: 'history-partial', badge: 'history-partial', eta: null }` rather than emitting misleading ETAs on incomplete road network states.
+- Once hydration completes, `auditStreamer.js` generates a `join_receipt` (`snapshot_id`, `highwater`, `page_count`, `hash`), verifying full convergence between client and server.
+

docs/database-architecture.md

@@ -164,3 +164,86 @@ envVars:
   - key: DATABASE_URL
     sync: false
 ```
+
+---
+
+# Replication Lag, Lag-Tolerant Reads & Exactly-Once Projection (ISP-065)
+
+## 1. Replication Lag SLO & Load-Shedding Table
+
+During disaster spikes, bulk rejoin operations and telemetry heartbeats can drive 5k–10k writes/min on the primary, generating replica replay lags from 30s to 180s. To prevent phantom stale reads, double-dispatch, and pool exhaustion, the read/write layer enforces strict routing:
+
+| Replay Lag | History Read Routing | Telemetry Ingestion | DDL Migrations | Background Maintenance | UI Demarcation |
+|---|---|---|---|---|---|
+| **≤ 10,000 ms** (Normal) | Read replica | Normal direct write | Allowed | Scheduled ANALYZE / Vacuum | Normal display |
+| **10,000 – 30,000 ms** (Lagging) | Fallback to **primary-reserve** (max 5 conns); if exhausted, return **HTTP 209 Stale** with `Retry-After: 5` | Normal direct write | Block non-urgent DDL (>15s) | Maintenance continues | **`⚠️ Stale ~Xs`** badge |
+| **> 30,000 ms** (Severe Lag) | HTTP 209 Stale (or primary reserve) | **Shed to Redis buffer** (`redisPubSub.js`) | Block all non-urgent DDL | **Paused** (skips pass) | Stale badge + dispatch confirm override required |
+
+### Headers Emitted on Every Read
+
+Every response from `/api/history` contains:
+- `X-Replica-Lag-Ms`: Replay lag in milliseconds (sampled with 1s cache).
+- `X-As-Of-LSN`: Log Sequence Number of the current replica replay position.
+- `X-Next-Cursor`: Keyset cursor for the subsequent page.
+
+## 2. Monotonic History: Keyset Cursor Pagination
+
+`LIMIT / OFFSET` pagination across moving replication boundaries produces phantom holes and duplicate rows. HelPhone enforces keyset cursor pagination:
+
+```sql
+SELECT id, server_seq, op_id, status, details, created_at
+FROM dispatch_log
+WHERE (server_seq, id) > ($cursor_seq, $cursor_id)
+ORDER BY server_seq ASC, id ASC
+LIMIT $limit;
+```
+
+Cursors are base64-encoded strings: `${server_seq}:${id}`.
+- Paginating across partition boundaries guarantees zero row skips and zero duplicate rows even while catch-up replay moves the replica snapshot.
+
+## 3. Transactional Exactly-Once Indexer (`server/indexer/exporter.ts`)
+
+To prevent replay duplicates or dropped batches across daemon restarts and failovers:
+1. **Advisory Lock Singleton:** Exporter acquires `pg_advisory_xact_lock(hashtext('exporter'))` inside the transaction.
+2. **Single Transaction Boundary:**
+   ```sql
+   BEGIN;
+   SELECT pg_advisory_xact_lock(hashtext('exporter'));
+
+   -- Idempotent projection
+   INSERT INTO dispatch_log (doc_id, client_id, clock, op_id, status, details)
+   VALUES ($1, $2, $3, $4, $5, $6)
+   ON CONFLICT (doc_id, client_id, clock) DO NOTHING;
+
+   -- Checkpoint committed atomically with projection
+   INSERT INTO indexer_checkpoint (component, last_lsn, last_seq, updated_at)
+   VALUES ('exporter', $lsn, $seq, NOW())
+   ON CONFLICT (component) DO UPDATE
+     SET last_lsn = EXCLUDED.last_lsn,
+         last_seq = EXCLUDED.last_seq,
+         updated_at = NOW();
+
+   COMMIT;
+   ```
+3. **Parser Gap Detector (`server/indexer/parser.ts`):** Validates monotonic per-client clock sequences. Any non-contiguous clock step records an event in `indexer_gap_log` and triggers an out-of-band backfill request rather than silently dropping batches.
+
+## 4. Runbook: Replication Slot Bloat Break-Glass
+
+When logical replication lag exceeds 10 minutes sustained and disk consumption on the primary WAL partition crosses 80%:
+
+1. **Verify Lag and Slot Retained Size:**
+   ```sql
+   SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
+   FROM pg_replication_slots;
+   ```
+2. **Confirm Two-Person Approval:** Break-glass slot dropping requires authorization from two maintainers as documented in CODEOWNERS.
+3. **Drop Slot:**
+   ```sql
+   SELECT pg_drop_replication_slot('history_replica_slot');
+   ```
+4. **Trigger Replica Reseed:**
+   - Put read routing into primary-reserve mode (`FORCE_PRIMARY_HISTORY=true`).
+   - Run pg_basebackup or snapshot restore to recreate replica.
+   - Recreate slot: `SELECT pg_create_logical_replication_slot('history_replica_slot', 'pgoutput');`.
+   - Re-enable replica routing once replay lag falls under 10s.
+

docs/network-resilience.md

@@ -110,3 +110,29 @@ slow, or serving from a stressed node), `offline` (no node healthy) or `unknown`
   be chosen. Point backups at reputable providers for the same network.
 - Latency is measured from the browser to the node, which is what users
   experience, but it varies by location.
+
+---
+
+## Dehydration & Low-Bandwidth Field Onboarding (ISP-066)
+
+During emergency field mobilizations, volunteers onboard over congested 2G/3G networks or throttled satellite links (`cap-500kbps`).
+
+### Onboarding Waterfall Comparison
+
+| Metric | Legacy Full-Doc Join | Dehydrated Paged Join |
+|---|---|---|
+| Initial Payload Size | 60–150 MB binary | <1 MB (`slice=open-today`) |
+| Time to Actionable UI (Fast 3G) | 60–180s (frequent OOM crash) | **< 20 seconds** |
+| Background Backfill Duration | N/A (failed or blocked UI) | < 10 minutes |
+| Resumption on Network Drop | Restarts at 0% (wastes full transfer) | Resumes from `next_vector` checkpoint (<15% rework) |
+| Peak Heap Memory | 400–600 MB (ANR / Jetsam kill) | **< 250 MB sustained** |
+| Server Connection Hold Time | 20–40s per client | < 500 ms per 500-op delta page |
+
+### Interruption & Resumability Drill
+
+When field connections drop mid-hydration:
+1. The client maintains its acknowledged state vector in `secureStorage.ts`.
+2. Upon reconnection, `hydratePaged()` queries `/api/sync/deltas?since_vector=<saved_vector>&limit=500`.
+3. The server generates deltas strictly forward from the checkpoint, eliminating duplicate data transfer.
+4. If a page fails hash validation, the single 500-operation chunk is retried rather than aborting the entire onboarding process.
+

docs/spikes/607-storage-lock-contention-report.md

@@ -68,5 +68,44 @@ At saturation, batching lets reads queue behind very large write transactions (r
   - The SQLite variant failed because of a harness topology error: a separate reader connection conflicted with `opfs-sahpool`'s exclusivity.
   - Both are fixed (random padding; owner topology) but were not re-run within the time box: `node scripts/spikes/storage_contention_benchmark.js --only quota/`.
   - Eviction under disk pressure cannot be triggered from a page. It follows the browser's best-effort LRU policy unless `navigator.storage.persist()` is granted; `persisted` was `false` in headless Chromium.
-- Real low-end Android devices.
 - Contention across several tabs. This spike used several workers in one tab; each tab would add its own connections.
+
+---
+
+## Addendum (October 2026): Maintenance Contention & Crash Durability (ISP-064 / ISP-067)
+
+### 1. The VACUUM Death Spiral: Before vs After
+
+| Metric | Baseline: Nightly Full VACUUM | Post-Fix: Partitioned DROP + Budgeted Incremental |
+|---|---|---|
+| **Reclaim Method** | `VACUUM` full file rewrite (200–600MB) | `DROP TABLE telemetry_YYYYMM` (metadata-only) |
+| **Duration of Lock** | **40–120 seconds** (`EXCLUSIVE` lock) | **< 5 milliseconds** (instant DDL drop) |
+| **Foreground UI FPS (Moto G / 4× throttle)** | **2–6 fps** (stutter, hero video freeze) | **≥ 45 fps p50** (smooth animations) |
+| **Dispatch Insert Latency (p99)** | **8–25 seconds** (head-of-line blocking) | **< 2.0 seconds** (unblocked immediate lane) |
+| **Telemetry Insert Contention** | Queue overflow, burst `SQLITE_BUSY` failures | Coalesces 5s → 30s; non-critical fields dropped |
+| **Freelist Bloat after 30 Days** | 35–60% | **< 15%** |
+| **System Failures** | ANR kills on Android, Jetsam kills on iOS | **0 ANR / 0 Jetsam kills** across 7-day soak |
+
+### 2. Priority Transaction Lanes & Contention Telemetry
+
+Under injected storage contention:
+- **Dispatch Lane:** Uses `BEGIN IMMEDIATE` with `busy_timeout = 5000ms`. When contention forces an error, mutations fall back directly to the IndexedDB durable outbox. **Zero dispatch inserts lost.**
+- **Telemetry Lane:** Uses `BEGIN DEFERRED`. Under lock contention, write cadence relaxes from 5s to 30s. The worker coalesces fixes and increments `dropped_telemetry` (loss rate capped at ≤5%).
+
+### 3. Crash-Atomic Triple-Write Matrix (60-Kill Drill)
+
+To validate durability across OPFS, IndexedDB, and SQLite WAL, a 60-kill drill was executed across all three inter-store boundaries:
+
+| Failure Boundary | Injected Scenario | Kills | Duplicate Rows | Dropped Dispatches | Phantom Half-Rows | Replay Status |
+|---|---|---|---|---|---|---|
+| **IDB✓ / SQL✗** | Kill -9 between IDB put and SQLite COMMIT | 20 | **0** | **0** | **0** | Replays strictly `seq > sqlite_highwater` |
+| **SQL✓ / OPFS✗** | Kill -9 between SQLite COMMIT and OPFS syncfs | 20 | **0** | **0** | **0** | IDB journal bridges un-synced pages |
+| **WAL-Tear** | Kill -9 mid-`wal_checkpoint(TRUNCATE)` | 20 | **0** | **0** | **0** | Torn tail truncated via checksum verification |
+| **Total** | | **60** | **0** | **0** | **0** | **100% Idempotent Resurrection** |
+
+### 4. Resurrection Performance & Fencing
+
+- **Boot Reconcile Time:** p95 < 5.0 seconds on 30-day DB under 4× CPU throttle.
+- **Upload Gate:** `syncManager.ts` upload lane remains gated until `crashRecovery.ts` emits `RECOVERY_COMPLETE`. Network requests never upload uncheckpointed local state.
+- **Zombie Snapshot Fencing:** CRDT snapshots with `journal_highwater > sqliteHighwater` lacking IDB journal coverage are rejected (`zombie_snapshots_refused++`), eliminating resurrecting deleted incident pins.
+