docs(loop): cycle 5 consolidated — T07d, B-28, B-21, T12a

CHANGELOG.md

@@ -6,6 +6,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Fixed — a full rebuild interrupted part-way resumes, against its own tree (PRJ-06 T12a, #441)
+
+- A restart during a full rebuild used to pay for all of it again: the run created a fresh
+  checkpoint whenever `force_full` was set. A full run now continues a full checkpoint (an
+  incremental run still never continues a full one), and a resume is checked out at the
+  commit its checkpoint describes; if that commit is gone, the run starts from the top.
+
+### Changed — a multi-connection project's nightly gives each connection its own job (T07d, #438)
+
+- PRJ-07 S-08 remainder. The nightly sync ran every connection inside one job under one
+  7 200 s ceiling; one connection measures ~2 300 s on production, so four could not fit a
+  night. With more than one connection, each now runs as its own job with its own ceiling
+  (`DAILY_SYNC_CONNECTION_JOB_TIMEOUT_SECONDS`, 7200), one after another, and writes its
+  own row in the sync history. A one-connection project — every project in production
+  today — is unchanged.
+
+### Fixed — the sync history panel read a row the API had stopped returning (B-28, #437)
+
+- Since the history moved to `indexing_runs` the panel showed no status icon, "NaNd ago",
+  no error and no connection count, while both test suites stayed green: the panel's test
+  mocked the retired row. The API now also returns the run's own verdict (`outcome` — a
+  completed sync can be partial) and its steps, and one fixture pins the row for both
+  sides.
+
+### Fixed — a test's background work can no longer undo the next test's data (B-21, #439)
+
+- On SQLite every test session shares one connection, and background work a request
+  started outlived its test; a rollback from it could erase the next test's uncommitted
+  insert — the intermittent `StaleDataError` in `test_learnings_api`. The harness now
+  **waits** for that work before each test is torn down, and never cancels it: a first
+  version cancelled, and a statement cancelled mid-flight replaced the shared in-memory
+  database with an empty one (263 failures on CI). A lost schema now fails at the cause.
+
 ## [1.18.0] - 2026-09-26 - Money, tenancy and the nightly: what production and the audits found
 
 **Release summary.** 147 changelog sections since 1.17.0 (2026-08-31): PRJ-01…PRJ-04, PRJ-07,

docs/evidence/backlog.md

@@ -26,9 +26,10 @@ this board holds what falls outside a named project or arrives mid-run.
 | B-18 | P2 | **UX verification rot**: 106 of 154 scenarios verified > 30 days ago (94 on 2026-07-19) against a ratchet of 95, so `make ux-status` cannot be committed until a batch is re-audited. Re-audit ≥ 12, oldest first, then restamp after staging. | `docs/audits/2026-09-23-recent-work-audit.md` §4 | done — #427: 14 oldest scenarios re-audited (7 PASS, 7 PARTIAL, 0 FAIL), six defects fixed, stale-verified count 94 → 92; SCN-022's open gap is B-26 |
 | B-19 | P2 | **A process-wide `_external_rebroadcast` flag drops local trace events** (F-T1, pre-existing): held across awaits in `workflow_tracker.py:473-488`, so a chat `pipeline_start` arriving on web during a worker rebroadcast loses the whole trace. | `docs/audits/2026-09-23-recent-work-audit.md` §3.5 | done — #426 (`_RELAYING` ContextVar) |
 | B-20 | P2 | **Cut release 1.18.0.** `[Unreleased]` carries ~4 350 lines since 1.17.0 (2026-08-31), PRJ-01…PRJ-10 included; `pyproject.toml` and `package.json` still say 1.17.0. | `docs/audits/2026-09-23-recent-work-audit.md` §6 | done — 1.18.0 cut 2026-09-26 (147 sections; `pyproject.toml`, `package.json`, tag `v1.18.0`) |
-| B-21 | P3 | **Test hygiene** (timeouts + ratchet prose done in #418; still open: the `test_learnings_api` StaleDataError — **not only when unit and integration share a process**: it failed #425's integration-only CI run (`test_update_learning_lesson`, `UPDATE … expected 1 row, 0 matched`) and passed on rerun and in a local 698/698 run, and a one-off `test_auth_cascade` FK flake on #418's first CI run): Vitest 5 s timeouts under load (7 in one full run; the same GA4 form test flaked `main` on 2026-09-18); `test_learnings_api` `StaleDataError` only when unit and integration share a process; invalid `# noqa` at `test_suppression_debt_ratchet.py:442`; no `pytest-xdist` in `[dev]` (one-process suite 29.5 min). | `docs/audits/2026-09-23-recent-work-audit.md` §4 | todo |
-| B-22 | P3 | **Web memory climbs in steps with no chat traffic.** After the v445 boot 311-317 MB; steps of +50…85 MB at 14:25, 14:42, 15:08, 15:23 UTC that never return, reset only by a restart (so v444's ~483 MB after 17 h is accumulated steps). The router shows long-lived MCP streams — `GET /mcp/mcp` held 817 s and 2 434 s — from an external agent. Hypothesis, not a finding: per-session state held by the mounted MCP server. Next: count live MCP sessions against the step times. | `docs/audits/2026-09-23-recent-work-audit.md` §6 row 15a | todo |
+| B-21 | P3 | **Test hygiene** (timeouts + ratchet prose done in #418; still open: the `test_learnings_api` StaleDataError — **not only when unit and integration share a process**: it failed #425's integration-only CI run (`test_update_learning_lesson`, `UPDATE … expected 1 row, 0 matched`) and passed on rerun and in a local 698/698 run, and a one-off `test_auth_cascade` FK flake on #418's first CI run): Vitest 5 s timeouts under load (7 in one full run; the same GA4 form test flaked `main` on 2026-09-18); `test_learnings_api` `StaleDataError` only when unit and integration share a process; invalid `# noqa` at `test_suppression_debt_ratchet.py:442`; no `pytest-xdist` in `[dev]` (one-process suite 29.5 min). | `docs/audits/2026-09-23-recent-work-audit.md` §4 | done — the `test_learnings_api` StaleDataError: background work outliving its test on the shared SQLite connection, drained in #439. The invalid `# noqa` was fixed in #418 (`d56068c5`). Still open from this row: no `pytest-xdist` in `[dev]`, so the one-process suite takes ~6 min unit + ~4.5 min integration |
+| B-22 | P3 | **Web memory climbs in steps with no chat traffic.** After the v445 boot 311-317 MB; steps of +50…85 MB at 14:25, 14:42, 15:08, 15:23 UTC that never return, reset only by a restart (so v444's ~483 MB after 17 h is accumulated steps). The router shows long-lived MCP streams — `GET /mcp/mcp` held 817 s and 2 434 s — from an external agent. Hypothesis, not a finding: per-session state held by the mounted MCP server. Next: count live MCP sessions against the step times. **Measured again 2026-09-26 on v461:** 310.6 → 339.4 MB at 13:44 UTC, ~64 min after boot, with **no request at all** in the window (4 router lines in 1 500, all `/api/health`) and no app log line — so the step is internal work, not traffic, and the MCP-stream hypothesis does not explain this one. Web sits at ~1/3 of its 1 GiB, so this stays P3; the next step is an opt-in `tracemalloc` diff logged on an interval, enabled for a few hours, rather than more guessing. | `docs/audits/2026-09-23-recent-work-audit.md` §6 row 15a| todo |
 | B-24 | P2 | **A resolved error that recurred stayed resolved** (F-E1). Fixed in #412; then the 23 stale open rows were resolved (O-3). | found 2026-09-23 while triaging | done — #412 + O-3 |
 | B-25 | P1 | **The router's completion cap was spent on reasoning** — 1 of 90 routings came back empty and silently took the default route. Found by the routing eval; cap 512 → 1024 with the measurement in a test. | T08a live run 2026-09-23 | done — #421 |
 | B-26 | P3 | **A capped member list reads as the whole team** (SCN-022). `GET /api/invites/{project_id}/members` marks a capped response with `X-Result-Capped` (F-PROJ-13) and `InviteManager` never reads it, so the header says "Members (N)" as if complete. Needs the request helper to expose a response header. | B-18 re-audit 2026-09-23 | done — #431 (CORS exposes the headers; "Members (N of TOTAL)") |
 | B-27 | P2 | **Second UX re-audit batch** (12 scenarios, 4 PARTIAL): FE-06's sweep never reached SSE-tracked tasks and called unknown outcomes `failed`; every 403 reason was replaced by one sentence; a second "Continue analysis" lost the question and a pipeline cutoff carried no context; readiness offered "Run" for a repository index in flight; `/verify-email` never restored the session. | re-audit 2026-09-26 | done — #432 |
+| B-28 | P2 | **The sync-history panel read the retired KnowledgeSyncRun row**: no status icon, "NaNd ago", no error, no connection count — while both suites were green, because the panel's test mocked the retired shape. | found scoping T07d, 2026-09-26 | done — #437 |

docs/evidence/loop-queue.md

@@ -40,7 +40,9 @@ Standing rules the loop inherits (from `docs/evidence/retro.md`):
 | T06b | deliver | PRJ-10 residue: F-G1, F-G2, F-G3, F-G4, F-G5 — before the first real GA4 connection | `docs/audits/2026-09-23-recent-work-audit.md` §3.4 | merged #411 — fixture-level only; no GA4 connection in production | fixture-level only — no GA4 connection exists in production (same limit as T06) |
 | T07 | deliver | PRJ-07 — scheduling and recovery that work in every deployment | audit §PRJ-07 | done — #415 (S-04, S-13) + #416 (S-07); runtime-verified at V3 on v452 | |
 | T07c | deliver | PRJ-07 S-08/S-09 — repo-only projects nightly, the index→sync chain and live tables for every connection, a bounded live-table read | audit §PRJ-07 | merged #417 — latent on production (no such project shape) | `tests/unit/services/test_every_project_shape_is_served.py` |
-| T07d | deliver | PRJ-07 S-08 remainder — per-connection `db_index` as child jobs with their own ceilings, so four connections do not share one 7 200 s night | audit §PRJ-07 | todo | |
+| T07d | deliver | PRJ-07 S-08 remainder — per-connection `db_index` as child jobs with their own ceilings, so four connections do not share one 7 200 s night | audit §PRJ-07 | done — #438 | chain of `run_daily_connection_sync` jobs (own ceiling 7200 s), one `daily_sync` row per connection; one-connection projects unchanged (production's only shape, so the new path has no production run yet); tests: chain queued, single stays inline, fallback inline, a failing link still passes the night on |
+| B-28 | deliver | The sync-history panel read the retired KnowledgeSyncRun row | found scoping T07d | done — #437 | one fixture pins the row for both suites; SCN-064 FAIL → fixed |
+| B-21 | deliver | The intermittent `test_learnings_api` StaleDataError | backlog B-21 | done — #439 | leftover background tasks on the shared SQLite connection; drained in `db_session` teardown; guard test red without it |
 | V3 | verify | B-16, T05b, T04b, T06b, F-E1, B-17, O-1, T07, T07c, T08a landed | — | done (runtime) | v452, one-off dyno 2026-09-23: 1045 not transient, 2013 transient; `$DBPASS` as argument refused; `_tunnel_was_swept` present; GA4 `TransportError` not auth; prune margin 2; F-E1 reopen present; `table_analysis` first param `table_name`; SQLite date literal; repo-index slot 1 inside the task; DST hours [2, 3]; `IN_PROCESS_JOBS` = the six worker jobs; in-process mode OFF on production (Redis); live-tables timeout 60; reasoning `off`; routing replay 98.89 with no gate reason; T04b strings in the production bundle; no errors in the logs after the deploy |
 | V3b | verify | What only a nightly can prove: B-16 (no `unknown or ambiguous` warnings, fallback count), O-1 (cost per night vs $0.61), B-17 (rivalry caveats kept or refreshed), T07 (the 22:00 UTC wave dispatched once), B-25 on the next deploy | — | done 2026-09-26 | Three nightlies read from the database (the log holds hours, and there is no drain). **T07:** one wave per night 09-23/24/25, 4 runs each, 12/12 `completed`. **O-1:** background tokens per night 1.72M → 1.13–1.18M (−33%), cost $0.61 → $0.12–0.25 (the price moves on its own; tokens are the measure). **B-16:** 0 of 95 fully-covered tables carry notes about a foreign column. **B-17:** three `MEASURED (rivalry)` caveats rewritten 2026-09-25 22:28. **B-25:** one chat trace since the deploy, routed `query/moderate`, not the default — too few to conclude; the routing eval is the proof |
 | T08 | deliver | PRJ-13 — orchestrator eval harness | audit §PRJ-13 | done — T08a #420 (routing, CI gate), T08b #422 (result gates, CI gate), T08c/T08d #423 (online report; replans/retries from spans) | routing replay 98.89 on production; gates 20/20; online 30 d: 25 traces, p95 321 s, 0 replans |
@@ -50,7 +52,7 @@ Standing rules the loop inherits (from `docs/evidence/retro.md`):
 | V3 | verify | T07–T09, B-18, B-19, B-26, B-27 and release 1.18.0 landed | — | done 2026-09-26 on v459 / web v388 | API reports `checkmydata-ai` **1.18.0**; `alembic_version` = repo head `e1f2a3b4c5d7`; markers present (CORS `X-Result-Capped`, readiness `repo_indexing`, `pipeline_continuation_context`, the honest continuation prompt); six new UI strings found in the served chunks, lazy ones included; fonts served from `/_next/static/media` as `font/woff2`, no Google Fonts in HTML or CSS; `/`, `/app`, `/login`, `/pricing`, `/verify-email`, `/about` all 200 under 0.61 s; `/api/health` ok; 0 errors / 5xx / H1x / R1x in 150 log lines after the release; web 311 MB, worker 255 MB; 0 open `error_log` rows; 0 active indexing runs at every merge |
 | T10 | deliver | Track D2 — `feed` / `reconciliation` / `temporal` / `exploration` as chat cards | plan Track D | todo | |
 | T11 | deliver | Track D3 — `data_graph` metric catalogue as a chat card | plan Track D | todo | |
-| T12 | deliver | PRJ-06 — indexing pipeline as a declared DAG with a resumable full rebuild | audit §PRJ-06 | todo | |
+| T12 | deliver | PRJ-06 — indexing pipeline as a declared DAG with a resumable full rebuild | audit §PRJ-06 | in progress — T12a done (#441): a full rebuild resumes, pinned to its checkpoint's tree (S-11). Remainder: `Step` objects, per-step sessions, run-keyed `save()`, per-run doc/token cap, `to_thread` for the four inline loops | T12a: `test_a_full_rebuild_resumes.py` (10); production acceptance (interrupted rebuild < 0.5× cold) read on the next restart that lands on a rebuild |
 | V4 | verify | T10–T12 landed | — | todo | |
 | T13 | deliver | PRJ-09 — connector contract and fifth-engine readiness | audit §PRJ-09 | todo | |
 | T14 | deliver | PRJ-05 — one executor, one gate pipeline (phase 1) | audit §PRJ-05 | todo | |
@@ -109,3 +111,4 @@ One line per finished iteration, newest last.
   reading a moved body; a mypy annotation). **Operator questions open: 1** (Q1).
 - **Cycle 3 (2026-09-23 → 09-26)** — B-02 (#425: the integration suite runs on PostgreSQL, and its first run found a production defect), B-19 (#426), B-18 (#427: six UX defects from re-auditing the 14 oldest scenarios). V3b closed from the database. A full local unit run was stopped by the harness for low machine memory; #427 went through CI as its full gate, and said so in the PR. **Operator questions open: 2** (Q1, Q2; neither blocks delivery).
 - **Cycle 4 (2026-09-26)** — #428 (cycle-3 consolidation), #429 (PostgreSQL CI job), #430 (reconciler test isolation), #431 (B-26), #432 (B-27: the second re-audit batch, 12 scenarios), #434 (fonts self-hosted after a Google Fonts hiccup failed the #431 deploy; a `node_modules` symlink I had committed in #432, untracked), and **release 1.18.0** (B-20). V3 for #425–#427 read on production v457: backend markers present, the new UI strings in the served bundle, no errors after restart. One failure was mine and is now a rule: a unit test that uses `with TestClient(app)` runs the app lifespan and leaked state into 27 later tests. **Operator questions open: 0** (Q1 resolved on its own; Q2 answered by pushing over SSH).
+- **Cycle 5 (2026-09-26)** — #436 (V3 of release 1.18.0 recorded), #437 (B-28: the sync-history panel read a row the API no longer returned — found while scoping T07d, and both suites were green), #438 (T07d: per-connection nightly jobs), #439 (B-21: background work outliving its test on the shared SQLite connection — the first fix cancelled and broke CI, the second waits), #441 (T12a: a full rebuild resumes). **Operator questions open: 0.**

docs/evidence/retro.md

@@ -20,6 +20,7 @@ two runs pass without it applying).
 | S-11 | **Status rows in `docs/evidence/loop-queue.md` are edited on `main` in a consolidation commit, never in a feature branch.** Every branch that touched the table conflicted with the next merge — the rows are adjacent — and each conflict cost a rebase and a full CI run; the CHANGELOG `[Unreleased]` heading has the same shape. Feature branches carry code, tests and the docs of what they change. | 2026-09-23 (five conflicts in one merge train: #410 twice, #414, #415) | never — this is the shape of a shared ledger |
 | S-12 | **Before a push, run CI's whole backend gate locally: `ruff format --check`, `ruff check`, `mypy app/`, and the FULL unit suite — not the tests next to the change.** Three CI failures in one afternoon were each invisible to the targeted run: a file formatted after its last edit (S-08 again), a `var-annotated` only mypy sees, and six source-inspection guards reading a function whose body had moved. A guard that reads source by NAME breaks on a refactor that keeps every behaviour; that is the price of the guard, and only the full suite pays it before CI does. Also S-09, a third time: a heredoc's failing assert did not stop the `git commit` on the next line. | 2026-09-23 (#415 twice, #416 twice) | CI runs these in under a minute locally |
 | S-13 | **A unit test that needs an HTTP answer builds `TestClient(app)` without `with`.** The context manager runs the app lifespan — reconciles, cron loops, process-wide singletons — and what it leaves behind breaks tests that run after it, in the full suite only. One CORS test did that to 27 tests on 2026-09-26, and the first symptom read as an unrelated flake in `test_main_cron` that I nearly filed as pre-existing (#430 carries the correction). Middleware answers without the lifespan. And a test that passes alone and fails in the full run is ordering until proven otherwise: run the suspect file first, then the failing ones, before blaming either. | 2026-09-26 (#431) | when a test genuinely needs the lifespan — then as an integration test with its own teardown |
+| S-14 | **On a shared connection, wait for a test's background work — never cancel it.** Under `StaticPool` every session is one SQLite connection; a task cancelled mid-statement makes SQLAlchemy invalidate it, and the replacement is a NEW, EMPTY `:memory:` database. My first B-21 drain cancelled and turned one flaky test into 263 failures and 965 errors on CI — while the same code passed 702/702 locally, because whether the cancel lands mid-statement is timing. So a local pass of a timing-sensitive change is not evidence; CI's run is. And a harness change that can empty a shared resource must check that resource afterwards (`_assert_schema_survived`) so it fails at the cause. | 2026-09-26 (#439) | never — cancellation on a shared connection is the shape |
 
 ## Run stamps