Skip to content

perf(crew-log): replay each unit's log from a watermark on boot - #34882

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/34846-session-tree-watermark
Nov 12, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/34846-session-tree-watermark

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway restart reads only the session-log bytes written since the last boot to rebuild the session tree, not every unit's whole log.

On a real home, any unit that got even one new line since the last boot had its whole log read again by the session tree seed, twice (once for adopt/release decisions, once for spawns), with every record JSON-parsed. The checkpoint held the tree's state but not how far into each log that state reached, so the seed could not tell what it had missed.

Why it matters

Startup session restore spends tens of seconds in this seed on a busy home (#34846: 14.3 s in find_last_tree_edge plus 5.0 s in find_spawned_entries). The cost grows with the size of the logs, not with what changed, so it gets worse the longer workers run.

Not a goal

What changed (motivation -> approach -> change)

The checkpoint now keeps, per unit, a watermark: the ResumePoint its last boot's CrewLog.iter_from walk reached (segment, record offset, seq, cut count), the session/spawned entries counted so far, and the identity of its oldest segment and newest file (inode, size, mtime). A unit whose files are unchanged is not opened. Any other unit is read through iter_from(resume=..., cursor=...), which verifies the point against the cut count and the record there, and the decisions and spawns it yields are folded with the rules the full read uses.

One unit is read in full when it has no watermark, its oldest segment changed (retention), its cut count moved or proves nothing (a rolled-back append is a cut), its newest file was replaced or is shorter than the mark, or its spawned entries would pass SPAWNED_PER_UNIT_CAP (so the cap drops the same entries main drops). A full read replaces that unit's held answer, so a decision retained out of the log is still dropped. A checkpoint decision or spawn row that does not decode drops that unit's watermark; a row that names no unit discards the checkpoint. If the newest decision past the mark is one the full read refuses, the held decision is dropped as the full read drops it. If iter_from skips a record in the walked range (damaged or oversized, the end of the log included: the walk's last record must end where the file ends) or refuses the segments, that unit is read the way main reads it, so an oversized record keeps the held evidence, reports the unit incomplete and leaves the mark before it.

A watermark is held only for a unit whose record the tree holds: when TREE_UNIT_CAP evicts the record, its watermark goes too, and a unit that comes back pays one full read, as on main. The negative-verdict scan caches (scans, spawn_scans) and tree_edge_scan_identity are removed; CHECKPOINT_VERSION goes from 5 to 6, so the first boot after upgrade rebuilds cold. No setting is added, and the tree is the same as main's (M4 below). The seed now opens each changed unit through CrewLog.open, so it takes that unit's append lock and can truncate a torn tail, where before it only read.

GIL time held by one boot seed on a busy-home copy, measured before the watermark stopped outliving cap eviction: 2.5605 s -> 0.5203 s. Real-home M1 was not re-measured on the current head: the busy-host ssh was refused at 00:05Z. Evicted units now pay one full read on return, as on main, so that number may rise; re-measure pending.

Backwards compatibility

Compatible: the only reader of the checkpoint file is this module; an older file fails the version check and is rebuilt by the existing cold scan, which is the documented recovery.

Reader: src/kiro_crew/crew_log/session_tree_projection_checkpoint.py:_load_checkpoint -- release -- reads marks instead of scans/spawn_scans; a v5 file is discarded (test_a_checkpoint_it_cannot_prove_loads_as_none[older-version])
Reader: src/kiro_crew/crew_log/session_tree_projection_seed.py:_seed_inner -- crew page -- the only caller of the removed _replay_spawned, now one _replay_tail pass (test_session_tree_projection_watermark.py)
Reader: src/kiro_crew/crew_log/session_tree_projection.py:SessionTreeProjection -- crew page -- _scans/_spawn_scans replaced by _marks at every site (test_crew_log_session_tree_spawned.py)

Tests

  • test/test_session_tree_projection_watermark.py (new):
    • records decoded after a one-line append are the same (3) at 4 log sizes; no append opens no log;
    • an adopt or release written while no gateway ran is applied from the delta;
    • rotation, shrink and inode change read only the changed unit in full;
    • a rolled-back spawn regrown to the same length, a whole final record with no newline, a malformed child payload, a unit with no .cuts, a re-spawn past the per-unit cap, and a creator past the unit cap behave as main;
    • equivalence: at 4 sizes and for 10 change cases (new unit, removed unit, rotation, shrink, inode change, adopt and release while down, unterminated last record, malformed child, cut rollback) plus the re-spawn case, the tree after a watermark boot equals the tree from the full read (SessionTree().reading); the equivalence cases also pass on main.
  • Seed, checkpoint, facade, spawned and adopt tests assert the watermark where they asserted the scan caches.

Benchmark (M1-M4 of the perf acceptance doc), same machine, before = main, after = this PR. M1 is time.thread_time() around the seed in its own thread, in a separate process, median of 5. M2 is a probe thread sleeping 5 ms while 4 seeds run at once (lateness in ms). M3 varies only the total log size (100 units with 200, 2000 and 20000 lines each, and a read-only copy of a busy home of 2.5 GB and 5218 units) with a fixed small change: one line on 10 units, one session/spawned, one session/adopted (the busy home took its own real appends over about 10 s). Each code boots cold, then replays (3 more boots on the busy home), then the change lands, then the measured boot runs from that code's own checkpoint. M4 compares a hash of the whole tree.

size store M1 s seed ms bytes read M2 p99 ms M2 max ms M4
1x 2.4 MB 0.0342 -> 0.0262 39.0 -> 33.3 427508 -> 1023160 8.74 -> 0.67 8.74 -> 0.67 identical
10x 23 MB 0.1815 -> 0.0289 186.9 -> 36.1 4738251 -> 7222934 23.11 -> 1.09 23.11 -> 1.09 identical
100x 230 MB 1.4011 -> 0.0315 1406.7 -> 38.6 41370368 -> 32362828 60.89 -> 0.67 74.63 -> 0.67 identical
real 2.5 GB 2.5605 -> 0.5203 2748.8 -> 724.4 308455033 -> 139336019 22.04 -> 8.74 47.47 -> 36.89 identical

Checkpoint size on the busy home: 1901384 bytes on main, 1630577 bytes here. Each changed unit pays a fixed read cost in CrewLog.open and iter_from (the 72 KiB tail window and header/first-record reads of up to 1 MiB each), so bytes read at the small sizes are above main's while M1 stays flat across the synthetic sizes (0.0262 / 0.0289 / 0.0315 s, main 0.0342 / 0.1815 / 1.4011 s). The busy-home row is a different store (5218 units, over the 4096 cap, with real appends), not the same fixed change.

bench_tree.py
"""Benchmark for #34882 (session tree watermark): M1-M4 per PERF_ACCEPTANCE.md.

Run with KIROCREW_HOME=<copy> and PYTHONPATH=<src of the code under test>. Modes:

  synth <entries_per_unit>   build a synthetic store: 100 units, each with a header,
                             session/opened, 3 session/spawned and N filler lines
  change                     the fixed small change, written as raw lines (another
                             process while no gateway ran): 1 line on 10 units, one
                             session/spawned and one session/adopted
  boot                       one seed that writes its checkpoint (setup only)
  m1                         one seed in its own thread, no checkpoint write; prints
                             thread_time (M1), wall, and a hash of the tree (M4)
  bytes                      like m1, but counts bytes read from unit logs
  m2                         4 seeds at once in 4 threads while a probe thread sleeps
                             5 ms in a loop; prints probe lateness p50/p99/max in ms

Every mode prints one JSON line. Nothing but the store under KIROCREW_HOME is touched.
"""

from __future__ import annotations

import builtins
import hashlib
import json
import os
import sys
import threading
import time
from pathlib import Path


def _no_pool() -> None:
    import kiro_crew.executors as ex

    ex.maintenance_executor = lambda: type(  # type: ignore[assignment]
        "_NoPool", (), {"submit": staticmethod(lambda *a, **k: None)}
    )


def _tree_hash(projection) -> tuple[str, int]:
    nodes = projection.nodes()
    rows = sorted((slot, n.parent_slot or "", bool(n.cycle)) for slot, n in nodes.items())
    blob = json.dumps([rows, projection.reading().incomplete]).encode()
    return hashlib.sha256(blob).hexdigest()[:16], len(rows)


def _seed_in_thread() -> tuple[float, float, object]:
    from kiro_crew.crew_log import session_tree_projection as stp

    out: dict = {}

    def run() -> None:
        projection = stp.SessionTreeProjection()
        w0, t0 = time.perf_counter(), time.thread_time()
        projection.ensure_seeded()
        out["cpu"] = time.thread_time() - t0
        out["wall"] = time.perf_counter() - w0
        out["projection"] = projection

    thread = threading.Thread(target=run)
    thread.start()
    thread.join()
    return out["cpu"], out["wall"], out["projection"]


def _unit_line(seq: int, kind: str, data: dict) -> bytes:
    return (
        json.dumps(
            {"type": kind, "seq": seq, "time": 1, "src": "gateway", "data": data},
            separators=(",", ":"),
        ).encode()
        + b"\n"
    )


def synth(per_unit: int) -> dict:
    from kiro_crew import crew_log as lg
    from kiro_crew.crew_log import CrewLog

    filler = b""
    for unit in range(100):
        sid = f"s-{unit:03d}"
        handle = CrewLog.create(lg.KIND_SESSION, sid, owner="o", agent="kirocrew", slot=f"slot-{unit}")
        handle.append(
            "session/opened",
            {"agent": "kirocrew", "slot": f"slot-{unit}", "model": "m", "cwd": "/w", "owner": "o", "resumed": False},
            src="gateway",
        )
        for child in range(3):
            handle.append("session/spawned", {"child": {"slot": f"w-{unit}-{child}"}}, src="gateway")
        path = handle._path
        del handle
        start = 5
        filler = b"".join(
            _unit_line(start + i, "turn/started", {"turn": i, "actor": "user", "depth": 0})
            for i in range(per_unit)
        )
        with open(path, "ab") as raw:
            raw.write(filler)
    from kiro_crew.crew_log.store import crew_log_root

    root = crew_log_root(lg.KIND_SESSION)
    size = sum(p.stat().st_size for p in root.glob("*/log*.jsonl"))
    return {"mode": "synth", "per_unit": per_unit, "units": 100, "log_bytes": size}


def change() -> dict:
    from kiro_crew import crew_log as lg
    from kiro_crew.crew_log.store import crew_log_root
    from kiro_crew.session_ledger import _store_name

    root = crew_log_root(lg.KIND_SESSION)
    written = 0
    for unit in range(10):
        path = root / _store_name(f"s-{unit:03d}") / "log.jsonl"
        seq = len(path.read_bytes().splitlines())
        lines = [_unit_line(seq, "turn/started", {"turn": 0, "actor": "user", "depth": 0})]
        if unit == 0:
            lines.append(_unit_line(seq + 1, "session/spawned", {"child": {"slot": "w-new"}}))
        if unit == 1:
            lines.append(_unit_line(seq + 1, "session/adopted", {"parent": {"slot": "slot-2"}}))
        blob = b"".join(lines)
        with open(path, "ab") as raw:
            raw.write(blob)
        written += len(blob)
    return {"mode": "change", "bytes_appended": written}


def boot() -> dict:
    from kiro_crew.crew_log import session_tree_projection as stp

    projection = stp.SessionTreeProjection()
    projection.ensure_seeded()
    projection.flush_checkpoint()
    projection.cancel_pending_checkpoint()
    return {"mode": "boot", "version": stp.CHECKPOINT_VERSION}


def m1() -> dict:
    cpu, wall, projection = _seed_in_thread()
    digest, slots = _tree_hash(projection)
    return {"mode": "m1", "thread_time_s": round(cpu, 4), "wall_s": round(wall, 4), "tree": digest, "slots": slots}


def bytes_mode() -> dict:
    from kiro_crew import crew_log as lg
    from kiro_crew.crew_log.store import crew_log_root

    root = crew_log_root(lg.KIND_SESSION)
    count = [0]
    real_open = builtins.open

    class _Counting:
        def __init__(self, handle):
            self._h = handle

        def read(self, *a):
            blob = self._h.read(*a)
            count[0] += len(blob)
            return blob

        def readline(self, *a):
            blob = self._h.readline(*a)
            count[0] += len(blob)
            return blob

        def readinto(self, b):
            n = self._h.readinto(b)
            count[0] += n or 0
            return n

        def __iter__(self):
            for line in self._h:
                count[0] += len(line)
                yield line

        def __enter__(self):
            return self

        def __exit__(self, *e):
            self._h.close()

        def __getattr__(self, name):
            return getattr(self._h, name)

    def counting_open(file, mode="r", *a, **k):
        handle = real_open(file, mode, *a, **k)
        if "b" in mode and "r" in mode and isinstance(file, (str, Path)) and Path(file).parent.parent == root:
            return _Counting(handle)
        return handle

    builtins.open = counting_open
    _cpu, wall, _projection = _seed_in_thread()
    builtins.open = real_open
    return {"mode": "bytes", "bytes_read": count[0], "wall_s": round(wall, 4)}


def m2() -> dict:
    from kiro_crew.crew_log import session_tree_projection as stp

    late: list[float] = []
    stop = threading.Event()

    def probe() -> None:
        while not stop.is_set():
            t0 = time.perf_counter()
            time.sleep(0.005)
            late.append((time.perf_counter() - t0 - 0.005) * 1000)

    def seed() -> None:
        stp.SessionTreeProjection().ensure_seeded()

    probe_thread = threading.Thread(target=probe)
    probe_thread.start()
    time.sleep(0.05)
    late.clear()
    w0 = time.perf_counter()
    seeds = [threading.Thread(target=seed) for _ in range(4)]
    for t in seeds:
        t.start()
    for t in seeds:
        t.join()
    wall = time.perf_counter() - w0
    stop.set()
    probe_thread.join()
    late.sort()

    def pct(p: float) -> float:
        return round(late[min(len(late) - 1, int(p * len(late)))], 2)

    return {"mode": "m2", "samples": len(late), "p50_ms": pct(0.50), "p99_ms": pct(0.99), "max_ms": round(late[-1], 2), "wall_s": round(wall, 3)}


def main() -> None:
    mode = sys.argv[1]
    import kiro_crew

    _no_pool()
    if mode == "synth":
        out = synth(int(sys.argv[2]))
    elif mode == "change":
        out = change()
    elif mode == "boot":
        out = boot()
    elif mode == "m1":
        out = m1()
    elif mode == "bytes":
        out = bytes_mode()
    elif mode == "m2":
        out = m2()
    else:
        raise SystemExit(f"unknown mode {mode}")
    out["code"] = str(Path(kiro_crew.__file__).parent.parent.parent.name)
    print(json.dumps(out), flush=True)


if __name__ == "__main__":
    os.environ.setdefault("KIROCREW_CREW_LOG", "1")
    main()
bench_driver.py
"""Drive bench_tree.py for main and the branch over ONE store, swapping checkpoints.

Usage: python bench_driver.py <home> <label> [synth <entries_per_unit> | real]

One store, so main and the branch see the same bytes in the same directory order. Each
code gets its own checkpoint: cold boot, then a replay boot, saved beside the store.
Then the fixed change (synth) or nothing (real: the caller appends the host's delta
before calling with 'measure'), then per code: M1 x5, bytes x1, M2 x1, each from that
code's saved checkpoint. Prints JSON lines; a summary line last.
"""

from __future__ import annotations

import json
import os
import shutil
import statistics
import subprocess
import sys
from pathlib import Path

HERE = Path(__file__).resolve().parent
PY = "<repo>/.venv/bin/python"
CODE = {
    "main": "<main checkout>/src",
    "branch": "<this branch>/src",
}


def run(home: Path, code: str, *args: str) -> dict:
    env = dict(os.environ, KIROCREW_HOME=str(home), PYTHONPATH=CODE[code], KIROCREW_CREW_LOG="1")
    proc = subprocess.run(
        [PY, str(HERE / "bench_tree.py"), *args], env=env, capture_output=True, text=True, check=True
    )
    line = proc.stdout.strip().splitlines()[-1]
    out = json.loads(line)
    out["code_label"] = code
    print(json.dumps(out), flush=True)
    return out


def ckpt(home: Path) -> Path:
    return home / "crew-log" / "sessions" / "projections" / "session-tree.json"


def prepare(home: Path) -> None:
    for code in CODE:
        ckpt(home).unlink(missing_ok=True)
        for _ in range(int(os.environ.get("BENCH_BOOTS", "2"))):
            run(home, code, "boot")
        shutil.copyfile(ckpt(home), home / f"ckpt-{code}.json")


def measure(home: Path, label: str) -> None:
    summary: dict = {"label": label}
    for code in CODE:
        cpus, walls, trees = [], [], set()
        for _ in range(5):
            shutil.copyfile(home / f"ckpt-{code}.json", ckpt(home))
            out = run(home, code, "m1")
            cpus.append(out["thread_time_s"])
            walls.append(out["wall_s"])
            trees.add((out["tree"], out["slots"]))
        shutil.copyfile(home / f"ckpt-{code}.json", ckpt(home))
        nbytes = run(home, code, "bytes")["bytes_read"]
        shutil.copyfile(home / f"ckpt-{code}.json", ckpt(home))
        probe = run(home, code, "m2")
        summary[code] = {
            "m1_thread_time_median_s": statistics.median(cpus),
            "seed_wall_median_ms": round(statistics.median(walls) * 1000, 1),
            "bytes_read": nbytes,
            "m2_p50_ms": probe["p50_ms"],
            "m2_p99_ms": probe["p99_ms"],
            "m2_max_ms": probe["max_ms"],
            "tree": sorted(trees),
        }
    summary["m4_identical"] = summary["main"]["tree"] == summary["branch"]["tree"] and len(
        summary["main"]["tree"]
    ) == 1
    print("SUMMARY " + json.dumps(summary), flush=True)


def main() -> None:
    home, label, what = Path(sys.argv[1]), sys.argv[2], sys.argv[3]
    if what == "synth":
        run(home, "branch", "synth", sys.argv[4])
        prepare(home)
        run(home, "branch", "change")
        measure(home, label)
    elif what == "prepare":
        prepare(home)
    elif what == "measure":
        measure(home, label)


if __name__ == "__main__":
    main()
Raw output (summary lines)
SUMMARY {"label": "synth-1x-final", "main": {"m1_thread_time_median_s": 0.0342, "seed_wall_median_ms": 39.0, "bytes_read": 427508, "m2_p50_ms": 0.18, "m2_p99_ms": 8.74, "m2_max_ms": 8.74, "tree": [["25a3d36b5f48b836", 401]]}, "branch": {"m1_thread_time_median_s": 0.0262, "seed_wall_median_ms": 33.3, "bytes_read": 1023160, "m2_p50_ms": 0.11, "m2_p99_ms": 0.67, "m2_max_ms": 0.67, "tree": [["25a3d36b5f48b836", 401]]}, "m4_identical": true}
SUMMARY {"label": "synth-10x-final", "main": {"m1_thread_time_median_s": 0.1815, "seed_wall_median_ms": 186.9, "bytes_read": 4738251, "m2_p50_ms": 3.07, "m2_p99_ms": 23.11, "m2_max_ms": 23.11, "tree": [["25a3d36b5f48b836", 401]]}, "branch": {"m1_thread_time_median_s": 0.0289, "seed_wall_median_ms": 36.1, "bytes_read": 7222934, "m2_p50_ms": 0.11, "m2_p99_ms": 1.09, "m2_max_ms": 1.09, "tree": [["25a3d36b5f48b836", 401]]}, "m4_identical": true}
SUMMARY {"label": "synth-100x-final", "main": {"m1_thread_time_median_s": 1.4011, "seed_wall_median_ms": 1406.7, "bytes_read": 41370368, "m2_p50_ms": 10.25, "m2_p99_ms": 60.89, "m2_max_ms": 74.63, "tree": [["25a3d36b5f48b836", 401]]}, "branch": {"m1_thread_time_median_s": 0.0315, "seed_wall_median_ms": 38.6, "bytes_read": 32362828, "m2_p50_ms": 0.12, "m2_p99_ms": 0.67, "m2_max_ms": 0.67, "tree": [["25a3d36b5f48b836", 401]]}, "m4_identical": true}
SUMMARY {"label": "real-eof", "main": {"m1_thread_time_median_s": 2.5605, "seed_wall_median_ms": 2748.8, "bytes_read": 308455033, "m2_p50_ms": 0.41, "m2_p99_ms": 22.04, "m2_max_ms": 47.47, "tree": [["2d4fb1b79f48bfb2", 3009]]}, "branch": {"m1_thread_time_median_s": 0.5203, "seed_wall_median_ms": 724.4, "bytes_read": 139336019, "m2_p50_ms": 0.1, "m2_p99_ms": 8.74, "m2_max_ms": 36.89, "tree": [["2d4fb1b79f48bfb2", 3009]]}, "m4_identical": true}

Manual verification

The benchmarks above ran on read-only copies; no live home was written.

Follow-up

PR C, not built here. After this PR and the guard PR (direct crew-log segment opens limited to an allowlist) both merge, and Audit B's list is fixed or allowlisted with reasons, PR C makes an over-budget crew-log read on the gateway process raise an error by default; off stays the only escape hatch. Direct segment opens are routed through the counted CrewLog entry points or covered by the guard test. PR C tests: a delta read passes, a from-the-top read on the gateway raises, a child process is unaffected, and every remaining boot and runtime caller stays under budget on a real-home copy.

Also a follow-up: trim the header and first-record reads in CrewLog.open / iter_from (up to 1 MiB each, jsonl_util.py _FRAME_CHUNK) so a resumed read of a changed unit costs only its delta.

Related Issues

Refs #34846

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 23aea8291350703e74880a60a2461bb11907115f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 23aea82

Verdict parsed from the review's SHA-scoped output markers for commit 23aea8291350703e74880a60a2461bb11907115f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 23aea8291350703e74880a60a2461bb11907115f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 23aea8291350703e74880a60a2461bb11907115f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 23aea82

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 23aea8291350703e74880a60a2461bb11907115f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 23aea8291350703e74880a60a2461bb11907115f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the watermark goes after the cause in #34846 (a boot's cost tracked log size, not what changed) and deletes the two scan caches it replaces.

Not checked: real-home M1 on this head. The description says it was not re-measured.

What this change ships

Inventory (8 items) — 8 justified

Intent: a gateway restart rebuilds the session tree by reading only the session-log bytes written since the last boot. A FIX (performance, #34846).

  1. A boot reads each changed unit's log only from where the last boot stopped — justified
  2. A unit whose files are unchanged is not opened at boot — justified
  3. Adopt, release and spawn records written while no gateway ran are applied from the delta — justified
  4. The first boot after upgrade rebuilds cold (checkpoint version 5 → 6) — justified
  5. The scans/spawn_scans caches and tree_edge_scan_identity are removed from the checkpoint, the projection and the store facade — justified
  6. The seed opens each changed unit through CrewLog.open, so it takes the append lock and can truncate a torn tail — justified
  7. A unit evicted at TREE_UNIT_CAP loses its mark and pays one full read when it returns — justified
  8. A boot that only advanced marks still rewrites the checkpoint — justified

[FIRST-PRINCIPLES-REVIEWED] 23aea82

Description read

[DESCRIPTION-READ] 043947717344eddb58a1643b268b5a070d80452eb251de9cfcf0e8e7d024b61c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from 2597b64 to 843c890 Compare November 11, 2026 03:07
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the GPT 6.1 review of 2597b64 (#34882 (comment)): both BLOCKING findings are fixed in head 843c890.

  1. store_retention.py:631, rollback then regrowth to the same length. Fixed: the watermark now carries the unit's settled cut count (read_cuts), so a mark is (oldest_first, newest_first, inode, offset, cuts). A rollback is a cut, so the count moves and that unit is read in full. A count that proves nothing (absent, unreadable, odd) also forces a full read, and a read during which the count moved stores no mark. Test: test_a_rolled_back_spawn_replaced_at_the_same_length_is_re_read (fails on 2597b64 with {'W1'} == {'W2'}).

  2. session_tree_projection_seed.py:532, malformed session/spawned child. Fixed: the delta-cap count is built from spawned_record results, so a malformed payload is dropped there as it is below. Test: test_a_malformed_spawn_after_the_mark_does_not_abort_the_seed[not-a-map|slot-not-a-str] (both fail on 2597b64 with an empty tree).

The store facade now binds and declares the new names (TreeMark, TreeRead, _MarkStale, _TREE_NEEDLES, _read_tree, strict_raw_records_with_offsets), which were the shard 1 and shard 7 facade reds.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the GPT 6.1 review of 843c890 (#34882 (comment)): the BLOCKING finding at store_retention.py:667 is fixed in head ec3530c.

A final record that is whole but has no trailing newline is now folded, as the tail reader folds it, and the watermark stays before it. The next read takes it again and folds it to the same answer. A final record that does not parse is still left for the next read, and nothing after it moves the watermark.

Tests: test_a_complete_release_missing_its_newline_is_applied_and_read_again_once (fails on 843c890 with assert 'slot-c' is None), and test_an_unparseable_partial_tail_is_left_for_the_next_read (guard for the deferred partial tail).

@KiroCrewCaptain

KiroCrewCaptain commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own.

Each unit now gets a saved mark recording how far the last boot read its log, and the next boot reads only the bytes after it. Every case where the mark might be wrong falls back to main's full read: the cut counter is missing, odd or has moved; the oldest or an older segment changed; the newest file has a new inode or is shorter than the mark; the read stopped short of end of file (checked with extent_at); the walk skipped a record or iter_from refused it; a resumed read's spawn count passes the per-unit cap. The mark is dropped on every path that removes a unit's evidence (forget, retention reconcile, record/edge/spawn eviction, units that disappeared, undecodable checkpoint rows). No code still uses the old scan caches. The version bump forces a cold rebuild on upgrade. A resumed read handles decisions and spawns the same way a full read does. CrewLog.open only holds the append lock briefly (no deadlock), and cutting off a torn tail is its existing behaviour under the writers' lock. The read budget only reports. Tests compare the result with the cold scan for each change case and cover the damage edge cases. CI is green. No blocking findings.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 23aea82913 · 1713 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 23aea8291350703e74880a60a2461bb11907115f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the per-unit watermark is the right shape, any doubt falls back to main's full read, and bumping CHECKPOINT_VERSION means an older checkpoint is just rebuilt cold. Easy to revert.

Notes

  • The description says the busy-home M1 number (2.5605 → 0.5203 s) comes from a previous head and was not re-measured; evicted units now pay a full read again. Re-measure before merge, because that row is the headline claim of this perf PR.
  • The seed now calls CrewLog.open instead of only reading. It takes the append lock and truncates a torn tail on every unit that changed, across every unit at boot. store_handle.py:open documents that truncation as safe under the lock, but it makes boot contend with live writers. Lock wait at boot belongs in the re-measure.

[DESIGN-REVIEWED] 23aea82

Description read

[DESCRIPTION-READ] 483039377bcccb7d0265b3dbb0ee2c10d4e90f2fb6fd0cd5d4ba8352f602cacf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 23aea8291350703e74880a60a2461bb11907115f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from ec3530c to 78af7aa Compare November 11, 2026 10:24
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the Opus 5.5 review of ec3530c (#34882 (comment)): the BLOCKING finding at store_retention.py:697 is fixed in head 78af7aa.

A unit with no .cuts file now keeps a watermark that only vouches for the exact bytes: newest segment inode, size and mtime. Unchanged, the next boot costs one stat and reads no log bytes; any change reads that unit in full. Readers still never write .cuts. Test: test_a_unit_with_no_cut_counter_is_read_once_then_skipped_until_it_changes (fails on ec3530c with {'s-old-...': 116150} bytes read).

The same head also makes a unit whose spawned entries would pass SPAWNED_PER_UNIT_CAP take a full read (as main's full read drops the oldest entries), keeps a watermark when TREE_UNIT_CAP evicts the unit's record, and keeps spawns for a creator re-read past the unit cap, as main does. M1-M4 before/after are in the PR body; the tree matches main at every size measured.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 11, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from 78af7aa to fc4b6e9 Compare November 11, 2026 11:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the GPT 6.1 review of 78af7aa (#34882 (comment)): the BLOCKING finding at session_tree_projection_checkpoint.py:501 is fixed in head fc4b6e9.

When a checkpoint decision or spawn row does not decode, the loader now drops the watermark of the unit the row names, so that unit is read in full and the decision comes back from the log. A row that names no unit discards the whole checkpoint (cold rebuild). Test: test_a_damaged_decision_row_reads_its_unit_in_full (fails on 78af7aa with assert None == 'slot-c').

The same head also fixes the two CI reds on 78af7aa:

  • test_crew_log_direct_segment_open_ratchet.py: _read_tree now counts its read through read_budget.count_read (fold tree; a full read counts as from the top), and is listed with that reason. find_spawned_entries and _scan_whole_for_types are off the boot path and stay for the cold scan and reconcile_edge.
  • test_a_creator_past_the_unit_cap_keeps_its_spawns_across_boots: the cold scan admits units in store-listing order, so with three units and a cap of 2 the creator was not always admitted on the first boot. The test now starts within the cap and adds the third unit after.

Real-home M1 2.6008 -> 0.9042 s, M4 identical tree; numbers are in the PR body.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from fc4b6e9 to 2697e58 Compare November 11, 2026 11:59
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 11, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from ca206b5 to f65d8f5 Compare November 12, 2026 02:10
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the GPT 6.1 review of ca206b5 (#34882 (comment)): both triggers at session_tree_projection_seed.py:660 are fixed in head f65d8f5. This is the final push for #34882.

Every way iter_from can skip or drop a record in _read_unit, and where each one ends:

Case Ends in Test
leading record skipped (oldest segment) seq gap from the oldest segment's first seq - 1 -> _read_whole test_an_oversized_first_entry_on_a_full_walk_is_a_fault_as_in_a_full_read
middle record skipped seq gap -> _read_whole test_an_oversized_spawn_after_the_mark_is_a_fault_as_in_a_full_read
trailing record skipped extent mismatch (last record must end at the newest segment's end) -> _read_whole test_an_oversized_spawn_at_the_end_of_the_log_is_a_fault_as_in_a_full_read
non-advancing seq (trigger 1) strict_seq=True raises -> _read_whole test_a_record_reusing_the_marked_seq_is_read_as_a_full_read_does (new; fails on ca206b5)
no record taken on a full walk (trigger 2) at is None -> _read_whole test_a_full_walk_that_yields_nothing_is_read_as_a_full_read_does (new; fails on ca206b5)
provenance refusal (segment gap, bad header) CrewLogError -> _read_whole test_a_segment_gap_is_read_as_a_full_read_does

Trigger 1 also needed the full read's decision to replace the held one outright: the held adoption and the release share a seq, so edge_supersedes kept the adoption. A full read is the unit's whole answer, as in main's cold scan.

Intact logs stay on the delta path: synthetic M1 0.0261 / 0.0287 / 0.031 s (main 0.0329 / 0.1854 / 1.388 s), M4 identical. Real-home M1 is still not re-measured (busy-host ssh refused again at 02:04Z).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes. I checked both items from my earlier review on this head:

  • Root-listing failure kept marks for dropped spawns: fixed. crew_log/session_tree_projection_seed.py:447 now filters the loaded marks to held records before the listing, and the s-gone case is back in test_spawns_are_kept_and_reported_when_the_root_cannot_be_resolved.
  • Marks outlived record eviction: fixed. Marks are filtered to held records at install (seed.py:373), eviction pops the mark, and _MARKS_CAP is gone.

One thing still needs a change before merge:

  1. test/test_session_tree_projection_watermark.py:441 and :178: two new tests run close to the per-test timeout. The CI duration tables on this head show test_the_tree_after_an_append_equals_a_full_read_at_every_size[16000] at 97 s of the 120 s cap on Linux and 166 s of the 180 s cap on Windows. test_a_boot_after_a_small_append_reads_only_the_append_at_every_log_size takes 64 s on Linux and 109 s on Windows. They are the slowest tests in both shards. On Windows the suite runs with --max-worker-restart=0, so a slower runner pushes the 16000 case past the cap. pytest-timeout then kills the worker and the whole shard aborts, which turns CI red for unrelated PRs. Almost all of the time is setup: _unit (line 133) writes each filler line through its own CrewLog.append. The 16000 equivalence case makes about 32k appends, and the small-append test makes 21,250. Smallest fix: write the filler with one raw bulk write per unit, as _raw_line and your bench synth mode already do, and keep CrewLog.append only for the records the tests assert on. Or drop the 16000 size: DELTA_DECODED does not depend on size, so 250, 1000 and 4000 already show that the read stays flat. Please check the slowest-durations table on both OSes after the change.

Optional:

  • The seed now goes through CrewLog.open, so for each changed unit it takes the append lock and can truncate a torn tail. Before, it only read. One sentence in the PR body would make that visible to reviewers.
  • The new checkpoint-loader branches in session_tree_projection_checkpoint.py around line 501 have no test. Two cases would pin them: an edges row with no sid (the loader returns None), and a damaged spawned row (that unit's mark is dropped).
  • The current CI reds in test_update_boot_apply.py come from main and were fixed there after your merge-base, so a rebase should clear them. The body line "the same file passes on main" is true only for the equivalence cases, because the file reads _marks.

@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from f65d8f5 to 7b1a8f6 Compare November 12, 2026 03:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@chenmingwei23
chenmingwei23 marked this pull request as draft November 12, 2026 04:02
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Thanks @KiroCrewCaptain for the review (5868782641 on f65d8f5; 7b1a8f6 is the same patch rebased onto main).

Both earlier code findings are confirmed fixed:

  • session_tree_projection_seed.py:447: loaded marks are filtered to units with a record before the root is listed; the s-gone case is back in test_spawns_are_kept_and_reported_when_the_root_cannot_be_resolved.
  • session_tree_projection_seed.py:373: marks are filtered to held records at install, cap eviction pops the mark, and _MARKS_CAP is gone.

The new blocker is real: two tests in test/test_session_tree_projection_watermark.py run close to the per-test timeout.

  • :441 (the 16000-entry case of the size-parametrized equivalence test): 97 s of 120 s on Linux, 166 s of 180 s on Windows.
  • :178 (the small-append test): 64 s on Linux, 109 s on Windows.
    The setup cost is the per-line CrewLog.append in _unit (:133).

Smallest fix we would take: write the filler lines with one raw bulk write per unit (as _raw_line and the benchmark's synth mode already do), and keep CrewLog.append only for the records the tests assert on. Or drop the 16000 size, since DELTA_DECODED does not depend on log size.

The PR is parked as draft. This fix waits for Mingwei's GO; nothing more will be pushed until then.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
The session tree checkpoint now stores, per unit, where the last boot's
read of that unit's log stopped (oldest and newest segment, newest
segment inode, byte offset). A boot reads only the bytes after it, in
one pass for decisions and spawns. A unit with no mark, or whose
segments changed under it (new or removed segment, replaced or shorter
file, offset not on a record boundary), is read in full. The scan
caches (scans, spawn_scans) leave the checkpoint; CHECKPOINT_VERSION
5 -> 6.

Refs #34846
@chenmingwei23
chenmingwei23 force-pushed the fix/34846-session-tree-watermark branch from 7b1a8f6 to 23aea82 Compare November 12, 2026 14:10
@chenmingwei23
chenmingwei23 marked this pull request as ready for review November 12, 2026 14:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the KiroCrewCaptain review of f65d8f5 (#34882 (review)): the test-duration blocker is fixed in head 23aea82. CI on that head: 98 ok, 0 fail.

Both earlier code fixes stand unchanged (session_tree_projection_seed.py:447 marks filtered before the listing; :373 marks filtered at install, eviction pops the mark, _MARKS_CAP gone).

The test fix is test-only: _unit in test/test_session_tree_projection_watermark.py now writes each unit's filler lines with one raw bulk write, and CrewLog.append is kept only for the records the tests assert on. The 16000 size is kept. The two optional loader tests are added (test_an_edges_row_that_names_no_unit_discards_the_checkpoint, test_a_damaged_spawned_row_drops_its_units_mark), and the PR body now says the seed opens each changed unit through CrewLog.open (append lock, can truncate a torn tail).

Slowest-durations tables, New-test stability jobs, slowest of the passes:

Test Linux before -> after Windows before -> after
equivalence at every size [16000] 97 s (of 120) -> 0.80 s 166 s (of 180) -> 0.73 s
small append reads only the append 64 s -> 0.55 s 109 s -> 0.56 s

Could you take another look?

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 12, 2026 15:59

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 23aea82. It read the whole diff. It ran the watermark, seed, checkpoint, spawned, adopt and facade tests on this head (all passed), and ran the same crew-log tests with the diff applied onto current main (all passed). It ran the equivalence cases against main's code, where they also hold. It confirmed that no reader of the removed names (tree_edge_scan_identity, scans, spawn_scans) is left, and that git merge-tree against main is clean.

Earlier review items:

  • Root-listing failure kept marks for dropped spawns: fixed on the previous head and still fixed here (seed.py:447, s-gone case restored).
  • Marks outlived record eviction: fixed on the previous head and still fixed here (filtered at install, popped on eviction, _MARKS_CAP removed).
  • Watermark tests near the per-test timeout: fixed. _unit now writes the filler in one raw write. The 16000 case ran in under 1 s locally, and no watermark test appears in the slowest-durations table of any backend shard on this head.

Done well: every case the watermark cannot vouch for falls back to main's own full read, and the equivalence suite proves this against a cold SessionTree().reading at four sizes and ten change-while-down cases.

Optional:

  • The mark drop on decision eviction at session_tree_projection.py:672 still has no test that fails without it. A test with TREE_UNIT_CAP patched small that evicts a held unit's decision and boots twice would pin it.
  • The busy-home row (2.5605 s to 0.5203 s) was measured on an earlier head. Re-running it on 23aea82 and posting the result would back the headline number for homes over the unit cap.
  • test_a_boot_after_a_small_append_reads_only_the_append_at_every_log_size asserts decoded records, not bytes read. A name like "decodes only the append" would match what it checks.

@KiroCrewCaptain
KiroCrewCaptain merged commit 80b3bdb into main Nov 12, 2026
104 of 109 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/34846-session-tree-watermark branch November 12, 2026 16:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants