Skip to content

feat(crew-log): fold a session's crew log into five projections and serve them - #11558

Merged
iamwhatever merged 2 commits into
mainfrom
feat/crew-log-projection
Sep 18, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/crew-log-projection

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A session's crew log is on disk and nothing can read it. #10091 (merged as
69aeb2803) landed the whole storage layer -- the envelope, seq, ref,
iter_from, page, resolve, torn-tail repair -- and the flag-gated session
emitter that writes it. What does not exist is any way to see what is in the
file: no fold, no route, no push. So a session's own history is written and then
unreadable, and the side panel RFC #10090 describes has nothing behind it.

The gap is not "add an endpoint". A caller that read the raw lines would have to
fold them itself, which means every client needing one number pulls the whole
file, and two clients folding the same bytes can disagree about what they say.

Why it matters

The crew log exists so that a session's state lives in a file rather than in a
context window. That only pays off once something reads it: until a reader can
ask "what did this session spend", "which tool calls are still open", "is this
turn still running", the file is cost with no return, and the flag has no reason
to be turned on.

It also matters that the first reader gets the reading rules right. This is an
append-only record whose whole claim is that two readers of the same bytes agree.
A first consumer that quietly treats an absent measurement as zero, or invents a
completion for a turn that was interrupted, teaches every later consumer to do
the same -- and the file cannot be corrected afterwards.

What changed (motivation -> approach -> change)

Goal: the session half of RFC #10090 section 5 -- the five folds, paged
reads, and the push -- with the reading rules stated where the next consumer will
find them.

Approach. Three decisions carry the design.

  1. The incremental form is the primitive; the whole-file form wraps it. A fold
    is a starting state, one step per entry, and a render. fold(name, entries)
    is advance from an empty checkpoint, one line. The alternative -- a batch
    implementation beside a resumed one -- is two codepaths free to disagree about
    one file, with nothing in the file to say which is right.
  2. A checkpoint is (name, last_seq, state) and is JSON-serializable, so the
    shape a later projections/<key>.json would carry already exists. State is
    deliberately not the rendered value: a fold keeps bookkeeping a reader has no
    use for (open tool calls keyed by call_id, the attempt an open turn is on).
  3. A page and a fold take opposite postures on a type they do not know. The
    fold passes its vocabulary to iter_from, so a required unknown type refuses
    rather than letting a total the line may have changed look authoritative. The
    page passes none, because paging renders history for a person and refusing the
    page would hide the history in front of the one unfamiliar line. That is the
    posture crew-log-core.md section 6 already states for page and resolve,
    applied to a range read.

What is built.

  • src/kiro_crew/crew_log/projection.py: status, usage, timeline, tools,
    approvals, each as fold_<name>(entries) -> dict plus the incremental
    advance(checkpoint, entries); fold_session folds all five from one pass.
  • GET /api/sessions/{id}/crew-log?from=&to=: entries in a seq range, oldest
    first, every ref on the page resolved to {status, entries, first_seq, last_seq}. The verdict and span, never the cited bytes -- those are a page of
    their own unit, which this route already serves. Identical refs resolve once,
    and a page resolves at most 25 distinct refs and reports what it left.
  • GET /api/sessions/{id}/crew-log/projection/{name}: the fold's value and the
    seq it folded through (FR-5).
  • A session_projection frame ({session_id, name, seq, value}) pushed when a
    log grows. crew_log.emit.add_growth_listener reports a drained BATCH,
    so a consumer is woken once per turn's burst rather than once per entry; the
    listener is registered, not imported, because the emitter is imported by the
    dashboard and calling back into it would close a cycle. The publisher hands the
    id to the loop from the writer thread, coalesces, then folds all five from one
    incremental read and sends a frame only for a projection whose seq moved.

Four rules the folds hold, each stated in the spec and pinned by a test:

  • Absent is never zero. turn/completed omits credits/tokens on a
    synthesized close, so every total rides beside the count of turns that reported
    one (turns.credits_reported, turns.tokens_reported).
  • Nothing is synthesized. An open turn and an unmatched call are reported
    open. Closing them is open(repair=True), which appends real closers under
    write ownership.
  • An unpairable id is counted, never paired. call_id and approval_id may
    be empty; keying a map by an empty id would let one completion close a
    different call's frame.
  • Every value is bounded, because it is pushed over a socket: the timeline is
    a window that reports what it dropped, tools details 100 names while keeping
    totals exact, open/pending lists are capped with their own omitted counts.

Two things the reviewer should look at directly:

  • The replay refusal. advance refuses an entry at or below its checkpoint
    rather than skipping it. Skipping is idempotent and therefore tempting, but it
    is wrong for the case that actually happens: a unit removed and recreated
    restarts seq, and a cached checkpoint at 50 would swallow the whole new log
    silently. Refusing names the collision, and fold_session discards a bundle
    whose seq is ahead of the file so the recreated-unit case rebuilds.
  • The lazy storage import. handlers/crew_log.py imports the ledger package
    only inside the calls that need it. The crew log is optional behind
    KIROCREW_CREW_LOG and this module is on the dashboard's boot path;
    test_a_flag_off_launch_does_not_load_the_storage_subsystem caught the eager
    version, and a test here pins the same invariant for this module.

Scope, per the RFC's PR 2 split: crew-kind folds are out (the crew kind has no
writer), on-disk checkpoints are out (the state shape is ready for them),
subagent lineage is out, and there is no SPA work -- the frame shape is specified
so the client can follow. #11185 (resolver) and #11193 (retention) touch the
same package; the only shared file this edits is crew_log/emit.py, and
the edit there is one listener registry plus one try/finally around the batch
writer's exits.

Naming: the feature is "crew log" in every identifier, route, docstring and doc
line this adds. The kiro_crew.crew_log package keeps its current names, which a
separate rename sweep owns.

Tests

test/test_crew_log_projection.py (61) and test/test_crew_log_routes.py (32),
93 new tests. What they lock in:

  • The contract, per projection: incremental folding equals folding from
    scratch at EVERY split point of a log exercising all five folds; a checkpoint
    survives a JSON round trip and keeps folding to the same answer; advance
    leaves its input untouched; a replayed entry is refused.
  • The reading rules: absent credits are not zero; an open turn is reported
    open; two calls with an empty call_id are never paired; a refused status and
    an asserted is_error each count, an absent is_error does not; the timeline
    window and the tool-name budget bound the value while totals stay exact.
  • The posture difference: a planted unknown required type is SHOWN by the
    page and REFUSED (409) by the fold, over the same file.
  • The routes: range, cursor, clamping, defaults, malformed ranges, the
    absent-log shape, ref resolution to ok and gone, dedupe, the ref budget,
    and that a non-owner caller is refused on both reads.
  • The push: one frame per projection with the right seq; nothing on a pass
    with no growth; the incremental read starts at the first new seq; no fold at
    all when nobody is watching; a bounded session cache; one session's fold
    refusal does not stop the others; notify does no work on the writer thread.
  • The boot path: importing the handler module in a clean interpreter loads no
    kiro_crew.crew_log module.

Mutation-verified: 14 mutations, one per load-bearing behaviour (the replay
refusal, the state copy, the credit-turn count, the empty-id rule, both bounds,
both vocabularies, the bundle guard, the ref dedupe and budget, the unchanged-seq
skip, the owner gate, the watcher check). 14/14 killed. The first pass killed 13:
the bundle-guard test was vacuous -- it rebuilt the stale bundle from the same
session's own state, so reusing it produced the right answer anyway. It now
carries a busier session's state under this session's id, which is the shape a
cache actually holds after a unit is recreated.

Local gates, all green: black, isort, flake8, mypy --follow-imports=silent
on the touched paths, check_black_formatting, check_brand_name,
check_changelog_history, check_comment_history, check_feature_map,
check_focus_cue, check_harness_parity, check_memory_store_seam,
check_lockdown_before_publish, docs_lint, docs-lint.sh. Existing suites
re-run: test_crew_log_emit.py, test_crew_log_core.py,
test_session_ledger_order.py -- 493 passed with the new files.

Manual verification

N/A -- backend only, and every path is covered by unit tests including the two
that need a real subprocess (the boot-path import census) and a real file (the
planted unknown type). There is no UI in this PR to exercise, and the push is
verified through the publisher's own seams rather than a live socket.

Related Issues

No linked issue. Implements the session half of RFC #10090 section 5; follows
#10091 (merged as 69aeb2803).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Other suggestions on the work

Two observations from implementing the spec, neither fixed here:

  1. turn/completed carries no attempt. crew-log-core.md section 5 says
    (turn, attempt) is the identity a fold groups on, and attempt is declared
    on turn/started only. A fold therefore cannot pair a completion with its
    start by field; the pairing has to be positional (a start opens the current
    attempt at that ordinal, the next completion for it closes whatever is open).
    That is what the file supports, and it is what this fold does, but the spec
    sentence promises something the type table does not provide. Adding attempt
    to turn/completed is additive.
  2. approval/* uses approval_id, the spec table says id. The declaration
    in entry_types.py is the one this folds against. Worth aligning the section 5
    table with it so the next reader does not fold the wrong key.

@chenmingwei23
chenmingwei23 requested review from a team and CrysisDeu September 17, 2026 16:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 90be3db63833f31fc5bdee306d1e21b411f7bb42 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Every "incremental" read still re-parses the whole log from the start, so per-turn push and per-GET cost grow linearly with session history.

Watch

iter_from(seq) walks every entry of every segment and skips entry.seq < seq (store.py:1716), so the publisher's per-burst pass and each projection/{name} GET are O(file), quadratic over the lifetime of exactly the long-lived sessions this feature exists to serve. The checkpoint's last_seq anchor is the right seam for a later seek, but nothing here measures or bounds the interim cost — it runs off-loop, coalesced, so it degrades quietly as CPU/IO rather than visibly.
Clears when: a measured push-pass and projection-GET cost at a realistic large log (e.g. tens of thousands of entries) is stated acceptable, or the store gains a seek-to-seq the fold uses.

Suggestions

Cache the resolved byte offset of each cached bundle's last_seq alongside it (invalidated with origin), so the hot push path stops paying the full re-scan without waiting for store-level seek work.

[DESIGN-REVIEWED] 90be3db

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 90be3db63833f31fc5bdee306d1e21b411f7bb42 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Eight public helpers ship with zero non-test consumers — groundwork for the deferred on-disk checkpoints, kept alive today only by the tests that call them.

Not justified as shipped

  • Item 8 — zero consumers: grep over src/kiro_crew for fold_status(|fold_usage(|fold_timeline(|fold_tools(|fold_approvals(|Checkpoint.from_dict|state_is_serializable|\.rendered() finds 0 consumers beyond the defining site — the handler uses only read_projection/fold_session, and the description defers the real consumer: "on-disk checkpoints are out (the state shape is ready for them)".
    Subtraction: delete the five one-line fold_<name> wrappers (tests call fold(name, entries)), Checkpoint.from_dict, state_is_serializable, and SessionProjections.rendered; reintroduce them with the checkpoint-file PR that reads them.
    Clears when: the deletion lands, or a non-test consumer (the on-disk checkpoint reader) counts ≥1.

What this change ships

Inventory (10 items) — 9 justified

Intent: let the session's on-disk crew log finally be read — the session half of RFC PR 2 (folds, paged reads, push). ADDITION, covered by rfc-append-only-ledger.md (in-progress on base; section 5 names these five projections, the from=&to= route, and the session_projection frame).

  1. A seq-range page of a session's crew log, oldest first, citations resolved to a verdict — justified
  2. One fold (status/usage/timeline/tools/approvals) served with the seq it folded through — justified
  3. Five folds with resumable, JSON-shaped checkpoints; incremental equals from-scratch — justified
  4. A session_projection frame pushed to owner sockets when a log grows (flag-gated; no SPA yet, declared) — justified
  5. The emitter gains a growth-listener registry, one wake per drained batch — justified
  6. Every pushed value and retained state is bounded, with omitted counts — justified
  7. A fold refuses an unknown required entry type (409) while a page shows it — justified
  8. Eight public helpers with no caller outside tests — zero consumers
  9. New crew-log-projection.md spec plus index and feature-map rows — justified
  10. Both reads and the push gated on the dashboard owner — justified

[FIRST-PRINCIPLES-REVIEWED] 90be3db

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 5.6 flagged blocking issues on 90be3db63833f31fc5bdee306d1e21b411f7bb42; Opus 4.8 adjudication downgraded every one of them to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/crew_log/projection.py:1188 -- Truncation collapses distinct omitted labels
return value[:TEXT_LIMIT] / if label in seen: return
Two names sharing a 200-character prefix -> truncation before deduplication -> names_omitted reports 1 instead of 2.
Anchor: a-bound-bounds-every-field-it-retains
Fix: Deduplicate omitted labels using a bounded digest of the original string.
[BLOCK-MERGE-DOWNGRADED] 90be3db
[GPT-REVIEWED] 90be3db

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

Confirmed the mechanism: _as_str truncates every label to 200 chars at projection.py:1188; _keyable (projection.py:1127) sends any label of length ≥200 to _note_omitted, whose dedup (projection.py:1150) collapses two labels that share a 200-char prefix into one — so names_omitted undercounts by one. Now weighing.

Harm rung: LOW.

  • Condition: two distinct tool/model/server names sharing an identical first 200 characters, both then routed to _note_omitted (projection.py:882→901, 1127, 1150). Real tool/model names are short; a 200-char shared prefix is a practically non-occurring input.

  • Recovery: the exact whole-session aggregate is unaffected — state["calls"] is incremented unconditionally at projection.py:915 before the row/omit path; only the names_omitted detail count drifts, and that field is already self-declared approximate via names_omitted_saturated (projection.py:1024, 1147). Consequence is a visible, self-correcting off-by-N in a diagnostic count, no data loss.

  • Remedy cost: the proposed bounded-digest dedup must thread the pre-truncation string past _as_str, whose entire design purpose is to bound memory at one coercion point (projection.py:1180-1184); it adds state and a mechanism serving only this path, carrying permanent reader load — clearly exceeding a cosmetic undercount under a non-occurring input.

    [ADJUDICATION] 90be3db total=1 uphold=0 downgrade=1
    DOWNGRADE F1 src/kiro_crew/crew_log/projection.py:1188 reason=disproportionate-remedy
    [GPT-ADJUDICATED] 90be3db

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 90be3db63833f31fc5bdee306d1e21b411f7bb42: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 90be3db63833f31fc5bdee306d1e21b411f7bb42 — this comment is updated in place on each push.

Review details

I have enough to falsify the sole candidate.

Falsification of CANDIDATE 1 — the claim is that context_by_source and by_decision grow unbounded, breaking the module's "every value is bounded, in count and size" invariant.

The invariant inconsistency is real at the code level: context_by_source keys label raw (no _keyable, no size cut, no count budget) and by_decision caps key size via _as_str but has no count ceiling — unlike by_model, by_name, etc. But the finding needs a concrete input occurring in practice with an observable wrong outcome.

  • context/composed sources[].kind is produced by emit.on_context_composed from context_blocks.split_blocks(), whose labels are a fixed ~37-entry constant vocabulary (_MARKERS, plus unclassified/your_message/other). The kind is never derived from prompt content — it is a hardcoded classifier label. So in live operation context_by_source is bounded by construction (~37 keys, all short constants).
  • approval/decided has no emitter at all — the projection doc itself states "No emitter writes these types yet," so by_decision has zero live producers.

The only way either map grows without bound is an attacker-planted or corrupted log line carrying many long, distinct kind/decision values. The candidate's own confidence is "low" and it concedes it "could not find a caller that feeds attacker-controlled kind/decision values in normal operation." Its grounding is the module docstring's stated threat model, not an executed path with a real input — exactly the "if a caller were to" case Step 1 forbids. Criterion (a) fails at 80+.

No new groundable defect surfaced in projection.py, emit.py (the _notify_growth in finally/landed is correct), or the handler while falsifying.

No findings.

[OPUS-REVIEWED] 90be3db

Verdict parsed from the review's SHA-scoped output markers for commit 90be3db63833f31fc5bdee306d1e21b411f7bb42.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 90be3db63833f31fc5bdee306d1e21b411f7bb42: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch 5 times, most recently from 2d6b946 to eb35b52 Compare September 17, 2026 21:29
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch 2 times, most recently from cac1ef0 to 4906a35 Compare September 17, 2026 22:22
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Red Windows shards on the previous head eb35b528db, by owner.

main-owned, #11563 -- shard 6, four tests in test/test_session_control.py:
test_private_creator_cannot_change_memory[writer-canonical] and
[public-worker-canonical] (assert 'agent_store_mismatch' == 'memory_delegation_denied'),
plus test_create_refuses_unreadable_delegation_identity[OSError] and [ValueError].
This branch touches none of those files. The branch is now rebased onto main at
2627b0407c ("fix(session-control): authorize a created child on the store it is
bound to", #11593), which is expected to clear this family.

Neither mine nor #11563 -- shard 8,
test/test_workflows_snapshot_commit.py::test_cancelled_write_drains_before_new_snapshot,
a TimeoutError. My diff does not touch the workflows snapshot path. Reporting it
rather than adopting it; it looks like a timing flake and a re-run is the remedy.

Mine, and fixed -- shard 3,
test/test_crew_log_projection.py::test_status_tracks_a_lost_remote_placement. It
appended remote/placed, which main has dropped from the session vocabulary, so
the fold refused the type. The rebase removes that test along with the fold's
remote/* handling, matching main's current vocabulary.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch from 4906a35 to 9bcff68 Compare September 17, 2026 23:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch from 9bcff68 to 6bd8ff6 Compare September 17, 2026 23:53
@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch from 6bd8ff6 to 4556b5d Compare September 18, 2026 01:17
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Board classification for 4556b5d4b243565bff7aaf97e0d01fce65a92a75

86 lane names: 68 success, 12 skipped, 6 failure. mergeable true. Every red is accounted for below.

GPT 5.6 Review -- a publication defect on this head, not a finding

The verdict shown in the GPT comment is stale. The workflow prepended its own dated notice saying so: the body is pinned to 6bd8ff61b8e9acb7660eaa9c55eebc813e0435f4, the previous head. Both findings it carries are already fixed here:

  • projection.py:558 "teardown erases an in-flight turn" -- session/closed no longer clears open_turn (src/kiro_crew/crew_log/projection.py:555-563; the close records closed_at and close_reason only, and the comment states why the open turn is left standing).
  • server.py:4423 "flag-off startup eagerly installs the optional subsystem" -- install_crew_log_publisher returns None on env_flag_enabled(CREW_LOG_ENV) before importing the emitter or constructing a publisher.

GPT did complete a pass on this exact head. Run 35294740723, falsification pass, 02:42:12Z, 178,325 tokens:

FINDING -- src/kiro_crew/crew_log/projection.py:464 -- an unknown ignorable tail entry is
filtered by `known=KNOWN_TYPES`, so checkpoints return a stale `seq` -> Fix: advance
checkpoint sequence through skipped ignorable entries. (origin: validation)
[GPT-REVIEWED] 4556b5d4b243565bff7aaf97e0d01fce65a92a75

One finding, in the advisory form (FINDING --, not BLOCKING --), and no [BLOCK-MERGE] marker. So GPT's own completed judgment on this head carries zero blocking findings.

Why the lane is red anyway. Pass 1 (discovery) died on infrastructure, not on the diff:

ERROR: Reconnecting... 1/5 ... 5/5
ERROR: stream disconnected before completion: idle timeout waiting for SSE
tokens used 20,233
##[warning]GPT 5.6 review pass 1 did not complete (exit 1); the verdict will fail closed.

Both calls are required, so pass 2's completed verdict was discarded -- the log says The pass-2 verdict was not published because both calls are required. -- and Gate on findings failed closed on a missing marker: GPT 5.6 review did not complete for 4556b5d4b2 (no [GPT-REVIEWED] marker). The adjudication steps were skipped for the same reason, so there is no adjudication on this head; the total=1 uphold=1 block in the comment belongs to the superseded head.

The advisory finding, recorded so the override does not bury it

The mechanism is real. It is not a correctness defect.

iter_from(..., known=KNOWN_TYPES) skips an unknown type only when the writer marked it ignorable (store.py:1718-1719); anything else raises unknown_entry_type. So a trailing ignorable entry is never folded, reached stays below handle.last_seq, and the bundle reports the lower figure.

Nothing reads a wrong value from that. The push compares the same figure across two consecutive folds, so a tail of ignorable entries correctly produces no frame: the projection genuinely did not change. The next fold resumes at reached + 1 and re-derives the same value, so it converges rather than drifting. The real cost is a re-scan: a log ending in ignorable unknown entries re-reads that tail on every fold instead of resuming past it, bounded by the number of trailing ignorable entries, and reachable only once a newer writer adds a session type this reader predates.

Advancing the checkpoint seq past a skipped entry is the right change, but it belongs where the skip is observed. The store's iterator is the only place that can tell a skipped entry from an absent one, and that is a module this PR does not own. Recorded as follow-up rather than reaching into store.py at this review round.

First Principles -- CONCERNS, advisory

Item 6: the five fold_<name> wrappers and state_is_serializable have no non-test callers. Correct as measured. They are the batch half of the RFC section 5 surface, and the incremental form's equivalence to them is exactly what the from-scratch-equals-incremental test asserts, so deleting them removes that test's other operand. Left as shipped for the maintainer's call; it clears by deletion if the smaller surface is preferred.

Test shards -- main-owned

Reproduced on pristine origin/main by a sibling session; cited rather than re-derived. My diff is 12 files and contains none of these test files.

Lane Failing tests Owner
Backend Tests (3.12, 1), Backend Tests (Windows) (1) test_acp_launch_goldens main, #11635
Backend Tests (3.12, 2), Backend Tests (Windows) (2) test_backend_cards main, #11634
Coverage Gate aggregate of the four above main, downstream

Two further main-owned families from earlier heads of this PR are not red here: test_workflows_service::test_finished_run_busy_slot_queues_turn (#10964) and test_xdist_host_budget (#11618). A single TestSyncKillProviderTree failure seen earlier was a flake.

This head

121 tests pass in the two files this PR adds, 318 with test_crew_log_core.py. mypy, black, isort, comment-history and docs-lint clean. Design passed, Opus 4.8 reported no blocking findings, First Principles is advisory.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 4556b5d: Pass 1 died on infra (SSE idle timeout, 5 failed reconnects), so pass 2's completed verdict for this SHA was discarded and the gate failed closed on a missing marker. Pass 2 emitted [GPT-REVIEWED] for this SHA with one ADVISORY finding and no [BLOCK-MERGE]. The displayed blocking body is pinned to 6bd8ff6 and both its findings are fixed here. Evidence and advisory disposition: issuecomment-5724449087.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 4556b5d4b243565bff7aaf97e0d01fce65a92a75.

Pass 1 died on infra (SSE idle timeout, 5 failed reconnects), so pass 2's completed verdict for this SHA was discarded and the gate failed closed on a missing marker. Pass 2 emitted [GPT-REVIEWED] for this SHA with one ADVISORY finding and no [BLOCK-MERGE]. The displayed blocking body is pinned to 6bd8ff6 and both its findings are fixed here. Evidence and advisory disposition: issuecomment-5724449087.

This decision applies only to this commit. A new push requires a new judgment.

…erve them

The crew log is append-only, so every view of it is a fold. This adds the
session side panel's five folds (status, usage, timeline, tools, approvals),
the two reads that serve them, and the frame that pushes one when the file
grows -- the backend folds and cuts pages, the client renders and pages.

The incremental form is the primitive and the whole-file form wraps it, so a
resumed answer and a from-scratch answer come out of one implementation. A
checkpoint is (name, last_seq, state), JSON-serializable, and a replayed entry
is refused rather than double-counted. Absent credits and tokens are never
read as zero: each total rides beside the count of turns that reported one.
An open turn and an unmatched tool call are reported open, never closed with
an invented outcome.

A page and a fold take opposite postures on an unknown type, deliberately: the
fold declares its vocabulary so a required unknown line refuses instead of
skewing a total, while the page declares none so one unfamiliar line cannot
hide the history in front of it.
…nly within one file

A cap on HOW MANY values a fold retains bounds nothing on its own, because each of
those values is a string off the wire. Every retained string is cut to TEXT_LIMIT
at the point the fold coerces it: an approval's tool and reason, a decision, a
per-tool server, a tool or model name, and the status echoes of agent, owner, slot,
cwd, model, provider, stop reason and error. A field whose ABSENCE is meaningful
keeps it -- a close reason, a stop reason and an error read as null when unset
rather than as a reason of no characters.

Cutting is safe only for a string that never distinguishes one thing from another.
A string used as a KEY is refused instead: one sitting exactly at TEXT_LIMIT cannot
be told apart from one that was cut, so keying on it would put two unrelated tools
or models in one row reporting each other's totals -- a wrong answer rather than a
big value. tools.by_name, usage.by_model and a row's server list give no detail to
a label at that length; it goes where a label past the count budget goes, counted
in totals that stay exact and reported as omitted detail. An IDENTITY is refused at
ID_LIMIT for the same reason, and both sides of a pair coerce it identically so a
call and its completion agree on what an identity is.

A count of omitted detail counts THINGS, not the events that mention them: a tool
name reaches that path from its call and again from its completion, a model once
per turn it ran, and a server once per call through it. All three counts share one
bookkeeper that dedupes against a capped list, stops at the budget rather than
climbing past the number of labels that exist, and exposes
names_omitted_saturated / models_omitted_saturated / servers_omitted_saturated to
say the figure is a floor rather than a total.

A seq is only comparable WITHIN one file, and two places had to learn that. The
push skips a projection whose checkpoint has not moved, but fold_session refuses a
bundle whose origin does not match the file and rebuilds from the start, so a log
removed and recreated can come back at the same terminal seq carrying different
values; the push compares the bundle's origin first and treats every projection of
a rebuilt bundle as new. And a page reports the tail it OBSERVED: a store handle's
last_seq is its own cached figure, authoritative only for its own appends, so a
writer growing the file after a reader handle opened was invisible to it and a page
could return rows up to `to` while reporting that nothing follows.

The push is a per-process singleton, so a restart inside one process rebinds it.
Rebinding only the loop left it holding the retired dashboard state, counting the
old hub's sockets and sending every frame to a room nobody is in. The rebind
repoints the loop and the state and clears the scheduling flags, which belong to
the loop going away: a timer armed there never fires and a flush marked in flight
there never finishes, so a stale flag silenced the publisher permanently.

Installing the push is gated on KIROCREW_CREW_LOG, and gated BEFORE the emitter is
imported. start_dashboard calls the installer unconditionally, so asking the
emitter whether it is enabled would import it on every disabled launch -- the cost
the flag exists to avoid, not a check of it. The variable's name is spelled in the
handler and a test pins it against the emitter's own constant so the two cannot
drift.

A close does not close a turn. A session cut off mid-turn writes session/closed
with no turn/completed, and clearing the open turn asserted that the turn finished
when nothing recorded it doing so, erasing the one fact a reader wants from that
log. Only turn/completed closes a turn, which is the rule the render already
stated.

A cold fold does not materialize the whole log: the pass is taken
FOLD_CHUNK_ENTRIES at a time, and each checkpoint takes only the part of a chunk it
has not consumed, so one chunk serves five folds sitting at different seqs. The
lazy module accessor is annotated ModuleType, because a module object is not a type
and the type checker refuses it.

Tests: 121 green across the two files, 318 with the crew-log core suite. Fourteen
properties are mutation-verified, each turning a test red: keys made
always-keyable, omitted counted per appearance, the id length guard removed, label
truncation removed, the server label retained raw, a status string retained raw,
the chunk bound raised past the log, page metadata taken from the cached figure,
origin dropped from the change check, the state left unbound on rebind, the
scheduling flags left set, the boot flag gate removed, the flag name drifted from
the emitter's, and the close clearing the turn again. mypy, black, isort,
comment-history and docs lint clean on the touched paths.
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-projection branch from 4556b5d to 90be3db Compare September 18, 2026 05:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever
iamwhatever merged commit 93cbfca into main Sep 18, 2026
88 checks passed
@iamwhatever
iamwhatever deleted the feat/crew-log-projection branch September 18, 2026 07:01
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…erve them (kirodotdev#11558)

* feat(crew-log): fold a session's crew log into five projections and serve them

The crew log is append-only, so every view of it is a fold. This adds the
session side panel's five folds (status, usage, timeline, tools, approvals),
the two reads that serve them, and the frame that pushes one when the file
grows -- the backend folds and cuts pages, the client renders and pages.

The incremental form is the primitive and the whole-file form wraps it, so a
resumed answer and a from-scratch answer come out of one implementation. A
checkpoint is (name, last_seq, state), JSON-serializable, and a replayed entry
is refused rather than double-counted. Absent credits and tokens are never
read as zero: each total rides beside the count of turns that reported one.
An open turn and an unmatched tool call are reported open, never closed with
an invented outcome.

A page and a fold take opposite postures on an unknown type, deliberately: the
fold declares its vocabulary so a required unknown line refuses instead of
skewing a total, while the page declares none so one unfamiliar line cannot
hide the history in front of it.

* fix(crew-log): bound every string a fold retains, and compare a seq only within one file

A cap on HOW MANY values a fold retains bounds nothing on its own, because each of
those values is a string off the wire. Every retained string is cut to TEXT_LIMIT
at the point the fold coerces it: an approval's tool and reason, a decision, a
per-tool server, a tool or model name, and the status echoes of agent, owner, slot,
cwd, model, provider, stop reason and error. A field whose ABSENCE is meaningful
keeps it -- a close reason, a stop reason and an error read as null when unset
rather than as a reason of no characters.

Cutting is safe only for a string that never distinguishes one thing from another.
A string used as a KEY is refused instead: one sitting exactly at TEXT_LIMIT cannot
be told apart from one that was cut, so keying on it would put two unrelated tools
or models in one row reporting each other's totals -- a wrong answer rather than a
big value. tools.by_name, usage.by_model and a row's server list give no detail to
a label at that length; it goes where a label past the count budget goes, counted
in totals that stay exact and reported as omitted detail. An IDENTITY is refused at
ID_LIMIT for the same reason, and both sides of a pair coerce it identically so a
call and its completion agree on what an identity is.

A count of omitted detail counts THINGS, not the events that mention them: a tool
name reaches that path from its call and again from its completion, a model once
per turn it ran, and a server once per call through it. All three counts share one
bookkeeper that dedupes against a capped list, stops at the budget rather than
climbing past the number of labels that exist, and exposes
names_omitted_saturated / models_omitted_saturated / servers_omitted_saturated to
say the figure is a floor rather than a total.

A seq is only comparable WITHIN one file, and two places had to learn that. The
push skips a projection whose checkpoint has not moved, but fold_session refuses a
bundle whose origin does not match the file and rebuilds from the start, so a log
removed and recreated can come back at the same terminal seq carrying different
values; the push compares the bundle's origin first and treats every projection of
a rebuilt bundle as new. And a page reports the tail it OBSERVED: a store handle's
last_seq is its own cached figure, authoritative only for its own appends, so a
writer growing the file after a reader handle opened was invisible to it and a page
could return rows up to `to` while reporting that nothing follows.

The push is a per-process singleton, so a restart inside one process rebinds it.
Rebinding only the loop left it holding the retired dashboard state, counting the
old hub's sockets and sending every frame to a room nobody is in. The rebind
repoints the loop and the state and clears the scheduling flags, which belong to
the loop going away: a timer armed there never fires and a flush marked in flight
there never finishes, so a stale flag silenced the publisher permanently.

Installing the push is gated on KIROCREW_CREW_LOG, and gated BEFORE the emitter is
imported. start_dashboard calls the installer unconditionally, so asking the
emitter whether it is enabled would import it on every disabled launch -- the cost
the flag exists to avoid, not a check of it. The variable's name is spelled in the
handler and a test pins it against the emitter's own constant so the two cannot
drift.

A close does not close a turn. A session cut off mid-turn writes session/closed
with no turn/completed, and clearing the open turn asserted that the turn finished
when nothing recorded it doing so, erasing the one fact a reader wants from that
log. Only turn/completed closes a turn, which is the rule the render already
stated.

A cold fold does not materialize the whole log: the pass is taken
FOLD_CHUNK_ENTRIES at a time, and each checkpoint takes only the part of a chunk it
has not consumed, so one chunk serves five folds sitting at different seqs. The
lazy module accessor is annotated ModuleType, because a module object is not a type
and the type checker refuses it.

Tests: 121 green across the two files, 318 with the crew-log core suite. Fourteen
properties are mutation-verified, each turning a test red: keys made
always-keyable, omitted counted per appearance, the id length guard removed, label
truncation removed, the server label retained raw, a status string retained raw,
the chunk bound raised past the log, page metadata taken from the cached figure,
origin dropped from the change check, the state left unbound on rebind, the
scheduling flags left set, the boot flag gate removed, the flag name drifted from
the emitter's, and the close clearing the turn again. mypy, black, isort,
comment-history and docs lint clean on the touched paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants