Skip to content

fix(session-ledger): answer a refused crew log fold with a code, not a bare 500 - #16666

Merged
iamwhatever merged 1 commit into
mainfrom
fix/ledger-record-crewlog-error
Oct 4, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/ledger-record-crewlog-error

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: an agent whose session-ledger write is refused is told what refused it
and whether a retry can help, instead of losing the update to an opaque 500.

session_ledger_record answers a bare, codeless HTTP 500:

Error: 500 Internal Server Error

Server got itself in trouble

Measured in local session transcripts: 61 calls across 20 sessions, all inside
one day, 01:43:41Z to 19:14:15Z. The failures are explained by neither the
caller nor a dead gateway:

  • Successes interleave with failures in the same session. One session had 39 OK
    and 4 failures; another alternates OK -> 500 -> OK -> 500 inside 22 minutes.
  • Payload size does not separate the populations: failing 216-2800 characters,
    succeeding 402-3079.
  • The two calls immediately before the first failure, same session, were the
    typed event: exceeds max length 2000 refusal. The caller trimmed to 1976
    and the next call 500'd, so the failure is downstream of validation.

Server got itself in trouble is aiohttp's default body for an exception that
left the handler unhandled.

Why it matters

A session ledger update is what survives context compaction, so losing one
costs an agent its own record of where the work stands.

The answer carries no code, so the caller cannot tell what failed or whether
retrying could ever help. The transcripts show the consequence: most affected
sessions hold exactly two of these failures - call, retry once, give up. The
update is gone, and the next cycle resumes from a record that never received
it.

The route's stated contract is typed answers. A crew-log refusal is the failure
class a crew-log-backed record is most likely to meet, and it was the one class
that escaped untyped.

Not a goal

  • Does not add a catch-all. An exception class nobody has reasoned about still
    surfaces as a server fault rather than being silently relabelled as a
    storage refusal.
  • Does not name IndeterminateAppend, the crew log's other bare Exception
    subclass. The ledger hands its append to the writer job rather than running
    it in the caller's frame, so it is not reachable from this route on the
    evidence here, and a branch for an unreproduced path would be speculation.
  • Does not touch the read route, which never could leak this (see below).
  • Does not change what the crew log considers damage, or how advance()
    refuses it.

What changed (motivation -> approach -> change)

Symptom to root cause to change:

  1. The state record is a projection of the session's crew log, so every write
    folds that log before appending to it:
    record_update -> session_ledger._fold_checkpoint ->
    crew_log.projection.fold_slot_warm -> advance().
  2. advance() refuses a log whose seq does not advance - the damage an
    append-only writer cannot itself produce - by raising CrewLogError
    (src/kiro_crew/crew_log/errors.py:91).
  3. CrewLogError subclasses Exception directly, not ValueError and not
    OSError, so none of the route's four branches named it:
    LedgerUnavailable 409, LedgerEntryTooLarge 413, ValueError 400,
    OSError 503.
  4. An unnamed exception out of an aiohttp handler is answered by aiohttp
    itself, which is the bare 500 above.

The change: api_session_ledger_record names CrewLogError and answers 409
crew_log_unreadable, carrying the crew log's own code in the message.

The approach was chosen over the alternatives because it is already the
repository's answer for this refusal. The sibling ledger route over the same
store does exactly this:

# src/kiro_crew/dashboard/handlers/work_ledger.py:1648
except CrewLogError as exc:
    ...
    return _refuse_409("crew_log_unreadable", f"the crew log could not be read: {exc}")

So no new status, code or vocabulary is introduced. 409 rather than 503 because
a refused fold is not a busy filesystem, so "try again" would be false; rather
than 400 because the request is well formed and nothing the caller sends
changes the answer.

Backwards compatibility

Compatible: the new branch is reached only where the route previously raised
through to aiohttp's 500 handler, so no request that succeeds today changes its
answer, and the four existing typed branches are untouched and ordered as
before.

Removes nothing: the diff adds one except clause and one import. It deletes
no key, record, validator or public function, hides no path, and prunes no
data. The only behaviour it replaces is an unhandled crash, which no caller can
hold a contract on - a 500 with an empty body carries no field or code for a
reader to depend on.

Tests

One test added, test_route_record_answers_a_typed_refusal_when_the_crew_log_refuses
in test/test_session_ledger.py, driving the real store and the real route
rather than a patched raise:

  1. create the unit, record one entry through the route, assert 200 - so the
    test cannot pass by the write never working at all;
  2. append a byte-identical copy of the newest record, so the last seq appears
    twice, which is the case advance() refuses with bad_data;
  3. drop the warm fold, so the damage is met by a fold that reads the file
    rather than answering from its cached cell;
  4. assert 409, code == "crew_log_unreadable", and bad_data in the message.

It locks in that a crew-log fold refusal reaches the caller as a code, and that
the code is the crew log's own.

RED on the base commit, before the source change:

FAILED test/test_session_ledger.py::test_route_record_answers_a_typed_refusal_when_the_crew_log_refuses
  kiro_crew.crew_log.errors.CrewLogError: entry 1 is at or below the previous...

That exception escaping the handler is the reported bare 500.

GREEN with the fix, with the two files that own this path run whole:

test/test_session_ledger.py              60 passed in 6.36s
test/test_session_ledger_projection.py  114 passed in 4.63s

Gates with the worktree's own .venv/bin/ tools: black clean, mypy
Success: no issues found in 1 source file, comment-history gate passed after
its own selftest (21 flagged probes, 15 clean probes).

Manual verification

Verified on a pod built from this worktree: the real route, over HTTP, against
a real crew log damaged the same way the test damages it. Output in the PR
comments.

The read route needs none and gets none: session_ledger.read_state folds
inside except Exception and answers the empty record, so GET /api/session-ledger cannot reach this path.

Related Issues

Fixes #16664

Pattern harvest

Rule candidate: review-prompt

Pattern: a handler enumerates exception types by the module it calls directly,
missing the refusal type of the store that module reads through - so the one
failure class most native to the backing store is the one that escapes untyped.

The generalizable check is cheap to state: for each except ladder in a
dashboard handler, list the exception types every transitive callee raises, and
flag any that subclass Exception directly rather than ValueError/OSError.
Here the sibling route over the same store already had the branch, so the
divergence between two handlers on one store is itself the smell. Same class as
#7790, fixed earlier for a different component.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) - N/A, no documented behaviour changes
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 4, 2026 05:24
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of c9c3523567d9c77e864e171990e08120aa317cfc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The field failures came and went (OK → 500 → OK in one session), but the fix types them as permanent damage and never finds why the fold re-reads entries.

Watch

  • Root cause is still open. The description says "Successes interleave with failures in the same session." Damaged bytes on disk would refuse every write after the first failure, not alternate with successes. projection.advance's own docstring names the other cause: "a caller that re-read a page it already folded". Here that would mean the warm fold or the writer job handing advance() entries it has already folded, which is a race in our code. The test plants duplicate bytes on disk and clears the warm cell with forget_slot_folds(), so it reproduces a different cause from the one in the transcripts. Result: the 500 becomes a 409, but the same writes still get refused at the same rate.
    Clears when: either the PR shows the field failures were real on-disk damage, or a follow-up issue tracks the warm-fold/writer race behind the interleaved refusals and is linked here.
  • The 409 tells the agent to stop retrying when retrying worked. The description says 409 means "try again would be false." In the transcripts, a later call in the same session did succeed. An agent that trusts the new code gives up even sooner, and the ledger update is lost anyway.
    Clears when: the 409 answer and its comment no longer claim a retry can't help, or the PR shows that every refused fold is permanent.

[DESIGN-REVIEWED] c9c3523

Description read

[DESCRIPTION-READ] 29efa434a2d50cf5eb30a539d89b1440e14a3fab35cac9e53d80773dc7bb2ebf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of c9c3523567d9c77e864e171990e08120aa317cfc — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that the field 500s really were CrewLogError. The test reproduces a fake duplicate-seq log, and the intermittent OK→500→OK pattern seen in real sessions is still unexplained.

What this change ships

Inventory (3 items) — 3 justified

Intent: an agent whose session-ledger write is refused learns why, and whether retrying can help, instead of getting a bare 500. This is a FIX (#16664).

  1. When the crew log refuses a ledger write, the caller now gets 409 with the code crew_log_unreadable and the crew log's own reason, instead of aiohttp's bare 500. This matches the existing branch at dashboard/handlers/work_ledger.py:1648. — justified
  2. The gateway logs a warning naming the slot and the crew log's code when this refusal happens. — justified
  3. A regression test drives the real store and the real route against a damaged log, and it fails on base. — justified

[FIRST-PRINCIPLES-REVIEWED] c9c3523

Description read

[DESCRIPTION-READ] 5419f07a4b1534186265fe2526be7ad6f75f2138fc909ded1345e22ff8b2b99e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c9c3523567d9c77e864e171990e08120aa317cfc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c9c3523

Verdict parsed from the review's SHA-scoped output markers for commit c9c3523567d9c77e864e171990e08120aa317cfc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c9c3523567d9c77e864e171990e08120aa317cfc: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c9c3523567d9c77e864e171990e08120aa317cfc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c9c3523

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c9c3523567d9c77e864e171990e08120aa317cfc: <one-sentence reason>

…a bare 500

The state record is a projection of the session's crew log, so every
session_ledger_record write folds that log before appending to it:
record_update -> _fold_checkpoint -> projection.fold_slot_warm -> advance().

advance() refuses seq damage by raising CrewLogError, which subclasses neither
ValueError nor OSError, so none of the route's four typed branches named it. An
unnamed exception out of an aiohttp handler is answered as a bare
"500 Internal Server Error / Server got itself in trouble" - a body with no
code, so the caller cannot tell what failed or whether a retry could ever clear
it. Local session transcripts show 61 such calls across 20 sessions in one day,
interleaved with successes in the same session and uncorrelated with payload
size.

The route now names CrewLogError and answers 409 crew_log_unreadable, carrying
the crew log's own code in the message. The import is inside the handler, not at
module scope: this module is on the gateway's boot path and the crew log's
storage package is optional behind its own flag, so AUTOSDE's
no-new-work-on-gateway-boot-path rule gates the import itself, and the sibling
crew log routes import it per function for that reason. That is the answer the sibling
work-ledger route already gives for the same refusal on the same store, so this
adds no new vocabulary. 409 rather than 503: a refused fold is not a busy
filesystem, so "try again" would be false.

The read route is unaffected and unchanged - read_state folds inside
except Exception and answers the empty record, so it cannot leak this.

Test drives the real store and the real route rather than a patched raise: it
records one entry, appends a byte-identical copy of the newest record so the
last seq appears twice, drops the warm fold so the damage is met by a fold that
reads the file, and asserts a coded 409. It fails on the base commit with
CrewLogError escaping the handler.

Fixes #16664
@chenmingwei23
chenmingwei23 force-pushed the fix/ledger-record-crewlog-error branch from 4c35d3a to c9c3523 Compare October 4, 2026 05:40
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Manual verification: pod output

Pod built from this branch at c9c3523567, serving on its own port with its own
data directory. The crew log written and damaged below is that pod's own file.

Part A: the pod's own running gateway, over its real socket

request answer
GET /api/health 200
POST /api/session-ledger/record, unknown session key 400 unknown_session
same, recognized key with no live crew log unit 409 crew_log_unavailable

Both refusals are typed and carry a code; neither is a 500.

The pod's own process cannot be driven all the way into the new branch from
outside, and the reason is worth stating: reaching it needs a live ACP provider
for the slot, and no HTTP route allocates one - a provider appears only when a
session runs a turn through the WebSocket. Part B supplies that single
precondition and leaves everything below the recognition gate real.

Part B: the real route over a real socket, pod interpreter and pod store

Run against the pod's data home with the branch's own interpreter. The request
travels over a real TCP socket into the real api_session_ledger_record.
Session recognition is bypassed the way this repository's own route tests bypass
it.

== pod store: .../crew-log/sessions/pod-verify-16666-054015-4d1d5192/log.jsonl
== real HTTP server on http://127.0.0.1:43939

-- 1. undamaged write over real HTTP
   HTTP 200  {"ok": true, "durable": true, "state": {"schema": 1,
              "goal": "pod verify 16666", "next": "damage the log next", ...

-- 2. damage the pod's crew log (newest record copied: seq repeats)
   appended 168 bytes; file now 559 bytes

-- 3. write again: the fold now refuses
   HTTP 409
   {"error": "the crew log could not be read (bad_data): entry 1 is at or below
     the previously read entry's seq 1; a duplicate or backward seq is damage
     the append-only writer cannot produce",
    "code": "crew_log_unreadable"}

-- 4. the read route still answers (it must not have been changed)
   HTTP 200  {"state": {"schema": 1, "goal": "", ...

PASS: a refused fold answers 409 crew_log_unreadable over real HTTP;
      the body names bad_data; the read route is unaffected.

Step 1 is a control and it earned its place. A second run that reused the same
session key answered 409 at step 1 instead of 200, because the earlier run's
damaged unit is still joined to that slot, so the fold refused before the fresh
write ever ran. Without asserting the control, that run would have reported a
pass at step 3 while proving nothing about the write path. The slot is now
stamped per run, and the output above is a clean 200 then 409.

Part C: the boot-path gate

Backend Tests (Windows) (4) on the previous head caught a real defect in this
change, and it is fixed here rather than retried:

test_a_flag_off_launch_does_not_load_the_storage_subsystem
  AssertionError: a flag-off launch imported storage: ['kiro_crew.crew_log.errors']

This module is on the gateway's boot path and the crew log's storage package is
optional behind its own flag, so AUTOSDE's no-new-work-on-gateway-boot-path
rule gates the import itself, not merely the calls. The import moves inside the
handler, above the try so the except clause has the name bound, which is how
the sibling crew log routes import it. Green now:

pytest test/test_crew_log_emit.py -k flag_off_launch
  2 passed

Remaining red

Issue Gate is red because #16664 carries no triage verdict label
(auto-fixable / needs-investigation / needs-human). Those are written by
the maintainer-operated pipeline outside this repository, so there is nothing in
this PR to change; the gate re-runs on a label event once triage reaches the
issue.

Backend Tests (Windows) (1) failed on the previous head inside
test_spawn_approval_crew_log.py, which this diff does not touch. That is the
open flake in #16621, whose own report says it keeps readiness red on PRs that
do not touch it.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 4, 2026 08:03

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: session_ledger route catches CrewLogError (subclass of neither ValueError nor OSError) and answers 409 crew_log_unreadable instead of a codeless aiohttp 500, matching the sibling work-ledger route; 1 handler + 1 test file.

@iamwhatever
iamwhatever merged commit e8794d9 into main Oct 4, 2026
143 of 145 checks passed
@iamwhatever
iamwhatever deleted the fix/ledger-record-crewlog-error branch October 4, 2026 08:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

session_ledger_record answers a bare untyped 500 when the crew log fold refuses (61 calls, 20 sessions)

2 participants