Skip to content

fix: start the log append deadline after the open, not before it - #12092

Merged
iamwhatever merged 1 commit into
mainfrom
fix/decisions-log-empty-day-12077
Sep 19, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/decisions-log-empty-day-12077

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backend Tests (Windows) goes red intermittently on branches that never touched
the decision log, with the same shape every time: a test appends a row, reads the
day file back, and json.loads raises JSONDecodeError: Expecting value: line 1 column 1 (char 0) because the file it just read holds nothing. The file exists and
is empty. Re-running that shard on the identical head passes.

Reported against test/test_decisions_log.py::TestAppend::test_an_unserialisable_value_is_survived
(run 35462706451, job Backend Tests (Windows) (3)). The same append, and the same
decision log append deadline expired message, has also reddened the row readers in
test/test_decisions_gate.py.

Why it matters

Two costs. In CI, a PR author sees a red Windows shard in a file they did not
change, re-runs a 15-minute shard, and that noise hides a real Windows red when one
arrives. In the product the outcome is worse than a missing log: append is
best-effort and swallows its own failure, so an operator is left a day file that
exists and holds nothing, and any reader of a JSONL log fails to parse it instead
of reading zero rows.

What changed (motivation -> approach -> change)

Symptom: a day file that exists and is empty, left by an append that reported
nothing to its caller.

Root cause: append_line armed its budget one line too early.

deadline = time.monotonic() + _APPEND_TIMEOUT_SECONDS
with _open_log(path) as fd:
    with platform_compat.file_lock(fd, exclusive=True, timeout=_remaining(deadline)):

_open_log resolves the anchor, creates and pins the log directory, opens the leaf
and fstats it. On POSIX that open carries O_CREAT; on Windows _win_open passes
OPEN_ALWAYS. Either way the leaf is created BY that open. All of the open's cost
was charged to the 0.5s budget, so on a loaded worker _remaining(deadline) raised
TimeoutError at the lock with the file already created and no row written.
decisions.log.append catches that, logs a warning and returns, and the empty file
stays behind for the next reader.

The change moves the deadline inside the with, so it spans the lock wait and the
write retries and nothing else. That is what the contract already says it spans
(docs/system-specs/common/platform-compat.md: "Lock acquisition and retries share
a finite deadline"), and it is the shape the sibling writer in this tree already
uses: session_ledger.py arms its lock deadline "after the pre-lock syscalls (which
it cannot bound)". The budget's value and its TimeoutError are unchanged, and the
budget keeps its own explicit test.

Sibling write paths checked, rather than only the one spelling in the report:
platform_log_append.append_line is the only append in the tree that carries a
deadline at all. crew_log/store.py's _append_line and _append_lines carry
none, and session_ledger.py is already correct.

Backwards compatibility

Compatible: a successful append is byte-identical, the budget keeps its value and
its exception type, and the only thing that changes is which work is charged to it.
An append that genuinely cannot take the lock within the budget still raises.

Tests

  • test/test_platform_log_append.py::test_the_open_is_not_charged_to_the_append_deadline
    wraps the real _pin_and_open_leaf, so the leaf is genuinely created and the open
    then overruns the whole budget, and asserts the row still lands.
  • test/test_decisions_log.py::TestAppend::test_a_slow_open_leaves_no_empty_day_file
    drives the same overrun through decisions.log.append, which swallows failures,
    and asserts the day file reads back as a row rather than as ''.

Both fail on this branch's base with the reported errors, TimeoutError: decision log append deadline expired and JSONDecodeError: Expecting value: line 1 column 1 (char 0), the second preceded by the swallowed
decisions: could not append log row: decision log append deadline expired.

Mutation-checked, restoring platform_log_append.py from a copy taken before any
mutation and aborting if the unmutated baseline is red:

run result
baseline, unmutated 0 failed
deadline put back above the open 2 failed: exactly the two tests above
_remaining stops raising 1 failed: test_failed_partial_row_rolls_back_before_next_append[interrupt]
restored from the copy 0 failed

Different counts on different test names, so the two new assertions pin this
behaviour specifically rather than breaking as a group.

Local selection (Linux, -n0): test/test_platform_log_append.py,
test/test_decisions_log.py, test/test_decisions_gate.py, giving 156 passed and
7 skipped. Gates run: black, isort, flake8, scripts/docs-lint.sh,
scripts/check_brand_name.py, scripts/check_comment_history.py,
scripts/check_loop_bound_locks.py.

Manual verification

N/A, unit coverage is sufficient. The failure is a timing window inside one
function, and both tests reproduce it deterministically by making the open overrun
the budget instead of waiting for a loaded host to do it.

Related Issues

Closes #12077

Pattern harvest

Rule candidate: review-prompt
Pattern: a deadline armed before setup work it cannot cancel, so the setup spends
the budget meant for the operation and the failure lands after a side effect the
setup already created (here, the log file) that nothing rolls back.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested review from a team and krishdhasmana September 19, 2026 21:03
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 19, 2026
@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ed8c5591c30124c26fdbf49d22b12a0d4e81655c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed8c559

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed8c5591c30124c26fdbf49d22b12a0d4e81655c: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of ed8c5591c30124c26fdbf49d22b12a0d4e81655c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the mutation table's claim: both new tests red on base with the quoted errors, since that is the defect's whole provenance.

What this change ships

Inventory (4 items) — 4 justified

Intent: stop a slow first open from spending the append budget and leaving an empty, unparseable day file — a FIX.

  1. A first append whose open outruns the budget now writes its row instead of leaving an empty day file — justified
  2. The 0.5s budget no longer charges any of the open's own time; a slow open alone can no longer raise TimeoutError — justified
  3. platform-compat.md deadline paragraph now states the budget starts once the log is open — justified
  4. Two regression tests pin the row landing when the open overruns the whole budget — justified

[FIRST-PRINCIPLES-REVIEWED] ed8c559

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of ed8c5591c30124c26fdbf49d22b12a0d4e81655c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ed8c559

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed ed8c5591c30124c26fdbf49d22b12a0d4e81655c — this comment is updated in place on each push.

Review details

The diff is a narrow, correct fix: the append deadline is computed after _open_log returns rather than before it, so a slow-but-uncancellable create-and-pin/open no longer consumes the lock/retry budget. This strictly relaxes the timeout for the lock wait and write, cannot introduce a new timeout path, and the docstring + platform-compat doc are updated in the same commit. The two added tests monkeypatch the module-level _pin_and_open_leaf that _create_and_open resolves by name, are bounded, and their time.sleep is test code. I re-derived the call path and could not ground any concrete input → wrong outcome on the changed lines.

No findings.

[OPUS-REVIEWED] ed8c559

Verdict parsed from the review's SHA-scoped output markers for commit ed8c5591c30124c26fdbf49d22b12a0d4e81655c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed8c5591c30124c26fdbf49d22b12a0d4e81655c: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/decisions-log-empty-day-12077 branch from 7e53927 to 80a8384 Compare September 19, 2026 21:19
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Attribution for the red lanes on head 80a83842469f3c408f7a42d4a4e4990dc3680325: all of them are inherited from main, none is caused by this diff.

This branch changes four files: src/kiro_crew/platform_log_append.py,
test/test_platform_log_append.py, test/test_decisions_log.py,
docs/system-specs/common/platform-compat.md. git diff origin/main...HEAD --name-only returns exactly those four, and every file named by a failing test is
byte-identical to origin/main here.

Evidence that the failures belong to main:

  • main's own CI run on its current head 39892f5c10 (run 35469148028) finished
    failure with this set: Real Adapter Contract Tests, Backend Tests (3.12, 1),
    (3.12, 5), (3.12, 6), (3.12, 8), Backend Tests (Windows) (1), (5), (6),
    (8), Coverage Gate. That is the same set this PR shows.
  • This PR's first head was based on 1c7f963706, whose own main run 35465194443
    failed all 8 Linux and all 8 Windows shards. main has since reverted that commit
    (fix: resolve private store from binding when transcript lags #11947 reverted by revert: resolve private store from binding when transcript lags (#11947) #12091), and this branch was rebased onto the revert.
  • Coverage Gate is not an independent finding: its log says
    backend-test=failure -- failing closed, so it clears when the shards clear.
  • test/test_real_adapter_gate.py::test_the_tree_carries_the_guarded_tests_this_lane_exists_for
    fails locally on this branch as well (1 failed, 19 passed), in a file this branch
    does not touch.

Fixes for both classes are already in flight, so no change here is the right
response:

Once those land this branch will be rebased onto the fixed main and the board
re-measured on the fresh head.

Local verification on this head: test/test_platform_log_append.py and
test/test_decisions_log.py give 68 passed, 7 skipped with -n0. All review lanes
are green: GPT 5.6 Review, Opus 4.8 Review, First Principles Review, Design Review,
Security Scope Review, PR Hygiene, Comment History Gate.

The append budget is documented to span lock acquisition and the write retries. It was armed above _open_log, so the create-and-pin it cannot cancel spent it: an open slower than the whole budget raised TimeoutError at the lock with the day file ALREADY created by that open, and decisions.log.append swallows the failure, so a reader found a file that exists and holds nothing.

Closes #12077
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/decisions-log-empty-day-12077 branch from 80a8384 to ed8c559 Compare September 19, 2026 21:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 19, 2026 22:07

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: moves the append deadline to after _open_log so a slow open no longer raises with the day file already created and empty (GH-12077). Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/platform-compat.md.

@iamwhatever
iamwhatever merged commit 0b6738e into main Sep 19, 2026
87 checks passed
@iamwhatever
iamwhatever deleted the fix/decisions-log-empty-day-12077 branch September 19, 2026 22:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 19, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…odotdev#12092)

The append budget is documented to span lock acquisition and the write retries. It was armed above _open_log, so the create-and-pin it cannot cancel spent it: an open slower than the whole budget raised TimeoutError at the lock with the day file ALREADY created by that open, and decisions.log.append swallows the failure, so a reader found a file that exists and holds nothing.

Closes kirodotdev#12077
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky on Windows: the decisions log append leaves an empty day file (test_an_unserialisable_value_is_survived)

2 participants