Repository navigation
fix: start the log append deadline after the open, not before it - #12092
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the mutation table's claim: both new tests red on base with the quoted errors, since that is the defect's whole provenance. What this change shipsInventory (4 items) — 4 justifiedIntent: stop a slow first open from spending the append budget and leaving an empty, unparseable day file — a FIX.
[FIRST-PRINCIPLES-REVIEWED] ed8c559 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] ed8c559 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsThe diff is a narrow, correct fix: the append deadline is computed after No findings. [OPUS-REVIEWED] ed8c559 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
7e53927 to
80a8384
Compare
|
Attribution for the red lanes on head This branch changes four files: Evidence that the failures belong to
Fixes for both classes are already in flight, so no change here is the right
Once those land this branch will be rebased onto the fixed Local verification on this head: |
The append budget is documented to span lock acquisition and the write retries. It was armed above _open_log, so the create-and-pin it cannot cancel spent it: an open slower than the whole budget raised TimeoutError at the lock with the day file ALREADY created by that open, and decisions.log.append swallows the failure, so a reader found a file that exists and holds nothing. Closes #12077
80a8384 to
ed8c559
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: moves the append deadline to after _open_log so a slow open no longer raises with the day file already created and empty (GH-12077). Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/platform-compat.md.
…odotdev#12092) The append budget is documented to span lock acquisition and the write retries. It was armed above _open_log, so the create-and-pin it cannot cancel spent it: an open slower than the whole budget raised TimeoutError at the lock with the day file ALREADY created by that open, and decisions.log.append swallows the failure, so a reader found a file that exists and holds nothing. Closes kirodotdev#12077
Problem / Motivation
Backend Tests (Windows)goes red intermittently on branches that never touchedthe decision log, with the same shape every time: a test appends a row, reads the
day file back, and
json.loadsraisesJSONDecodeError: Expecting value: line 1 column 1 (char 0)because the file it just read holds nothing. The file exists andis empty. Re-running that shard on the identical head passes.
Reported against
test/test_decisions_log.py::TestAppend::test_an_unserialisable_value_is_survived(run 35462706451, job
Backend Tests (Windows) (3)). The same append, and the samedecision log append deadline expiredmessage, has also reddened the row readers intest/test_decisions_gate.py.Why it matters
Two costs. In CI, a PR author sees a red Windows shard in a file they did not
change, re-runs a 15-minute shard, and that noise hides a real Windows red when one
arrives. In the product the outcome is worse than a missing log:
appendisbest-effort and swallows its own failure, so an operator is left a day file that
exists and holds nothing, and any reader of a JSONL log fails to parse it instead
of reading zero rows.
What changed (motivation -> approach -> change)
Symptom: a day file that exists and is empty, left by an append that reported
nothing to its caller.
Root cause:
append_linearmed its budget one line too early._open_logresolves the anchor, creates and pins the log directory, opens the leafand fstats it. On POSIX that open carries
O_CREAT; on Windows_win_openpassesOPEN_ALWAYS. Either way the leaf is created BY that open. All of the open's costwas charged to the 0.5s budget, so on a loaded worker
_remaining(deadline)raisedTimeoutErrorat the lock with the file already created and no row written.decisions.log.appendcatches that, logs a warning and returns, and the empty filestays behind for the next reader.
The change moves the deadline inside the
with, so it spans the lock wait and thewrite retries and nothing else. That is what the contract already says it spans
(
docs/system-specs/common/platform-compat.md: "Lock acquisition and retries sharea finite deadline"), and it is the shape the sibling writer in this tree already
uses:
session_ledger.pyarms its lock deadline "after the pre-lock syscalls (whichit cannot bound)". The budget's value and its
TimeoutErrorare unchanged, and thebudget keeps its own explicit test.
Sibling write paths checked, rather than only the one spelling in the report:
platform_log_append.append_lineis the only append in the tree that carries adeadline at all.
crew_log/store.py's_append_lineand_append_linescarrynone, and
session_ledger.pyis already correct.Backwards compatibility
Compatible: a successful append is byte-identical, the budget keeps its value and
its exception type, and the only thing that changes is which work is charged to it.
An append that genuinely cannot take the lock within the budget still raises.
Tests
test/test_platform_log_append.py::test_the_open_is_not_charged_to_the_append_deadlinewraps the real
_pin_and_open_leaf, so the leaf is genuinely created and the openthen overruns the whole budget, and asserts the row still lands.
test/test_decisions_log.py::TestAppend::test_a_slow_open_leaves_no_empty_day_filedrives the same overrun through
decisions.log.append, which swallows failures,and asserts the day file reads back as a row rather than as
''.Both fail on this branch's base with the reported errors,
TimeoutError: decision log append deadline expiredandJSONDecodeError: Expecting value: line 1 column 1 (char 0), the second preceded by the swalloweddecisions: could not append log row: decision log append deadline expired.Mutation-checked, restoring
platform_log_append.pyfrom a copy taken before anymutation and aborting if the unmutated baseline is red:
_remainingstops raisingtest_failed_partial_row_rolls_back_before_next_append[interrupt]Different counts on different test names, so the two new assertions pin this
behaviour specifically rather than breaking as a group.
Local selection (Linux,
-n0):test/test_platform_log_append.py,test/test_decisions_log.py,test/test_decisions_gate.py, giving 156 passed and7 skipped. Gates run: black, isort, flake8,
scripts/docs-lint.sh,scripts/check_brand_name.py,scripts/check_comment_history.py,scripts/check_loop_bound_locks.py.Manual verification
N/A, unit coverage is sufficient. The failure is a timing window inside one
function, and both tests reproduce it deterministically by making the open overrun
the budget instead of waiting for a loaded host to do it.
Related Issues
Closes #12077
Pattern harvest
Rule candidate: review-prompt
Pattern: a deadline armed before setup work it cannot cancel, so the setup spends
the budget meant for the operation and the failure lands after a side effect the
setup already created (here, the log file) that nothing rolls back.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)