Repository navigation
fix(snapshot): validate notification records the copy branch installs - #8576
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound core (read-once, abort parity with merge), but the ordering guarantee rests on a private cross-module seam, and the reader that turns corruption into deletion is untouched. Watch
Suggestions
[DESIGN-REVIEWED] 1fd5d0c |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/snapshot.py:2751 -- function-local False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All evidence gathered. Composing the review now. First-Principles-Verdict: CONCERNS The fix is real and derived, but it validates one of two install paths while the amplifier that destroys the file, and replace mode, both stay. What this change shipsIntent: stop a snapshot restore from silently installing notification bytes the live reader refuses and later destroys. This is a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 1fd5d0c |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsI've independently traced the safety-critical path: No findings. [OPUS-REVIEWED] 1fd5d0c Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
74993d0 to
3da7581
Compare
3da7581 to
7f199c5
Compare
7f199c5 to
6a0364d
Compare
6a0364d to
1ba8d9d
Compare
1ba8d9d to
079b80c
Compare
e50d320 to
f816314
Compare
Restoring a snapshot chose between two branches for notifications.jsonl and only one validated anything. With a live file present, _merge_notifications validated every source record's encoding and aborted on one it could not deliver intact. With no live file -- a fresh install or a first restore -- the restore took shutil.copy2, which validated nothing. A byte-exact copy is correct as a copy and that is the defect: it faithfully installed bytes the destination's own reader refuses. _load_notifications decodes the whole file inside one try that returns [], so one invalid byte costs every row, and the next _rewrite_notifications -- any delete, ack or clear -- persists that empty view over the file. Both call sites now go through _copy_notifications. It reads the source ONCE, whole, under a byte cap, validates the bytes it is holding, and writes them to a destination created O_CREAT|O_EXCL|O_WRONLY|O_APPEND|O_NOFOLLOW|O_BINARY. Reading once is what makes a class of defect unrepresentable rather than detected. The predecessor read the file twice -- validate, then install -- and four review rounds each found a different way for the source to change inside that window: swapped for a symlink to a credential, reopened by name, and truncated so the install pass met a clean EOF and reported success having installed fewer records than it validated. With the bytes held there is no name left to resolve and no handle left open. The trade was decided on a measurement. The live file is self-bounding at 400 records by _maybe_trim_notifications, the largest record observed on a real install is 20,821 bytes, and peak for this implementation is 15.9 MiB at that product-bounded worst case against the 128 MiB the same file already accepts for a single record. _NOTIFICATION_SOURCE_CAP bounds the archive-derived source at 32 MiB and refuses over-cap with the size named, never truncating. The copy also runs on the dashboard's notification worker, because O_APPEND stops a concurrent row being overwritten and then orders it BEFORE the archive's rows, and _load_notifications keeps the last 200 rows POSITIONALLY -- so a full history import pushed the live row out of the window. Measured: line 0 of 201, reader returned 200 rows without it. That serialisation ACQUIRES the executor rather than asking whether one exists. Two of its three branches shared one mistake: they read the absence of an observable writer as the absence of a writer. The shortcut assumed "no pool" means "no writer"; the writer is what makes the pool, so on a fresh gateway a delivery created the executor and appended concurrently. The broad except on the import was the same error again, turning "I could not check" into "there is nothing to check". Acquiring collapses both and leaves one inline case -- being ON the worker, where submitting would deadlock -- which is the only branch that was ever sound. Relying on that executor for ordering means its own creation has to be ordered too, so this also locks the lazy init in dashboard/state.py (issue #8788). The unlocked check-then-set let two threads each build a pool and never serialise against one another, which voids the guarantee this change exists to make. A shared-module edit appears in a snapshot-restore PR for that reason: the ordering promised here is not a promise without it. A restore therefore briefly blocks notification writes: a restore is rare and user-initiated, a lost notification is silent and permanent. Where O_NOFOLLOW does not exist the copy is REFUSED rather than attempted. The predecessor checked is_reparse_point by name and then opened the same name, which against the threat model this code is written for -- a concurrent agent replacing the extracted file, which therefore chooses the timing -- is a check-to-open window, not a floor. fstat does not rescue it: it asserts S_ISREG, and a reparse point resolving to a regular .env passes S_ISREG. The capable alternative, a ctypes CreateFileW with FILE_FLAG_OPEN_REPARSE_POINT, is declined by this repo twice in writing (eval/bench/safepath.py, skill_trust.py), so refusing extends two existing decisions rather than inventing one. The costs are not symmetric: refusing loses notification history on one platform for one operation and the records stay in the snapshot, while falling back can install attacker-chosen bytes -- the very defect this change exists to close. The refusal is loud, names the platform and the missing primitive, and both call sites report the skip instead of claiming a copy; the rest of the restore proceeds, because this is a statement about the platform rather than about the archive. Closes #8181 Closes #8788 Refs #8667 Refs #8755
f816314 to
1fd5d0c
Compare
|
Answering F1 on The window is real and I am not disputing it. A delivery queued before this call's What I dispute is "merge if it now exists", and the reason is a number. I measured the merge path against a live-file destination: 4 source records produced 2 installed records. So the proposed fix trades a loud failure that a retry fixes for records that disappear without telling anyone. On a pull request whose entire subject is that snapshot restore installed bytes nobody validated, silently dropping records to avoid a 500 inverts the posture. If the import fails, the operator retries; the destination now exists, the merge branch runs, and both the delivered row and the archive's rows are preserved by a path the operator chose. Nothing is lost that a retry does not recover. The current behaviour is deliberate and already stated in the function's docstring, which is why it is not an oversight to be patched:
There is also a specific reason this path does not try to recover in place. An earlier revision of this change did exactly that -- it created the destination and unlinked it on refusal. Review blocked it, correctly: Finally, this review's own adjudication reaches the same conclusion, recorded in the comment above as
Four earlier rounds on this pull request returned What changed on the current head anyway, since one of the two items in this verdict was correct and is fixed:
I have not silenced this finding and will not. If the block stands, it stands on a security-class fence rather than on the adjudicator's reading, and that is a maintainer's call to make explicitly rather than mine to route around. |
bolichen97
left a comment
There was a problem hiding this comment.
Tech Lead review: approved.
Correctness of the copy-branch validation. The abort is real and it lands before anything is installed: the source is opened once (O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_BINARY), judged S_ISREG on the descriptor, read whole under _NOTIFICATION_SOURCE_CAP enforced on bytes already read, and every record is framed by strict_raw_records and passed through _notification_key before the destination os.open runs. UndecodableRecord and OversizedRecord both derive from UnreadableRecord, so the except (OSError, UnreadableRecord) arm prints the diagnosis and re-raises with opened_dst still False -- no file created, nothing to roll back, and neither Notifications: copied nor notifications (copied) is emitted. That is the same posture _merge_notifications already carried, so the two branches of the if now agree instead of one aborting and the other installing bytes _load_notifications refuses.
Read-once is the right shape. Holding the bytes removes the swap/truncate/reopen window rather than adding a fifth check for it, and the memory argument is made from the destination's own bound (200 records, trimmed at 400) rather than from taste -- 15.9 MiB peak at the product-bounded worst case against the 128 MiB this file already accepts for a single record. Chunked accumulation rather than read(cap + 1) is the detail that keeps the peak proportional to the source. O_EXCL on the destination is correct given the caller's is_file() is advisory, and declining the unlink rollback is right: apply_import_zip runs inside the live gateway.
No loop-stall risk from the new serialisation. _serialise_with_notification_writes blocks on .result(), but the dashboard route reaches apply_import_zip through await asyncio.to_thread(...), so the wait is off the event loop; the CLI path is synchronous by nature. The notif-io re-entry guard covers the one case where submitting would deadlock.
Scope. The dashboard/state.py double-checked lock (#8788) is in scope rather than riding along: an unlocked lazy init means two callers were never serialised against each other, which voids the exact ordering this PR promises. 6 files, +489/-10 in src/, the rest tests.
Accepted as residual, not overlooked:
- The Windows refusal removes a working capability (notification history is skipped on a fresh restore there). The cost is asymmetric in the right direction -- history stays in the snapshot, nothing is destroyed, and the skip is loud in both the CLI line and the import summary -- and it follows the precedent
safepath.pyandskill_trust.pyalready set by declining unexercisable Windows security code. snapshot.pyreaching fordashboard.state._notification_io_executor()and matching thenotif-iothread-name prefix is a private cross-module seam with no owner; renaming the prefix instate.pyconverts the deadlock guard into a deadlock. Worth promoting to a public ordering API indashboard/state.py, but not a defect today.- The amplifier (
_load_notificationsreturning[]on one bad byte,_rewrite_notificationspersisting it) and replace mode are untouched, so this validates one of two ingresses. Correctly named as out of scope with the mechanism measured; the reader fix would cover every origin at once and belongs in its own change.
All 56 checks green including CodeQL, SAST, Cross-Platform Portability and the Windows shards; GPT 5.6 and Opus 4.8 report no blocking findings; Design and First Principles are CONCERNS (advisory) and their points are the three residuals above. No unresolved blocking findings, no inline threads, one commit on main.
Problem / Motivation
Restoring a snapshot puts the archive's
notifications.jsonlinto the live data home one of two ways, and only one of them validated anything:portability.py's import had the same shape. So the same snapshot aborted the restore when a live file existed and was installed silently when one did not.The copy is byte-exact, which is correct as a copy and is exactly the problem: it faithfully installs bytes the destination's own reader refuses.
dashboard/state.py:_load_notificationsputspath.read_text(encoding="utf-8")inside the outertrywhoseexcept Exceptionreturns[], so one invalid UTF-8 byte anywhere costs the WHOLE file rather than the bad row, and only a debug log records it._rewrite_notifications-- reached by any delete, ack or clear -- then persists that empty view.Measured end to end, driving the real
_do_mergeand the real reader withKIROCREW_HOMEpointed at a temp dir:Unlike the merge path this fires on every locale, because nothing decoded on the way in, and its trigger is the ordinary one -- a fresh install or a first restore, where the operator has the least reason to suspect anything.
Why it matters
Notification history is destroyed on the restore path an operator is most likely to be on, with a success line printed over it.
apply_import_zipreportsnotifications (copied)in its summary and the dashboard handler turns that intook: True, so an API caller is told the import succeeded. The loss is not even visible at restore time: the file looks populated until the first delete or ack, which is when it goes to zero bytes._maybe_trim_notificationsreads with the same strict decode inside atry, so on a poisoned file trimming also becomes a silent no-op and the file stops being bounded.What changed (motivation -> approach -> change)
Both branches of that
ifnow agree. The posture is the merge branch's -- abort by raising, never accept, never skip -- and that is not a new product decision: it is the decision_merge_notificationsalready carries, applied to the branch that never got it.Both call sites call a new
_copy_notifications. It is a separate function rather than a call into the merge with an empty destination, and both halves of that were measured rather than assumed:_merge_notificationsopens the destination for READ first, so a missing one raisesFileNotFoundErrorout of the arm that guarantees a destination-scan failure is a true no-op. Teaching that arm to tell "missing, fine" from "unreadable, abort" reopens the fail-closed posture that arm exists for.ts, two byte-identical without one -- through a merge into an empty destination: two land. There is nothing here to deduplicate against, so keying source records against each other converts a faithful copy into a lossy one.The shape is a single read of the source, and the ordering is the design:
apply_import_zipruns inside the live gateway, so the dashboard's notification sink could append to that file first and the unlink took the operator's notification with it.O_CREAT|O_EXCL|O_WRONLY|O_APPEND|O_NOFOLLOW|O_BINARYand the validated bytes written.O_EXCLdecides inside one syscall, so a name that filled after the caller'sis_file()check is refused rather than written through -- a dangling symlink at that name included, whichis_file()reports as absent andcopy2followed, writing the archive's bytes outside the data home.Framing and the memory cap come from
strict_raw_records, run over aBytesIOon the held bytes rather than a hand-rolled split, so the record boundaries are the same implementation the merge branch uses -- a second splitter is how two paths drift apart, which is the defect this change exists to fix. The encoding check is_notification_key's, called for itsUndecodableRecordand its key discarded, for the same reason.Reading once rather than twice is the whole point and is argued from a measurement in the section below; the four findings that forced it are recorded there too.
Two earlier revisions of this PR were blocked, and both findings are why the shape above is what it is. Recording them because each one removed a design, not just a line:
O_EXCLand rolled back by unlinking. An exclusive create proves this call CREATED the file, not that it is the only thing that has written to it since -- andapply_import_zipruns inside the live gateway, so the dashboard's notification sink can append to a file the copy just created, and the rollback then deleted that operator's notification along with the prefix. Validating the source before creating anything is the version of that with no window at all, rather than a narrower one.os.link. A temp file is published through a NAME, and a same-user process that can list that directory can swap what the name holds between the write and the publish, linking bytes this function never validated into place asnotifications.jsonl. The mitigations are inode verification on both ends plus a non-hardlink fallback --pinned_fs.put_back_no_clobberis the repo's audited version, and its own docstring notes the landed check narrows that window rather than closing it. Writing straight to anO_EXCLdestination needs none of it, because there is no intermediate name to swap.A third revision read the source twice -- validate, then install -- and that is what the single read replaces. Its residue was documented and believed bounded, and a test I wrote then falsified the bound: a source truncated between the passes gave the install pass a clean EOF, so nothing raised, the count was never printed because the count only printed on the failure path, and the restore reported success having installed fewer records than it validated. Measured at 5 validated, 2 installed. That is filed as #8755 in the past tense, because the reasoning -- a detected window and an absent window are different guarantees -- is worth finding again.
Records are written verbatim, so what is validated is the source's bytes and never a decoded form of them. One repair: an unterminated final record gains a terminator. That is not cosmetic once the write is record-wise --
_persist_notificationappendsjson.dumps(note) + "\n", so the first notification after the restore would otherwise glue onto an unterminated last line and produce one line that parses as neither row. It is the same repair the merge makes throughdst_unterminated. The create passes mode0o666so the kernel applies the umask, giving the same mode as that function's ownopen(path, "a").Tests
TestNotificationCopyWhenNoLiveFileExistsintest/test_snapshot.pydrives the real_do_merge, so the CALL SITE is under test and not only the helper. Every fixture is real bytes on a real file: a synthesizedUnicodeDecodeErrorroutes the copy down a healthy path and proves nothing.test_an_undecodable_record_is_never_installed-- raises, installs no file, and neither theNotifications: copiedline nor the component success line is printed. The raise is the contract with the caller, since neither the import summary nor the dashboard handler sees stdout.test_the_reader_loads_every_record_the_copy_installs-- the consequence, through_load_notifications. This is the test that separates a real fix from a no-op, and it caught one during development:strict_raw_recordsframes and bounds records but does not decode, so framing alone reproducescopy2exactly while every byte-level assertion still passes.test_a_concurrent_notification_is_not_deleted_by_the_rollback-- the first blocking finding. The delivery is injected at the validation of the record that aborts, which is the interleaving that broke that revision.test_the_data_home_gains_nothing_but_the_notifications_file-- the second blocking finding. Asserted as "the directory holds nothing else" rather than "no file called.tmp", so any future intermediate is caught whatever it is named.test_a_source_that_changes_between_the_passes_leaves_a_valid_prefix-- the residual, and its bound: the prefix is exactly the records that validated, and it still decodes.test_the_installed_file_is_no_tighter_than_one_the_product_writes-- compared against a file written byopen(path, "a")in the same directory, becauseos.opentakes an explicit mode and this is a real choice rather than a default.test_a_clean_source_is_copied_record_for_record-- byte-exactness is the propertycopy2had and the fix keeps: a\r\nterminator survives and a bare\rends a record without being rewritten.test_an_unterminated_final_record_gains_a_terminator-- so the first append after the restore does not glue onto it.test_an_oversized_record_installs_nothing-- a DIFFERENT exception out of the same reader; anexceptnarrowed to the encoding one would let this reason escape.test_a_live_file_that_appears_mid_copy_is_not_replacedandtest_a_dangling_symlink_at_the_live_name_is_refused--O_EXCLrefuses both, and neither the operator's file nor the link's target is touched.test_import_merge_notifications_refuses_an_undecodable_recordintest/test_portability.pypins the same property on the import path, through a real export zip.Five more pin the single-resolution invariant and the append flag. Each needed a CONCURRENT or swapped actor to be observable at all -- a test that only copies into an empty destination passes with or without any of these flags, which is the trap the mutation table below exists to catch:
test_a_source_swapped_for_a_symlink_between_the_passes_is_not_followed-- the swap is made invisible by the held descriptor, so the assertion is that the archive's own bytes land and the credential does not.test_a_source_that_is_already_a_symlink_is_refused--O_NOFOLLOWrefuses at the open rather than reading the target.test_a_non_regular_source_is_refused_not_read_as_empty-- theS_ISREGjudgement on the descriptor. A character device is the caseseekcannot catch, because it is seekable: without the check the read returns EOF and the restore installs an EMPTY notifications file and reports success.test_a_notification_delivered_during_the_copy_survives--O_APPEND, with the delivery injected mid-copy.test_a_source_that_became_a_fifo_fails_instead_of_hanging--O_NONBLOCK. Dropping the flag makes this TIME OUT rather than fail, and the timeout is the evidence.Mutation-verified, each mutation reddening only the test that owns that property:
..._source_truncated_after_the_read_cannot_affect_the_install-- "the injection point never ran, so this test proved nothing"..._source_over_the_cap_is_refused_with_its_size_named(DID NOT RAISE OSError)..._source_cap_is_the_value_this_design_was_measured_for--assert 536870912 == 33554432O_APPEND..._notification_delivered_during_the_copy_survives-- "the delivered notification was overwritten"O_BINARYfrom either flag set..._both_descriptors_ask_for_binary_modeS_ISREGjudgement..._non_regular_source_is_refused_not_read_as_emptyand the FIFO test..._refuses_the_copy_ENTIRELY,..._restore_CONTINUES_and_reports_the_skip...,..._refusal_happens_before_anything_is_opened_or_acquired..._refusal_happens_before_anything_is_opened_or_acquired..._refuses_the_copy_ENTIRELY-- a silent refusal is the bug class being fixed..._note_delivered_during_the_copy_survives_the_READER-- "the append ran while the copy was still writing, so the two are concurrent rather than ordered"..._running_on_the_notification_worker_does_not_deadlock-- verified in a BOUNDED CHILD, not pytest:with guard: completed/without guard: STILL BLOCKED after 20spool is Noneshortcut..._FRESH_gateway_still_orders_the_copy_against_a_delivery-- "a delivery on a fresh gateway ran concurrently with the copy: 'no pool' was read as 'no writer'"except ImportError, widened back toException..._import_failure_that_is_not_ImportError_is_not_swallowed(DID NOT RAISE RuntimeError)..._concurrent_callers_all_get_the_SAME_executor-- "N different executors were created", red on 5 of 5 roundsTwo tests in this class were caught proving nothing, both by mutation runs, and that is the main argument for running them. The
S_ISREGguard first: the FIFO test passed with or without it, because a FIFO is caught byseekfailing on a pipe, so a character-device case was needed. ThenO_APPEND: the single-read redesign stopped calling_notification_keyduring the write, so that test's per-record injection point went dead and it delivered nothing while still passing.The serialisation test was caught the same way and is worth spelling out, because its first version looked right. It submitted the append and then checked the file -- and passed with the serialisation removed, because the copy's remaining writes happened to finish first. A race, not a proof. It now asserts the ordering itself: the single worker cannot run a queued append while the copy occupies it, so a bounded wait times out with the fix and returns at once without it. Deterministic in the direction that matters.
The tests pin the precondition, not the consequence. "No partial install" passes trivially once the read window is gone, so it cannot catch a regression.
..._source_truncated_after_the_read_cannot_affect_the_installreplaces the retired two-pass residue test and asserts the reverse of what that one asserted: truncating the source after it has been read changes nothing. It carries its own liveness guard --assert fired, "the injection point never ran"-- which is what caught the mutation above. The cap gets a value ratchet because the cap is the part a future edit would actually change.Mutations are applied from a file-based patch script that asserts its marker matched exactly once and exits non-zero otherwise, because a mutation that never applied is indistinguishable from a passing run. A mutation whose expected outcome is a HANG is kept out of that harness entirely and verified in a bounded child process -- learned the hard way: driving the deadlock mutation through pytest got the harness killed mid-run, so its restore never executed and it left the guard stripped from the working tree.
The lock's test needed its window widened, and that was measured rather than assumed. With 16 threads released from a barrier and the lock removed, it passed 5 of 5 rounds: CPython's switch interval lets the first thread finish constructing before another is scheduled, so the natural race window is real but far too small to observe. Making the constructor slow enough that a second caller is guaranteed to be inside it turns the test red on 5 of 5. What is asserted is the mutual exclusion, not the timing -- and without the widening the test would have shipped proving nothing, which is the third guard in this class to fail that way.
test/test_snapshot.py test/test_portability.py test/test_dashboard.py: 213 passed (run with-n 0; the dashboard file is included because this diff now touchesstate.py). ruff clean, the repo black gate passes in scope,scripts/check_lockdown_before_publish.pypasses, andmypyis back to the 2 base-ownedtranscribe.pyerrors after a name collision of mine was fixed.One residue worth naming so it is not mistaken for this diff's:
mypy src/kiro_crew/snapshot.py src/kiro_crew/portability.pyreports 2 errors, both insrc/kiro_crew/transcribe.py, which this branch does not touch. They arrive through a transitive import and reproduce byte-identically from the same invocation on a detached worktree at main's tip with none of this branch's code present, so they are base-owned.Manual verification
The measurement in Problem / Motivation is one manual pass: a temp
KIROCREW_HOME, a staging dir carrying five valid records and one truncated multi-byte character,_do_merge, then the real_load_notifications/_persist_notification/_rewrite_notifications. Before: success printed, 0 rows loaded, 0 bytes after the rewrite. After:UndecodableRecord, no file installed, no success line.The first blocking finding was reproduced against both publishes before it was fixed, with the delivery interleaved after the first record: the
O_EXCL-and-unlink version leaves no live file and no delivered note, the replacement leaves the delivered note byte-intact.Out of scope, measured and named
Replace mode installs the same bytes unvalidated.
_do_replacerestoresnotifications.jsonlthrough the generic_backup_and_copycore-file loop (pinned_fs.copy_file_pinned), and on the same fixture it prints its component success line over a file that does not decode as UTF-8. It is left out deliberately rather than overlooked: the change shape is different -- a per-file validator inside a loop shared with config and security, carrying a rollback ledger -- and the operator contract is different too, since replace moves the live file into a backup at the operator's explicit request. It belongs in its own issue, not folded into this diff.The invariant, and why it became "read once" rather than "resolve once"
Five blocking findings landed on this span, and four of them were one defect: the source was read twice, and another process could change it inside that window. Swapped for a symlink to a credential; reopened by name; and finally truncated, so the install pass met a clean EOF, installed fewer records than it validated, and printed a green tick. Each fix closed one variant and the next round produced another, because a check only catches the case someone thought of.
So the fix is not a fifth check. The source is read ONCE, whole, under a byte cap, and everything after that reads the bytes in hand. There is then no name left to resolve and no handle left open, so there is nothing to swap, truncate or extend -- the class is unrepresentable rather than detected. The two loops that validate and then install are two passes over the same immutable bytes, which is safe for exactly the reason two passes over the file were not.
That was decided on a measurement, not a preference, because streaming would have been mandatory if these sources could be large. The destination is self-bounding:
_MAX_PERSISTED_NOTIFICATIONSis 200,_maybe_trim_notificationsruns after every append and trims past200 * 2, the loader keeps the last 200 and every rewrite writes the last 200 -- so the live file holds at most 400 records at any time and anything beyond 200 is discarded on the next read. Installing more is transient by construction.notifications.jsonl, real installtracemallocon_copy_notifications_NOTIFICATION_RECORD_CAPThere was no memory argument for streaming: it was defending an 8 MB file with an allocation ceiling sixteen times its size.
The whole-file cap is the load-bearing addition
Single-read needs one where streaming did not -- the per-record cap bounds one record, not a file made of many, and the source is archive-derived so nothing bounds it from outside.
_NOTIFICATION_SOURCE_CAPis 32 MiB: ~4x the product-bounded worst case and a quarter of the per-record cap. Over-cap REFUSES with the size named -- never a truncation, never a silent skip, because a cap that dropped the tail would rebuild the very defect this removes, one layer up.Two things about it are measured rather than assumed. The peak scales with the source, not the cap: the first implementation used a single
read(cap + 1), which preallocates the whole limit, so an 8 MB source cost 32 MiB and the peak tracked the constant instead of the file. Only measuring the shipped code showed it; the read accumulates in 1 MiB chunks for that reason. And the cap is enforced on bytes already read, inside the loop, rather than againstst_size-- a separately-stated size can be stale by the time it is compared, and the bytes in hand cannot be.The copy is serialised against the dashboard's notification writer
O_APPENDprevents a concurrent notification being overwritten. It then orders that row before the archive's, because the dashboard's append reaches end-of-file immediately while this copy's writes are still buffered. And_load_notificationskeeps the last_MAX_PERSISTED_NOTIFICATIONSrows positionally, not by timestamp -- so importing a full 200-record history pushes the live row out of the window. Measured:The row is written, kept on disk, and then silently dropped by the reader. The operator was told the notification was delivered; after the next reload it is gone. That is the same flag with a different consequence, which is the shape that produced a new finding on this span four rounds running -- so it is fixed here rather than filed as a fifth filter.
Notification persistence runs on one worker in submission order, so the copy now runs on that same worker: a queued append is ordered strictly after it and lands at the end of the file, where the cap keeps it. The cost, stated rather than left implicit: a restore now briefly blocks notification writes. A restore is rare and user-initiated; a lost notification is silent and permanent.
Three cases deliberately run inline instead -- the dashboard module not importing (the CLI restore has no writer to order against), no executor existing yet (nothing has persisted a notification in this process, so creating a pool would be the cost without the benefit), and already being ON the single worker, where submitting would wait for a queue only the submitter can drain. The executor is released as soon as the work returns or raises:
result()re-raises, so the failure path needs no separate release and cannot leave notification writes blocked.Two of those three branches were wrong, and it is one mistake twice
The shortcut assumed "no pool" means "no writer"; the writer is what makes the pool. A fresh gateway has persisted nothing, so the pool was
None, so the copy ran inline -- and a delivery arriving at that moment created the executor and appended through it, concurrently, putting the live row back at line 0 of 201 and outside the reader's window. Reproduced:Review found that instance. Auditing the other two branches found the same error a second time, one line up: the broad
except Exceptionon the import turned "I could not check" into "there is nothing to check", so an import failing inside a live gateway would have degraded the ordering in silence. The generalisation is worth naming because it is not specific to this file: treating the absence of an observable writer as the absence of a writer.So the serialisation now acquires rather than asks.
_notification_io_executor()creates the worker if there is none and returns the existing one if there is, which collapses both holes and leaves exactly one inline case -- being ON the worker, where submitting would deadlock rather than race. That branch is not an absence-reading at all: it means we are the ordering point. It is the only one that was ever sound and the only one kept. The import clause is narrowed toImportError, so a genuinely absent dashboard still falls back while any other failure surfaces.Acquiring costs the CLI path one idle worker thread it did not previously create. That is a short-lived process and the thread exits at interpreter shutdown; a silent ordering hole is permanent.
What this does NOT close
The single read closes the read-window class only. The destination side is a separate problem and its protections stay necessary on their own terms:
O_EXCLso a name that filled after the caller'sis_file()check is refused rather than written through,O_APPENDso a concurrent row is not overwritten, and the serialisation above so it is not ordered out of the reader's window either. Reading the source once says nothing about other writers to the live file -- the ordering finding arrived after the redesign and is the proof of that.What remains open on this function is the ancestor chain and the sibling
_merge_notifications, both named below.The one residue inside the ordering guarantee is now closed rather than named: the lazy init of the executor itself is locked.
_notification_io_executor()was an unlocked check-then-set, so two threads could each observeNone, each construct a pool, and each proceed -- one assignment won the global while the loser's worker was already live with its job queued, and the two callers were never serialised against one another at all. That is #8788, and it is fixed here with a double-checked lock, the same shapeatomic_writealready uses for its umask cache.Why a shared-module edit appears in a snapshot-restore PR. It was initially scoped out as a reviewability call -- a concurrency change to
dashboard/state.pysitting in front of reviewers reading about notification imports -- and that was reversed, because this PR's entire value is an ordering guarantee and a race in the executor's creation means there is no ordering to guarantee. It is outside the diff by line count and inside it by consequence. It also made the PR unreachable: the review lane blocked on exactly this, and a lane has no channel to be told a maintainer accepted it as residual, so no amount of code elsewhere could clear it.The change is deliberately the smallest one that makes the second caller wait: a module-level lock and a double-check. No redesign of the executor, no refactor of the module.
A redesign sold as closing everything is how the next finding arrives as a surprise instead of an expectation.
The audit that produced this
Every path access in the span was audited rather than the flagged lines patched. Two source reads collapsed to one; the destination's single
os.openkept, because a create must name something andO_EXCLdecides inside one syscall; three uses justified as resolving nothing (_safe_name, and thepathargument tostrict_raw_recordsand_notification_key, both of which only interpolate it into an error string -- verified by reading both callees); and the caller'sis_file()checks recorded as advisory and refuted by the authoritative resolution rather than trusted.The audit also found a hazard no lane named, and it is one no lane could: a missing
O_NONBLOCKproduces no wrong answer, it produces no answer at all. A name that became a FIFO after the caller's check blocks the open forever, so the restore hangs instead of failing. Measured in a wall-clock-bounded child process --with O_NONBLOCK: open returned/without: STILL BLOCKED after 5s.What the audit deliberately left open
What is not closed is the ancestor chain: both opens name a directory rather than a pinned descriptor. That is deliberate and stated in the docstring -- every core-file copy in
_do_mergereaches its path the same way, so pinning one of ten sites would be the point patch an earlier review lane already named. It is an axis for its own change.The invariant is descriptor-bound on POSIX, and where it cannot be the copy is REFUSED
getattr(os, "O_NOFOLLOW", 0)is zero on Windows, so nothing there refuses a reparse point at the source name. The previous revision put a by-namepinned_fs.is_reparse_point()check in front of the open and called it a floor. A review lane was right that it is not a floor, it is a check-to-open window, and the argument is short: the threat model this function is written for is a concurrent agent replacing the extracted file, so the adversary chooses the timing, and a by-name check followed by a by-name open is exactly what it waits for. The descriptor check does not rescue it either --fstatassertsS_ISREG, and a reparse point resolving to a regular.envpassesS_ISREG.So on a platform with no
O_NOFOLLOWthe copy does not happen._copy_notificationsraisesNotificationCopyUnsupportedbefore the executor is acquired and before anything is opened, and both call sites report the skip: the CLI printsNotifications: SKIPPED -- ...andapply_import_ziprecordsnotifications (SKIPPED: ...)instead ofnotifications (copied).Three reasons, and the third is why this is not a preference:
The capable alternative is declined by this repo twice in writing, which is what makes refusing the consistent move rather than a new one:
src/kiro_crew/eval/bench/safepath.py-- a ctypesCreateFileWwithFILE_FLAG_OPEN_REPARSE_POINT"is no longer worth considering here: it would buy the same property exclusive creation already has, at the price of security code that cannot be exercised on the machine this harness is developed on."src/kiro_crew/skill_trust.py-- Python "does not expose an equivalent handle-relative, no-reparse walk on Windows, where a path lookup can initiate SMB authentication before a later containment check can reject the target."Both decline the capable-but-unexercisable route, so the repo has already chosen less capability on that platform over a hand-rolled walk. Refusing extends those two decisions; falling back contradicts them. The
lstat-vs-fstatidentity comparison is declined on its own merits and remains declined: it is a check-to-use window wearing a seatbelt, narrowing a race while reading as though it closed one.What this costs, stated plainly: on Windows, restoring a snapshot into a data home with no existing
notifications.jsonlimports every other component and skips notification history, loudly. The by-name check is gone rather than bypassed, andtest_no_by_name_reparse_check_is_made_on_either_pathcounts the calls at zero so reintroducing the racy remedy reddens.Because the copy genuinely cannot run there, the 25 copy-behaviour tests carry
requires_o_nofollowand the refusal tests do not. That split was verified rather than assumed -- deleting the constant before pytest collects reproduces the Windows shard exactly:4 passed, 25 skipped, which is the refusal covered on the platform where it is the only reachable behaviour.O_BINARYis a convention fix, and explicitly not a corruption fixA review lane held that
os.openwithoutO_BINARYcorrupts records on Windows -- collapsing\r\n, truncating at0x1A-- and therefore "silently truncates or byte-corrupts the installed notification history while reporting success." That mechanism was tested and did not occur.test_a_clean_source_is_copied_record_for_recordasserts byte-exact\r\nsurvival through these very descriptors, carries no skip marker, and ran green on the Windows lane (20309 passed, 601 skipped, 0 failed, nothing--ignored).What survives is real but much smaller, and the flag is added on that basis alone: every sibling passes it,
crash_dump_store.pyreaches for this exact read-flag triple, there are 7+ such sites, and one documents it as required on Windows. This function was the outlier.getattr(os, "O_BINARY", 0)is provably a no-op on POSIX, where the constant does not exist.Base
Rebased onto
35afc7f1c(headf81631424).The same defect exists in the sibling function, and is filed separately
_merge_notificationsresolves its source by name twice as well, atsnapshot.py:2608(its validation pass) andsnapshot.py:2629(its copy loop). The exposure is the same shape and worse in effect, because that function appends into the LIVE file: a source swapped for a symlink between those two opens appends the link target's own bytes into the operator's notification history, and the validation pass reports nothing wrong because it validated a different file.Filed as #8667 rather than folded in. Those lines are unchanged by this diff -- they arrived with #8184 and are on main -- and the same invariant resolves them. The issue carries both line numbers, the mechanism, and the note that its destination scan at
snapshot.py:2587has the identical shape one layer down.Related Issues
Closes #8181
Refs #8667 -- the same double-resolution defect in the sibling
_merge_notifications, filed from this PR's audit. DeliberatelyRefsand notCloses: this diff does not touch those lines.Closes #8788 -- the notification executor's unlocked lazy init.
Closesand notRefs: it was scoped out as a follow-up and then brought in, because the ordering this PR promises is void without it and the review lane blocked on exactly that.Refs #8755 -- the truncation-between-passes defect this redesign removes, filed in the past tense as the record of a defect that existed and how it was closed.
Refsrather thanClosesbecause the issue is the reasoning, not an open task.Related: #7771 / #8184, which fixed the MERGE branch of this same
ifby removing a text-mode decode/encode round trip. This is the sibling branch, which never decoded at all, so the change here ADDS a validation pass rather than removing a round trip._copy_notificationsreuses thejsonl_utilreaders that #8184 introduced.Pattern harvest
Rule candidate: review-prompt
Pattern: a validating operation that reads its input TWICE -- validate pass, then use pass -- where something else can change the input in between. Four review rounds on this one function each found a different way through that window (symlink swap, reopen by name, truncation to a clean EOF), and each fix closed one variant and revealed the next, because a check only catches the case someone thought of. The prompt to add: when a function validates then uses, ask whether it re-reads, and whether holding the input instead is affordable. Decide that on the measured size of the input against the allocation the code already accepts, not on preference -- here the streaming design was defending an 8 MB file with a 128 MiB per-record ceiling, so reading once cost nothing and made the whole class unrepresentable. Also mechanisable as a semgrep shape: two reads of the same path argument in one function body.