Repository navigation
feat(crew-container): carry transcripts and the slot table across a task replacement - #13374
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The description and the Watch
Suggestions
[DESIGN-REVIEWED] 47b6639 Description read[DESCRIPTION-READ] c5fd76c17380a1ff49d963741f745f3fad77a4af9378182cae4890850c153511 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS About 7,000 lines of durability code ship switched off for everyone: no shipped producer sets Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: a replaced AWS Control crew task comes back with its conversations and conversation list intact (ADDITION: the writer half of a recovery path).
[FIRST-PRINCIPLES-REVIEWED] 47b6639 Description read[DESCRIPTION-READ] 80ea52a559d3ff041781ceeffe6918819c29849165be5411ce29f444db377e82 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings block this merge. Two advisory issues remain. FINDING — src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/supervisor/main.py:131 — FINDING — docs/system-specs/modules/aws-control.md:2021 — The spec added in this PR disagrees with the code in two places. It names the generation prefix [OPUS-REVIEWED] 47b6639 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
4e80c2a to
c25d86c
Compare
ffdfd6c to
dd962af
Compare
dd962af to
b356286
Compare
b356286 to
7b786da
Compare
d9a5c74 to
e0bf809
Compare
|
/ai-review override gpt 271db9b: F1 is not reachable: the skew needs a whole atomic open_slots.json write inside the gap between two open_snapshot calls, and its named cold start happens on every replacement anyway because build_backend_env sets KIROCREW_HOME and no KIRO_HOME, so sid transcripts sit outside the backed-up data home and SessionMap.get prunes each unmatched sid. The proposed pair check would withhold forever on an opened-but-unused tab, which backup.py forbids. Verified in tree. |
Human judgment recorded@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
Read-only review note (not blocking, but worth a reply before merge).
Data-safety otherwise reads sound: pointer written last, restore validates before writing, no list/delete calls, taskdef only adds stopTimeout. Leaving approval to a maintainer given the size. |
|
Reviewed, not approving from my side: ~14.6k added lines across 41 files (carrying transcripts and the slot table across a crew-container task replacement). At that size this needs a maintainer's own review, ideally split into smaller PRs. CI is green. |
There was a problem hiding this comment.
Holding this on scope, not on the code.
This PR implements #9464, which is labelled tier:T3. It adds a new durability subsystem: the sidecar backup and restore, a new S3 object layout, and a new supervisor start order. A T3 change needs a maintainer design sign-off before it merges. I will re-review the code once that is done.
🤖 AI-assisted review 🤖
…ask replacement Durability sidecar + supervisor for the AWS-control crew container: a replacement task restores the authority pair and lazily fetches transcripts, so conversations and the slot table survive a task replacement. Review fixes (latest head): - GPT 6.1 (cold-replacement open-slot loss) -- a replacement task restores open_slots.json from a remote committed snapshot but NOT the transcripts (they load lazily, per turn), so every restored slot has empty-but-readable local metadata. The backend's _apply_restored_open_slot treated that as a confident "tab gone", dropped the key without a reopen seed, and the next flush wrote an empty open_slots.json the sidecar committed -- silent loss of the whole restored slot table on an ordinary replacement. Fix per the lane's remedy (b), scoped so it changes no ordinary-boot behaviour: the supervisor exports KIROCREW_CONTAINER_AUTHORITY_RESTORED when it restored authority from a remote snapshot (supervisor/__main__.py, backend.py ENV_AUTHORITY_RESTORED); the backend's open-slot restore reads it (chat_persistence._transcripts_may_be_remote_only) and, for a listed slot with no local transcript that is NOT a ✕-closed tab, keeps the key as a reopen seed until lazy retrieval completes. Unset on an ordinary (non-container) boot, where an absent transcript stays a genuine answer and its key is pruned so dead tabs do not resurrect forever (test_absent_session_is_dropped_from_reopen_seed, kept green as the negative control). New tests: a remote-only slot is kept, a closed tab is still dropped, and the supervisor sets the flag only on a real remote restore. - GPT 6.1 F1 -- a front/sidecar spawn failure after readiness releases the ownership claim (supervisor/__main__.py). The restore/readiness abort path already released; the spawn-failure teardown path now does too, so a fork/PID failure after the claim does not leave the incarnation bump raised and fence a live predecessor. - GPT 6.1 F2 -- BoundedReader.read serves a request in <=64 KiB underlying slices (common/objects.py), re-checking the deadline and the stop BETWEEN underlying reads. A 1 MiB read on a link trickling below the 5s socket timeout would otherwise block in one read past the drain window with no check getting a turn; the slice loop honours the window on a trickle, not just a stall. - GPT 6.1 F3 -- the generation-0 legacy migration threads the cycle's stop into its GET and PUT (backup.py _migrate_legacy_transcripts) and stops with the pointer WITHHELD when a stop fires. An interval migration of a large legacy object over a slow link no longer eats the drain window the final cycle needs. - Opus 5.5 -- a provisional CLAIM must not fence a live predecessor terminally (backup.py _commit_generation). A replacement bumps the pointer's incarnation before restoration, then may fail readiness and abort (release gated on a 180s timeout); a healthy predecessor serving turns at its 60s interval would read that claim, raise SidecarSuperseded and be torn down long before the claim is released -- an outage, and the predecessor's turns since its last commit lost. A commit fenced by a claim pointer (committed.is_claim) is now refused as RETRYABLE (BackupIncomplete, superseded unset); the predecessor keeps serving and steps aside only once a REAL, unmarked commit carries a newer incarnation. - Ownership F1 -- the release CAS validator comes from the claim's own PUT, not a follow-up GET (store.py, generation.py). ObjectStore.put now returns the ETag from its own PutObject response, and claim_ownership takes the release validator from there. A confirmation GET after the claim could fail on a degraded link even though the PUT committed -- and that exception escaped before any ClaimedOwnership existed to release, fencing a live predecessor for good -- or read a newer writer's ETag. The response ETag has neither hazard. - Ownership F1/F3 -- a claim carries the last committed owner through a nest of provisional claims (committed_incarnation on the pointer body). The release restores that real committed owner, so overlapping aborting claims each un-fence the live predecessor rather than reinstating one another's dead tokens; whichever release wins the CAS lands the predecessor and the rest no-op. - F2 -- interval-cycle preflight reads thread the stop predicate through to the store GET (store.py get/get_with_etag gain cancel; generation.py read_pointer and read_transcript_index forward it; backup.run_cycle passes yield_when). An interval has no deadline but a trickling multi-megabyte index read is now cut the instant a stop arrives, so it cannot eat the drain window the final cycle needs. - Store-double protocol parity -- the test ObjectStore doubles accept cancel to match the real store; without it the superseded-exit test's interval cycle raised TypeError that the loop swallowed as a failed cycle and looped (120s timeout) instead of exiting. - Corpus-growth ceiling -- test_source_corpus.py blocking-on-loop ceiling held at 915; the new async container/sidecar source legitimately grows the "async"-matching corpus past main's bound (911 after the rebase onto a60fda2), and 915 is verified against the real collected count on the rebased tree. Retained from prior rounds: - Budget and cancel the final-cycle transcript-index upload (backup.py): under a deadline it carries the one-attempt budget (BACKUP_ATTEMPT_COST_SECS) and the cycle's yield check as a cancel, so a slow trickle cannot outlast the 75s drain window and be SIGKILLed mid-upload. An interval cycle (no deadline) stays unbounded, like the pointer PUT. - Restore and backend-spawn failures release the claim: restore_authority and start_backend run inside the ownership-release scope, so a failure after a successful claim releases the incarnation bump instead of fencing a live predecessor; the backend is torn down only if it started. - Generation-zero startup pre-mints its incarnation and exports it for the sidecar to adopt, so a predecessor's first commit in the restore-to-sidecar window cannot be overwritten by a higher derived token. Still writes no pointer. - The transcript entry cap is enforced during the directory scan (bounded memory); a superseded sidecar ends the process (SidecarSuperseded -> exit 5); _evict_prior_live_blob runs only after the new blob is recorded durable; _migrate_legacy_transcripts carries authority-named legacy transcripts into the first index at generation 0; startup_incarnation propagates PointerUnusable rather than minting a base token; store-derived incarnation ordering; the bounded transcript index (byte cap); archive segments keyed by path + content digest; and the supervisor spawn-teardown scope. GPT 6.1 review fixes (head 6ef07a0): - F1 (fenced, crash-data-loss) supervisor/__main__.py -- a stop in the window between the startup signal guard restoring the default handlers and _wait_for_shutdown installing its own would kill the process (SIGTERM) or escape as KeyboardInterrupt (SIGINT) without draining children or releasing the ownership claim, fencing a live predecessor. The `watched` build and the supervise try now live INSIDE the guard scope, so a stop there is still _StartupInterrupted; the handoff abort path releases the claim and the finally tears the children down. _wait_for_shutdown installs its handlers first, so no instant is unguarded. New test: a stop during the handoff releases the claim and tears down. - F2 (tests-are-deterministic) test_sidecar_durability_gaps.py -- the two clock patches set monotonic on the stdlib `time` SINGLETON (backup_mod.time), handing every xdist worker the finite synthetic sequence. Replaced with a _ClockProxy that overrides monotonic and delegates all else to the real module, patched onto the module-local binding. - F3 (tests-are-deterministic) test_sidecar_durability_gaps.py -- the in-flight-cancel test asserted wall-clock `elapsed < 2.0` with a sleep barrier, which a loaded runner flakes. Rewritten to assert the ordering fact (the cut lands while the underlying read is still blocked -- release is never set) via signals, with a bounded failure wait; no wall-clock. - F4 (finding) backup.py -- corrected the two docstrings that said a failed authority PUT leaves a mixed committed pair; it leaves an unpublished partial generation under gen/<id>/ while the committed pointer is unchanged.
|
Triage note (automated PR-overlap scan, read against Needs a rebase. This branch (head Overlap check: no other open or merged PR implements the sidecar backup/restore, No new P0 or breaking-change finding from this scan. One rollout point: |
|
Reviewed again at the new head, still not approving (crew-pr-reviewer). A change request is still open, and the PR is very large (+14710/-441). It is left to the author and reviewer. |
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE (code). Merge stays gated by bolichen97's open CHANGES_REQUESTED, which is a T3 design sign-off and not a code finding. See below.
What it brings: a replaced crew task can restore its conversation list and each conversation's transcript from S3, instead of booting empty and then flushing that empty state over the real history. The writer is inert until a producer sets SMC_BACKUP_BUCKET and grants S3.
Reviewed at head 47b663925205f7b519c56559cb2c134dd2981217 (merge base 276d153fa9). git merge-tree against the current origin/main exits 0. CI on the head: 90 success, 9 skipped, 0 failure. Design Review, First Principles Review and Sensitive Change Review are all green as check runs (Design and First Principles post advisory CONCERNS). Auto-merge is off.
Prior human review: bolichen97 (CHANGES_REQUESTED @ e12da62b0c)
| Finding | Status at 47b6639252 |
|---|---|
Hold on scope: #9464 is tier:T3 and needs a maintainer design sign-off before merge |
Not resolved. #9464 is still open and labelled tier:T3 / needs-human. Its comment thread holds automated triage and the author's pickup note, and no maintainer sign-off. This is a process gate, not a logic defect, so it does not change my code verdict. Only bolichen97 or a maintainer can clear it, on the PR. |
Triage note 2026-10-06: needs a rebase, conflict in dashboard/chat_persistence.py with #17196 |
Fixed. The branch is rebased onto 276d153, and git merge-tree --write-tree origin/main HEAD exits 0. The preserve_remote_only hunks now sit on the post-#17196 codec. test/test_open_slots_persistence.py passes 87/87. |
iamwhatever's notes are comments, not a CHANGES_REQUESTED. His #1 (empty SMC_CREW_NAME would make two crews share a key namespace) is not reachable in the boot path. supervisor/bundle.py:579 refuses an empty crew name inside install_bundle (supervisor/__main__.py:1017). That runs before the ownership claim (:1054) and the restore (:1073), and the sidecar is spawned only after both. The shipped producer also derives SMC_CREW_NAME from a CrewBinding whose crew must match ^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$ (cloud/fargate/identity.py:70), so a crew name can never carry a /.
Blocking
None found in the paths I verified (scope below).
Cross-member / cross-session isolation (focus)
- Between crews. Every key is
<prefix>/<crew>/…, built in one place (common/keys.py:232object_prefix). The crew name is a validated slug, and the boot path refuses an empty one before any S3 call. Generation ids and blob digests are shape-checked (keys.py:110,:345) when the pointer is read (sidecar/generation.py:243) and again when a key is derived, so a value read from the bucket cannot steer a key outsidegen/<id>/ordata/blob/. - Between conversations in one crew. The front resolves a slot only through the committed index (
front/transcript.py:275). A stem the index does not name is a fresh conversation, so a recycled slot id cannot pick up an older object. A referenced index or blob that is absent refuses the turn rather than serving an empty history. Blobs are content-addressed, so two stems can share a blob only when their bytes are identical.is_fetchable_slot_idalready keeps a folded id from pulling someone else's transcript before the backend can refuse it (pre-existing). Restore writes only the two authority names intoconfig_dir, and only aftervalidate_authority. - Deleted conversations.
_merge_transcript_index(sidecar/backup.py:1714) carries a committed mapping forward only while the captured authority pair still names that slot, so a deleted conversation drops out of the index on the next commit. There is a theoretical window: delete a conversation and reuse its slot id inside one interval, and the old bytes come back. I found no route that reaches it, because the front serves onlyPOST /v1/chat/completionsand/healthand exposes no delete.
Upgrade impact
- Host and dashboard users (non-container).
chat_persistence.pychanges only whenKIROCREW_CONTAINER_AUTHORITY_RESTORED=1, and only the container supervisor sets it. On an ordinary gateway restart, slot restore and pruning behave as before (87/87 pass intest_open_slots_persistence.py). - Existing crew task definitions.
FINGERPRINT_SCHEMEmoves 3→4 andstopTimeout=115is added (cloud/fargate/taskdef.py), so every crew task definition re-registers on its next deploy, including crews with no bucket. A stop can now take up to 115s instead of 30s, though a task with no sidecar drains in front plus backend time and finishes sooner. This is intended and documented, so it does not block. - A crew task with
SMC_BACKUP_BUCKETset by a caller.runtasklets a caller supply that variable, since it is in neitherDERIVED_ENVnorREFUSED_ENV. Such a task now runs an ownership claim plus a restore before the backend, and refuses to boot when S3 is denied. On main it booted and refused only the turns that needed a fetch. The current task role grants no S3, so this case was already non-functional, and the PR names it as a deliberate tightening. Documented, so it does not block. - Buckets in the legacy layout. The legacy
data/sessions/<stem>.jsonlanddata/<authority>.jsonfallback is gone, so a bucket with no pointer boots empty. I swept origin/main and found no writer that ever produced that layout, so there is nothing to migrate. - macOS developers. The container suite now skips off Linux. This is dev-only.
Non-blocking follow-ups
-
The spec contradicts the code, and an operator could act on the wrong text (the Opus FINDING 2 and the Design CONCERNS are both valid):
docs/system-specs/modules/aws-control.md:2021saysgenerations/<id>/, but the code usesgen/(keys.py:100).:2034–2045describes the "FIVE cases", including "No pointer: the legacydata/keys are GENERATION 0" and "legacy keys AND a committed generation". The code has no legacy read (restore.py:307,front/transcript.py:275).:2072–2077says "both sidecars then write the same keys". The code now uses immutable per-generation keys, content-addressed blobs and an incarnation fence.- The overlap residual should be restated for the fence. A draining predecessor's final commit is now refused once the replacement has claimed ownership, so turns it flushed during an overlap still reach the bucket as blobs, but no committed index names them.
- AGENTS.md requires the owning spec to match, so fix this before merge even though it does not gate the code.
-
Stale docstrings and comments that tell a reader a legacy fallback exists:
front/transcript.py:648("the LEGACY per-stem key is what holds the bytes, so the resolution falls back to it")common/keys.py:216-217(pointer absence "read as GENERATION 0")sidecar/generation.py:222("the legacy keys are this bucket's truth")keys.py:118("the boot reads generation 0")
The design bot's point stands: a maintainer who trusts these comments could re-add the fallback, and that would resurrect deleted conversations.
-
Large parts of the PR body are stale. "Two slots and a single pointer", "A bucket with no pointer is read by surveying its generation slots" and "existing keys … read as generation 0" all contradict both the code and the body's own Backwards-compatibility section. The body never mentions the blob/CAS/incarnation-fence layer that actually ships. Since the body becomes the squash-commit body, it should be rewritten before merge.
-
supervisor/__main__.py:131restore_authorityreturnsTrueeven when nothing was restored (Opus FINDING 1). In practice this matters only when both files werekept_local, which a fresh Fargate disk never produces, or when the bucket is empty, where there are no keys to keep.return bool(result.restored)matches the docstring. Related, from the Design review: a listed slot with no transcript is carried inopen_slots.jsonacross every replacement with no prune bound. That is low volume, but nothing ever drops it. -
First-boot pointer failure is unprotected. The slot survey that adopted an orphan pair was removed. If the very first cycle's pointer PUT fails and the task is then replaced, the replacement boots empty and the next cycle commits that empty pair. The non-zero exit does report it, so it is not silent, but the residual should be written down.
-
Test gaps, where the code is correct but no test pins it:
- The
StoreUnusable→RestoreFailedbranch inrestore._authority_source. The test_DeniedStoreraisesRuntimeError, so it goes throughPointerUnusableinstead. In production this is backstopped becauseclaim_ownershipreads the pointer first and propagatesStoreUnusable. - The
is_blob_digestcheck infront/transcript.py::_digest_for_stem.keys.blob_keyre-checks it, so a bad digest raisesValueErrorrather thanTranscriptUnavailable. - The id check in
keys.authority_generation_key.read_pointeralready validates the id.
- The
-
Size. iamwhatever and bolichen97 both asked for a split. At +14.7k lines, of which about 7k is inert until the IAM change in #9464, landing the fence/CAS layer together with the producer would get it exercised somewhere real.
What I verified
- I read the PR body, all bot comments, both human reviewers' notes, and the diffs of
common/keys.py,front/transcript.py,sidecar/restore.py,supervisor/__main__.py,supervisor/backend.py,dashboard/chat_persistence.pyandcloud/fargate/taskdef.py. Withinbackup.pyI read the merge, commit and withhold section in full (run_cycle1460–1580 and_merge_transcript_index), plus the generation pointer read and claim, and the spec section at 2010–2077. - I did not line-review all 2,879 lines of
backup.pyor the 5,158-linetest_sidecar_durability_gaps.py. For the deadline, cancel, procfs-link and fwalk machinery I relied on the author's mutation table, CI, and my own mutations below. - Baseline:
test_front_transcript_fetch,test_sidecar_round_trip,test_sidecar_restore_refusals,test_sidecar_durability_gapsandtest_supervisor_startup_ordergave 280 passed.test/test_open_slots_persistence.pygave 87 passed. These ran in a scratch venv with the image's fastapi, httpx and uvicorn pins. - Mutations: each matched exactly one site, was restored from a hash-verified backup, and left
git statusclean.
| # | Mutation | Result |
|---|---|---|
| M1 | Front: an index miss falls back to the legacy per-stem key | RED test_a_committed_index_that_omits_a_stem_does_not_resurrect_the_legacy_object |
| M12 | Front: an absent committed index is treated as fresh | RED test_the_front_refuses_when_the_committed_index_object_is_absent |
| M13 | Front: an absent referenced blob is treated as fresh | RED test_the_front_refuses_when_an_index_named_blob_is_absent |
| M2 | Front: drop the is_blob_digest check |
survives (see follow-up 6) |
| M3 | Sidecar: merge carries every committed entry | RED test_a_deleted_conversation_is_not_carried_forward, test_a_deleted_conversation_is_pruned_from_the_lifetime_state_maps |
| M4 | Sidecar: merge carries no committed entries | RED test_a_replacement_cycle_carries_forward_unfetched_transcript_mappings |
| M5 | Keys: authority_generation_key accepts any id |
survives (upstream read_pointer validates) |
| M6 | Restore: a kept local file is not validated | RED test_a_malformed_local_file_refuses_the_boot_even_with_a_valid_bucket_copy, …_with_nothing_in_the_bucket |
| M7 | Restore: a generation missing a member is accepted | RED test_a_committed_generation_that_lost_a_member_refuses_the_boot[open_slots.json], [session_map.json] |
| M11 | Restore: StoreUnusable on the pointer read is treated as absent |
survives (see follow-up 6) |
| M8 | chat_persistence: preserve_remote_only branch disabled |
RED test_remote_only_slot_is_kept_as_reopen_seed_after_authority_restore |
| M9 | Supervisor: never sets ENV_AUTHORITY_RESTORED |
RED test_a_remote_authority_restore_tells_the_backend_transcripts_are_remote |
| M10 | Supervisor: backend starts before the restore | RED test_the_authority_files_are_restored_before_the_backend_starts, test_a_failed_restore_stops_the_task_before_the_backend_starts, test_a_failed_restore_releases_the_ownership_claim, test_a_stop_signal_during_startup_releases_the_ownership_claim |
- I did not run the full suite, and I did not exercise any real S3, IAM or Fargate behaviour.
|
@bolichen97 Understood — holding for a maintainer design sign-off on the tier:T3 scope of #9464. The code is unchanged at 47b6639 (PR Readiness green, approved by buluoray; merge-tree against current origin/main is clean), and there will be no further pushes until that call is made. |
|
Closing per a lane audit: #9464 is a design decision (tier T3, needs-human) and this is a +14.7k new backup subsystem rather than a minimal fix. Thanks @bolichen97 @iamwhatever @buluoray for the reviews; the branch fix/crew-container-transcript-durability-9464 is kept so the design can be picked up from it. |
|
Correction: my previous comment was wrong — this PR was already merged by @bolichen97 at 19:55Z (merge commit 27e7dfd), before the lane-audit close instruction reached me, so it was never closed and the audit CLOSE verdict is moot. The maintainer merge stands. Apologies for the confusing comment. |
Problem / Motivation
Goal: A crew task that gets replaced comes back with its conversations intact, instead of showing the customer an empty chat and then overwriting the real history with that emptiness.
A crew task in the AWS Control app is replaced routinely: every deploy, every failed health check, every platform update. Its filesystem goes with it, and the transcripts plus the two files that turn a slot id back into a conversation (
session_map.json,open_slots.json) exist only there.Half the recovery path already shipped.
front/transcript.pyfetches one slot's transcript from S3 on the turn that continues it, caps it, and refuses the turn rather than presenting an empty chat. So there is a reader and nothing writing. The supervisor's own module docstring said so: no restore phase, no sidecar.Refs #9464.Why it matters
A customer returns to a conversation and finds it empty, and nothing reports a fault. The next whole-object write then overwrites the real history with the empty one. The conversation list fails the same way with one extra step: the backend flushes the slot table from its own memory, so a task that boots without those two files persists an empty table over them.
The task looks healthy throughout. Every gate is green, the front answers, and the loss is silent.
Not a goal
session_index.db. A derived search index whose own module documents that it may be absent, stale or empty without changing a result.ssmmessages, so a task withSMC_BACKUP_BUCKETset cannot read the bucket and refuses to boot. Prefix-scoped S3 in the role and its permissions boundary is a change with its own review surface, and no shipped producer sets that variable.What changed (motivation → approach → change)
The writer that was missing.
container/sidecar/is the durability pair, deliberately one package rather than two, because a backup whose restore does not return the bytes is worse than an honest gap.store.pyis the bucket asputandgetand nothing else. Nolist, nodelete: alistturns "bring back this task's own files" into "enumerate the bucket", and adeleteputs retention inside the process whose job is to keep things.backup.pyruns one cycle. Each object is copied from ONE open descriptor, bounded to that file's length at open time, so the copy is coherent with no lock and no staged duplicate. Nothing is skipped for its size: an object above the ceiling the restore side reads is uploaded anyway and named in the result, because an operator must hear that from the writer rather than from a customer's failed turn.restore.pybrings the two authority files back and validates them first. Both of the backend's own readers ignore an authority file they cannot parse and carry on with an empty result, which is right for them and wrong here: malformed bytes written by a restore are read once as "no conversations" and then replaced by the flush. The check is exactly as strict as those readers require and no stricter.One derivation of the key.
container/common/keys.pyis the single definition and the front'sobject_keydelegates to it. Drift between a writer and a reader is invisible in both directions, since a GET that misses looks exactly like a customer who never had a conversation.The ordering the restore depends on. The supervisor restores the authority files to completion BEFORE the backend starts, and a failure there raises so the task does not start at all. It starts the sidecar after the front and drains it LAST, so the final cycle uploads what the backend's own drain flushed. Transcripts are not restored eagerly: the front fetches the one a turn continues, on that turn, which keeps the property that a task only holds the conversations it has itself served.
The index must never name bytes that are not in the bucket. A replacement reads the pair to decide which conversations exist and the front then fetches each named transcript, so an index published ahead of its transcripts sends the front to an absent object — which it reads as a conversation that never had history. A cycle therefore uploads in two phases, transcripts then authority, and the pair is opened BEFORE a single transcript is listed so it is one coherent snapshot of the index taken before the enumeration it indexes.
A pair a reader can trust: a generation. The pair is written into one of two slots and a single pointer object names the committed slot. A cycle writes the slot the pointer does not name and commits it LAST, so an interrupted publication damages a slot no reader looks at, and commitment is one object with no half-commitment to observe. The prefix is incremental: existing keys stay where they are and read as generation 0, so no object is deleted, moved or rewritten and a bucket written before the protocol is readable without migration.
A failure no retry resolves ends the process. A denied
PutObject, a bucket that does not exist, an invalid credential: the next interval meets the same answer, so retrying fills the log while the task keeps taking turns nothing will save.StoreUnusableclassifies those on the error code and exits non-zero. A throttle is still retried, because exiting on a transient fault tears the task down and loses the state the backup exists to keep.The platform waits for the drain it is asked to wait for. Fargate's default
stopTimeoutis 30s and the front and backend drains alone can consume it, so the writer was SIGKILLed before its final cycle started. The three windows and their sum live incommon/config.py, the supervisor reads them from there, andcloud/fargate/taskdef.pysetsstopTimeout(CREW_STOP_TIMEOUT_SECS = 115, inside Fargate's 120s cap). The taskdef holds a DELIBERATE DUPLICATE of that sum rather than importing it, because the container tree runs inside the image on a layout wherekiro_crewis absent; the suite reads the image'sconfig.pyas source text withastand asserts the two agree.The final cycle is bounded and says what it did not reach. It carries a deadline, stops before starting an upload it cannot finish, and records every object it did not reach by name — so the cycle is incomplete, the process exits non-zero, and the loss is enumerated instead of silent. The budget is DERIVED from the client's own retry configuration rather than guessed, and the attempt count is measured rather than read off a key's name:
total_max_attemptsis the total wheremax_attemptscounts retries.Deriving that budget was not enforcing it.
connect_timeoutandread_timeoutbound one connect and one read, which is not a bound on a request: a bucket handing over small chunks inside the read timeout trips neither however long the object takes, so the gate reserved 10s and the PUT was free to overrun it. The only thing that then ended a slow upload was the drain window's SIGKILL, arriving mid-request — losing that object and saying nothing about the ones behind it, which is precisely the outcome the deadline exists to replace. A 64 MiB transcript needs about 6.7 MB/s to fit the reserved window, so the condition is an ordinary slow link rather than anything contrived. The cycle now hands the store the window it just reserved and the body stops the transmission when it passes, so the number the gate reserves and the number a PUT may spend are one number. A cut upload is an ordinary refusal: transient, so the pair is withheld and the next cycle attempts the object again, and never classified permanent — that would end the process over a slow network. The store judges by its own reader's clock rather than by the exception that surfaced, because a body raising mid-send may arrive wrapped in a transport error or converted into a retry that fails on its own. An INTERVAL cycle passes no window: it has none to protect and a next cycle to finish the object, so bounding it there would refuse an object slower than the window forever.An unbounded upload must still be ENDABLE. That is the other half of the same rule, and without it the exemption above pays for itself out of the drain. The stop flag was consulted only BETWEEN objects, so a PUT already in flight when the signal landed ran to its own completion — and because the loop begins the final cycle only once the current one returns, while that cycle's deadline is anchored at the signal rather than at the handover, the overrun came straight out of the window that uploads the turns the backend had just flushed. Overrunning the signal by a few seconds is enough to cost them, which is ordinary latency for a multi-MB transcript. The body now takes a predicate asked before each chunk, exactly where the deadline is checked, so the upload ends while it is still progressing; a rewind does not clear it, because botocore's own retry would otherwise restart the upload the stop just ended.
UploadCancelledandUploadDeadlineExceededshare anUploadCutbase: the caller records either as this cycle's refusal and leaves the object to the next cycle, and the store decides which cut happened by asking its reader rather than the exception that surfaced, so neither a slow network nor an ordinary shutdown can reachclassify_permanentand end the process. Both upload phases take it, because an index PUT outlives a stop exactly as a transcript can, and a refusal there leaves the pointer naming the previous whole generation rather than a half-replaced one. The final cycle passes none: the flag is what made it final, so standing down on it would cut the one cycle whose uploads matter. The generation pointer takes neither bound, and the code says why — its body is a few hundred bytes the transport reads in ONE call, so a between-chunks predicate has no second chunk to refuse and there is no transmission to outgrow the client's own per-operation timeouts.The window has to cover every step the cycle spends, not just the uploads. The cycle reads the generation pointer before it accounts for anything, and that
GetObjectcarries no budget — its bound is the client's connect and read timeouts, so it can spend a whole attempt's cost and still succeed. That came off the front of the window while the authority reservation came off the back, which left the data gate an admission window of five seconds from the stop: a pointer read answering in six seconds made the drain cycle refuse every changed transcript unattempted, withhold the pair and exit non-zero with most of the window unused — the replacement then adopting the previous generation's index, which is the loss this change exists to prevent, reached through the machinery meant to prevent it.SIDECAR_DRAIN_SECSnow covers the pointer read too, so the total asked of the platform is 115s and still inside the 120s cap. The arithmetic is pinned from both ends: once against the parts, and once through_reserve_for_authorityand_time_for_one_morethemselves, because a pin that recomputes the sum can agree with a window the production functions reject. The literal drain figure in the sidecar entrypoint's comment is replaced by the constant's name, so a window that moves cannot leave a comment describing the old one.Publication links an inode, not a name — and only reports success when that inode is reachable. The temporary is created through the same directory descriptor the link publishes into, and the link names the still-open descriptor through procfs, so the bytes published are the bytes written. Pinning the parent is not the same as reaching it, though: a rename of that directory after the descriptor is opened is invisible to every step at once, so the temporary is created in the detached directory, the link publishes into it, both agree, and
link_newreturnsTruefor bytes at a path nothing resolves to. The caller reads that as published, so the backend starts the conversation with no history and the next backup uploads the emptiness over the bucket's copy — the same loss, through the one path that claims to prevent it, reachable by a single rename in a directory this module documents as agent-writable. The published inode is therefore compared with what the parent path resolves to at the end, on device and inode rather than on existence, because the name can resolve to a different file in the replacement directory. A mismatch unlinks the copy through the pinned descriptor — so it cannot reach a bystander at that path — and raises. These are directories the agent writes in, so the window was reachable.A symlink above a file is as dangerous as one at it.
O_NOFOLLOWguards the last component only, soos.walkdescended a link planted atsessions/archiveand uploaded files behind it. The open now walks down from the data home one component at a time with the link refused at each, and a linked archive root is refused before anything under it is listed.What the backend discards, its next flush deletes. Validation refuses individual entries as well as whole-file shapes, because the backend's session-map loader meets an unusable value with a bare
continueand its open-slots screen folds a bad member to nothing — so a file that parses and whose entries it drops one by one is the same loss arriving quietly.sidmust itself be a string; an empty one is legal, because the backend writes one when a session id stops resolving. The one discard this deliberately does NOT refuse is an open-slot key holding a path separator: the backend rejects that as an escape from the sessions directory and warns, so its removal is the designed outcome, and refusing the boot on it would let one planted byte deny every replacement task permanently.The copy that is KEPT is now read too.
link_newrefuses an existing target, so a local authority file is kept instead of the bucket's copy being written — and that branch was the one branch that never read what it kept. A malformed local file was loaded as no conversations and the first flush spent the valid bucket copy: the loss this module exists to stop, arriving through the branch meant to prevent it. The local file is now read with the symlink refused and the shape taken off the descriptor, and judged by the same check the bucket's bytes get. It is refused rather than overwritten, because the local copy leads the bucket by up to one interval and replacing it would roll a conversation list backwards; an operator repairs either copy, and both are still there. The same reading covers a name the bucket holds nothing for, since a file there boots the backend whether or not anything was restored over it.Withholding the index is a WAIT, so only something that will arrive earns it. The cycle withheld the authority pair whenever anything was refused — including a refusal decided by an entry's SHAPE, which every later cycle meets as well. One symlink or FIFO named
*.jsonlundersessions/, or any non-regular file undersessions/archive/, therefore frozesession_map.json,open_slots.jsonand the generation pointer indefinitely while transcripts kept uploading, and the next replacement restored an index predating every conversation served since. Refusals are now split by cause: a transient one (a directory missing right now, a descriptor that could not be opened, a failed upload, an object the drain window could not fit) still withholds the pair, because the next cycle publishes one the bucket supports; a permanent one does not, because that wait never ends. Either way the cycle is incomplete, the entry is named, and the exit code says so. The residue is stated rather than hidden: if a planted name collides with a session the index does name, that one conversation's history is absent behind an index that advanced past it — bounded, named in the cycle's report, and repaired by removing the file, where the freeze was unbounded, silent and repaired by nothing.The reserved number is the whole request, not just the transmission. A body-enforced deadline stops at the last chunk: once the body is drained the transport waits for the RESPONSE and never asks the body again, so that wait is bounded only by
read_timeout. A PUT could therefore finish transmitting exactly on its allowance and still spend one more timeout — the gate admitting an upload the window cannot hold — and with the drain deadline anchored at the SIGKILL instant, an ordinary S3 slowdown at teardown was enough to reach a kill mid-PUT.BACKUP_PER_OBJECT_BUDGET_SECSnow covers that wait,BACKUP_TRANSMISSION_BUDGET_SECSis the smaller share the store hands the body, both derived so they cannot drift apart, and the drain window grows with the reservation to 75s — 115s asked of the platform, inside the 120s cap.The root guards list through the descriptor they checked. Checking a descended descriptor and then re-resolving the root by name is two lookups of one directory: the root can be swapped between them, so the listing walks the very tree the check refused while the check reports it sound, and the refusal buys nothing. The sessions root is scanned through its descriptor and the archive is walked with
os.fwalkfrom its own, so the check and the enumeration address one inode in both — the same rule publication follows.Some of the authority pair is not the same as none of it. With no whole pair there is no generation to commit, so the pointer stays where it is — and a restore that finds no pointer reads the legacy keys, which this writer never writes. Uploading the one file it holds therefore put it in a generation nothing can reach, while the cycle reported itself complete and exited zero: an index silently absent rather than an index preserved. The two files have independent writers, so a one-file window is ordinary timing skew rather than an extreme state, which is exactly why it must not be the quiet path. None of the pair stays a non-failure, because a first boot has no index to preserve and nothing to say; some of it is refused, which withholds the file that is there and makes the cycle incomplete, so the next cycle publishes a whole pair into a generation a reader can reach and the exit code names the wait instead of hiding it. Transcripts keep uploading either way — the index is what waits, not the conversation.
A refused data ROOT withholds, which is where it parts from that split.
os.scandirandos.walkresolve the root they are given, so a link planted atsessions/orarchive/lists a stranger's files as this task's, and each one is then refused on the way down as a per-ENTRY refusal. Classified as the residue above, every transcript is refused while the pair publishes, and the replacement reads each absent object as a conversation that never had history — silent loss of all of it rather than of one name. The residue is priced as one planted NAME among transcripts that are otherwise reaching the bucket, where freezing the index would cost every one of them its updates; a root is not one name, nothing is reaching the bucket, and holding the pointer at the last complete generation gives up nothing that was going to be uploaded. The sessions root gains the descent check the archive root already had, so neither is listed through a link.A bucket with no pointer is read by surveying its generation slots. At generation 0 the writer's own safety inverts: the reader looks at the LEGACY keys, so the slot a cycle published into is not merely unreferenced, it is invisible. A pointer PUT that fails there leaves a COMPLETE pair nothing names — complete because the pointer is written last, only once both files are already in the bucket — so that pair is a publication and is adopted, winning over the legacy keys, which no cycle of this writer writes and which are therefore older. Refusing it would be unrecoverable rather than cautious:
restore_authorityruns in the supervisor beforestart_backendand the sidecar, so a refusal leaves no writer to publish the pointer it waits for, every replacement repeats the same refusal, and repair is manual bucket surgery. Two states are still refused, and each names manual repair instead of a cycle that would not run: two complete pairs with no pointer cannot be ordered, and the protocol never produces that state since a second slot is written only once a pointer names the first; and a slot holding PART of a pair is a publication torn before anything referenced it, refused only when the legacy keys cannot supply the missing name either, because a bucket whose legacy pair is whole can still boot and refusing it would strand a readable index. Without the survey the hole runs the other way: no pointer has ever existed, so the restore reads the legacy keys, the pair went to a slot instead, both names come back absent, and the task boots on an empty index while the real pair sits one key away. A slot read that fails for any reason other than absence refuses, on the same rule the pointer read follows.A conversation that vanishes mid-cycle must not leave the index ahead of it. The pair is captured BEFORE the enumeration, so it still names a conversation deleted during the cycle — and the skew argument that calls an older index harmless rests on every slot it names already having bytes in the bucket. A conversation created and deleted inside one interval was never uploaded by any cycle, so committing that pair named a slot the bucket has never held, and the front read the absent object as a conversation that never had history; on the final drain cycle there is no later cycle to repair it, and that is the pair a replacement boots from. A vanished transcript now withholds the pair when the captured index NAMES it and the bucket does not hold it. Membership is the test rather than the disappearance: an index that never named it cannot send a reader to it, and an owner deleting a conversation is ordinary use, so that case stays the non-failure it has always been. Unlike an unreachable entry this cannot freeze the index — the verdict is a race within one cycle, not a shape on disk, so a file that is really deleted is not listed by the next cycle and the pair publishes then. The referenced set is read with
os.preadoff the snapshot's own descriptor, so the offset the upload reads from is untouched and the bytes examined are the bytes that would be published; a captured index that cannot be read counts every candidate rather than none, because guessing "names nothing" would commit an index this cycle could not read against bytes it knows are absent.A complete pointerless pair is adopted only once its two files agree with each other. Presence of both names proves the pair COMPLETE, not that one cycle wrote it: a failed pair PUT leaves the half that landed, and the next cycle is still pointerless, so it targets the same slot and can fail on the other half — two consecutive cycles then assemble one slot out of two publications, which is the mismatched tabs and session bindings themselves. Origin cannot be established from the bucket without the object that failed to be written, so what is checked is the property the reader depends on: every open tab must be a conversation the session map knows. A pair naming a tab its own map does not know is refused, naming manual repair, since no pointer will ever name it. That is deliberately a weaker claim than same-cycle origin, and it is the one the backend actually rests on.
The archive root is the one root whose refusal must not withhold the pair. A link or a non-directory there is a SHAPE, so every later cycle meets the identical error: a withholding started there never ends, and the index freezes at the moment the name appeared while live transcripts keep uploading past it — the unbounded freeze
RefusedEntrynames as the thing that must never happen, reached through the guard added to stop the bounded loss. It buys nothing even once, because the front deliberately never fetches archived segments (finding them would require listing), so the published pair cannot send a reader into the subtree that went unenumerated. A shape refusal there is recorded as unreachable instead: the cycle still fails loudly and names the entry, and the pointer advances. The live root keeps withholding, because the pair does name the transcripts a reader fetches.Every error the archive walk meets is collected rather than skipped.
os.fwalkswallows anOSErrorand continues whenonerroris unset, so a directory the task's uid could not open contributed no segments, no refusal and no log: the cycle reported itself complete, the pointer advanced over an index naming conversations whose older halves were never uploaded, and the archive lives on an ephemeral task disk — so those segments were gone with no record of which ones. They are split on the same permanence rule as the root.The index and the filesystem name a conversation differently, and comparing them directly matched nothing. The index names it by SLOT KEY (
cust-8831) while its transcript isdashboard_cust-8831.jsonl, so the vanished-transcript check compareddashboard_cust-8831againstcust-8831and never fired — the rule was dead in this branch and the loss it describes was still reachable. The slot keys are now mapped FORWARD through the front's owntranscript_stem, the function that decides the real filename, rather than the prefix being stripped here: stripping by hand would also miss that function's character substitution, so a key holding an unsafe character would map to a name no comparison made. The suite named the index with transcript stems, which is why it agreed; it now names slot keys and asserts the two are related by that function.One spelling of the shared judgements.
common/objects.pyowns absence, permanence and the bounded read, and both the writer's store and the front's reader use it. The two private copies had already diverged on exactly the missing-bucket case above; the front's_error_code,_ABSENT_CODESand chunked reader are deleted.Two subtractions. The
SMC_BACKUP_INTERVAL_SECSfloor of five seconds asserted a cost nothing had measured, so only the derived refusal remains: zero or less is a busy loop, any positive value is accepted.keys.sessions_prefixhad no caller and is gone.docs/system-specs/modules/aws-control.mdis updated as AGENTS.md requires: four processes rather than three, the restore as step 3 with its ordering rule, what durability covers and what it does not, and the overlap residual below.Backwards compatibility
Breaking only for a bucket layout no shipped writer ever produced: this version removes the generation-0 legacy-compatibility path, so a bucket with no committed pointer now reads as empty (a first boot) instead of being read through the pre-generation
data/sessions/<stem>.jsonlanddata/<authority>.jsonkeys. Writer sweep on origin/main @ 5786c77 (the merge-base): nothing in this repository writes that legacy layout —grepfinds noPutObject/putofdata/sessions/*.jsonlor the legacy authority keys on any entry point — so no bucket in existence contains it, and the pointer-based protocol this PR ships is the only layout a shipped writer produces. A pointer naming an authority name this version does not know is still dropped rather than refused, so a rollback onto a bucket a newer writer committed a third file into stays bootable.The removal takes away four public key-derivation helpers. Each reader of what they derived, found by grepping origin/main across every entry point (chat, cron, subagent, app bundles, crew page, release):
Reader: src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/front/transcript.py:_resolve_transcript_key -- app -- the only caller of
object_key; it was the generation-0 fallback and is removed in this same diff (no-pointer now resolves to absent), so nothing calls the deleted helper -- test_front_transcript_fetch.pyReader: src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/sidecar/restore.py:restore_authority -- app -- the only reader of
keys.authority_key(the legacy authority location); the no-pointer branch that read it is removed here, so a pointerless bucket boots empty rather than reading those keys -- test_sidecar_restore_refusals.pyReader: src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/sidecar/backup.py:_merge_transcript_index -- app --
keys.transcript_keyandsessions_prefixwere used only by the deleted_migrate_legacy_transcripts; no caller remains after the migration layer is removed -- test_sidecar_durability_gaps.pyTwo deliberate tightenings, neither reachable from a shipped configuration. A task with
SMC_BACKUP_BUCKETset and no usable credential now refuses to boot rather than starting with an empty slot table — the intended direction, and no shipped producer sets that variable.BackupSet.refusedbecomesrefusedplusunreachableandCycleResultgainsunreachable; both types are private to the image package and have no caller outside it.Tests
The container lane collects 581 on this head and passes with
-n0;test/test_fargate_taskdef.py,test/test_fargate_runtask.py,test/test_ci_surface_tests.py(which loads this suite's conftest and measures its branches) and the container-image import audit pass alongside it.test_sidecar_round_trip.pyis the acceptance test. It writes a conversation under one data home, runs a real backup cycle into an in-memory bucket, then builds a SECONDSettingsover an empty directory with the same crew identity and copies nothing between them. The replacement reads the transcript back through the shipped front reader and both authority files through the restore step, and the bytes are asserted identical. It also pins the consistency property directly: a transcript appended to after its descriptor is opened uploads the length it had, not the length it grew to.test_sidecar_backup_refusals.pycovers every branch that can end in "not uploaded", and now pins both halves of the withholding rule: a failed upload withholds the pair, a permanently unreachable entry does not, and a SECOND cycle with the same planted name still republishes the index — which is where a freeze shows itself.test_sidecar_restore_refusals.pycovers the fail-closed direction, including that a denied read is not read as absence, and now that a malformed LOCAL authority file refuses the boot both when the bucket holds a valid copy and when it holds nothing, that a valid local file is still kept unchanged, and that a FIFO or symlink at either name is refused rather than followed.test_supervisor_startup_order.pypins both orderings on the recorded SEQUENCE of calls, since a test that only checked each step happened would pass on either order, and pins that the drain hands back the SIDECAR's status.test_supervisor_exit_code.pypins that a committed final cycle stays successful while a failed one, a drain-window kill and an unreadable status each exit non-zero.test_fargate_taskdef.pypins the platform half:stopTimeoutcovers the three declared windows, equals their sum, and stays inside Fargate's cap. A fourth case pins the pin itself, so renaming a constant in the image reds here instead of making the other three vacuously pass.test_sidecar_durability_gaps.pypins the durability paths one group each, plus the two directions that must NOT change: a throttle is still retried, and ordinary nested archive segments are still uploaded. Its newest group pins the enforced upload window from both ends — a body past its window stops sending and names the bytes that did not go, a rewind does not refresh it, the final cycle hands the store exactly the number its gate reserved while an interval cycle hands it none, a cut upload is refused rather than lost and leaves the pair withheld, and a transport that swallows the body's own error is still read as the window rather than as a permanent fault. The group after it pins the other half: a body cut by a stop stops sending and names the bytes that did not go, the stop is checked before the read rather than after it so the refusal never waits on one more chunk, a rewind does not clear it, a body given no predicate reads to the end, an interval cycle hands the store the stop flag while the final cycle hands it none, both phases are cut, a cut upload is refused rather than lost and never recorded as unreachable, a cut index leaves the pointer where it is, and a transport that swallows the body's error is still read as the stop. The last group pins the window against the work it has to hold: the pointer read, the authority reservation and one worst-case data PUT, checked both against the parts and through the real gate after a slow pointer read has been spent, and that the total stays inside the platform's cap. The two groups after it pin publication and the roots: a parent renamed under a publish is refused rather than reported, including when the replacement directory already holds that name so mere existence would be satisfied, while an unchanged parent still publishes and an existing target is still reported as already there; and a refused root withholds the pair for both roots while one planted name under a healthy root still does not, with the healthy transcript beside it still reaching the bucket. The last group pins the pair's completeness in all three directions: one file present is refused and withheld rather than half-published, none present is still a clean first boot, and a whole pair still publishes and commits. The last group pins the request accounting: the reservation equals the transmission share plus one response wait, the store hands the body the smaller of the two, and the window still covers every step with the response allowance added. A further pin swaps the sessions root after the check has passed and asserts the listing still reports the checked directory's own entries. The newest group pins the generation-0 slot survey in both directions: a complete pair in either slot with no pointer is ADOPTED and boots, and beats the legacy keys when both are present; two complete pairs refuse as unorderable; a torn slot refuses only when the legacy keys cannot supply the missing name, and does NOT refuse a bucket whose legacy pair is whole; an empty bucket still boots because that is the genuine first boot; a slot that cannot be read refuses; and a bucket that cannot be read at all staysStoreUnusablerather than being reported as a generation problem. The group after it pins the vanishing-conversation rule in all five directions: a gone transcript the captured index names with nothing in the bucket withholds the pair and leaves the pointer alone, one the index does NOT name still publishes, one the bucket already holds still publishes, an index that cannot be read withholds rather than assuming it names nothing, and a name that appears only inopen_slots.jsonis enough to withhold — plus that the very next cycle publishes the pair, which is what separates this from the permanent freeze an unreachable entry would cause.Every group is checked for discrimination by putting the reviewed defect back one at a time and re-running only the affected pins, restoring each file from a copy taken outside the repository and verified by hash —
git checkoutwould revert the uncommitted fixes and score every later mutation against HEAD. Each mutation is asserted to match exactly one site, and a run whose output carries no explicit pass/fail count is refused rather than scored, because a selector naming nothing collects nothing and exits non-zero. Re-keying the withholding on unreachable entries reds 3 pins; keeping a local authority file without judging it reds 2; making the body never expire reds 4; bounding the interval cycle too reds 1; classifying an expired window as permanent reds 1. For the cancellation group, all eight mutations are caught: a body that never cuts on the stop reds 4; checking the stop after the read reds 1; either phase handing the store no flag reds 2 and 1; the store classifying a cut by the exception rather than by its reader reds 1; the two cuts not sharing a base reds 1; the pointer moving over a refused authority phase reds 1; and a body with no predicate cutting immediately reds 1. Narrowing the drain window back to the uploads-only size reds 2, and widening it past the platform cap reds 1. Dropping the sessions-root refusal reds 1, listing that root through the link again reds 1, and routing either root's refusal back into the non-withholding list reds 2. Publishing without the reachability check reds 1, and comparing existence instead of the inode reds 1 — that one needed the case where the replacement directory already holds the name, since existence alone decided every other case and the mutation survived until it was added. Publishing a partial pair without the refusal reds 2, and widening that refusal to cover the no-file first boot reds 1. Listing the sessions root by name after the check reds 1, replacing its descent with a following open reds 1, and handing the body the whole reservation instead of the transmission share reds 1. For the generation-0 slot survey, all nine mutations are caught: never adopting a complete pair reds 3 and a source that ignores the adoption reds the same 3; resolving two complete pairs by picking one reds 1; letting a torn slot through reds 2, and widening that refusal so it fires on every first boot reds 3 — those three being the pre-existing must-boot pins, so the over-refusal direction is guarded by tests written before this fix; counting an unreadable slot as absent reds 1; translatingStoreUnusableinto a generation problem reds 1; surveying only the first slot reds 2; and treating a partial slot as complete reds 3. For the captured-index membership rule, all six mutations are caught: judging nothing referenced reds 3, a gate that ignores the verdict reds 1, reading an unreadable index as naming nothing reds 1, skipping the durability test reds 1, acompletethat ignores the withholding reds 4, and reading only the session map while ignoring open tabs reds 1 — that last one is why a pin exists in whichopen_slots.jsonalone names the conversation. For round nine, all eight mutations are caught: reverting the namespace mapping to the direct comparison reds 3, which is what proves that rule was dead before; making the archive root withhold whatever its cause reds 1; treating every errno as permanent reds 1; letting the walk swallow errors again reds 1 and dropping the ones it collected reds 1; ignoring the coherence check reds 1, making it always pass reds 1, and never collecting the open tabs it compares reds 1. Forty-two mutations, forty-two caught. The unmutated tree is green before and after, with every touched source verified hash-identical to its pristine copy.Manual verification
N/A — the subject is a container-image process with no interactive surface, and the round-trip test exercises the real backup and restore code paths against a second
Settingswith nothing copied between them.Related Issues
Refs #9464. Residuals split out: #13706, #13707.
Pattern harvest
Rule candidate: review-prompt
Pattern: a guard that WAITS for a condition to clear must be keyed on whether that condition can clear. Both defects fixed in this round are the same shape — a branch that skips work as a temporary safety measure, reached by an input whose cause is permanent, so the measure never lifts. The reviewable question is: for each "skip this, a later pass will handle it", what makes the later pass different?
Checklist