Skip to content

feat(aws-control): retire old cloud backups by version after each push - #12159

Merged
iamwhatever merged 1 commit into
mainfrom
feat/cloud-backup-retention
Sep 20, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/cloud-backup-retention

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

1. What is the problem?

Cloud backup uploads an archive and never deletes an old one. Both push paths mint
a key carrying a timestamp (sessions-<stamp>.tar.gz, kirocrew-snapshot-<stamp>.tar.gz),
so nothing is ever overwritten: every nightly run adds one object and removes none.
The word delete did not appear anywhere in backup.py.

Measured on a real drive: 15 snapshot archives, 10.1 GB, the newest 2.49 GB, the
oldest three weeks old. The bucket's size had never once gone down.

The local side already bounds itself. snapshot_main([tmp, "--keep", "1"]) prunes
after each local snapshot. The cloud side had no counterpart.

2. Why this issue matters to the user

It is the user's own AWS bill, growing at roughly one archive a night forever, for
history nobody asked to keep. Nothing in the UI says it is happening and nothing
stops it: the drive is the user's bucket, so the cost lands on them and the only
remedy available today is deleting objects by hand in the console.

3. How our fix solves it

The symptom is a bucket that only grows. Walking it back:

  • the bucket grows because nothing deletes -> add a sweep;
  • a naive sweep would delete the archive the run just uploaded -> never retire the
    newest;
  • a naive sweep would delete another machine's archives -> scope it;
  • a naive sweep would not actually free a single byte -> delete versions, not
    objects.

That last step is the one worth reading. The drive has versioning enabled and
no lifecycle rule. A delete-object without a version id writes a delete marker
and turns the bytes into a noncurrent version that goes on being billed, so a
retention pass built that way empties the listing and saves nothing. A lifecycle
noncurrent-expiry rule does not solve it either: the 15-object pile is 15 current
versions under distinct keys, and a time-based rule cannot express "keep the newest
three". So the sweep is version-aware end to end. Two new storage primitives,
list_object_versions (complete-or-raised, folder-scoped) and
delete_object_versions (version-pinned, with no prefix argument that could widen
the blast radius), erase the bytes and leave no marker behind. storage.py already
named this purge as missing in its own docstring; retention is its one caller.

backup._prune_remote_archives runs as the LAST step of a successful push:

  • It ships OFF, and there is no count that applies without being configured. A
    fresh install and an upgraded one both keep every archive, exactly as before, so no
    first run after an upgrade deletes anything. Retention turns on only when this
    account's state holds a usable count under retention_keep; that key IS the
    switch. Deleting an object version cannot be undone, and these are the operator's
    bytes in the operator's bucket, so the two ways of being wrong do not compare:
    shipping off costs storage they can see in a listing and reclaim with one command,
    while shipping on silently destroys archives somebody was deliberately keeping and
    leaves them nothing to restore from. That asymmetry is not this PR's invention: the
    repository already recorded it. docs/request-for-change/rfc-s3-backup.md O5
    ("retention, partly answered", lines 370-383 on main) states that a lifecycle rule
    "can express delete older than N days but never keep the newest N", that on a host
    which stopped backing up such a rule "would delete the last surviving copy of its
    memory exactly when it is needed", that "unbounded growth is the cheaper failure", and
    that what remains open is "whether operators want an opt-in, count-based remote prune,
    which needs a lister and a deleter rather than a lifecycle rule". Opt-in, count-based,
    a lister and a deleter: that is this change, in the RFC's own words, including the
    reason the newest archive is protected by name. What the RFC leaves open is whether
    operators WANT it, and shipping off is precisely what leaves that open -- an install
    that sets no count is in exactly the state the RFC describes today. It is also the
    house shape rather than a new
    policy -- nightly ships off, reads fail-closed, and is never inferred from an
    adjacent grant, and a retention default that deleted would be the first capability
    here to act destructively on operator data unasked. The switch costs one key read,
    so there is no engineering case for omitting it. This does not ignore the request
    that started the work: "keep only the most recent one" is someone choosing a value
    for their own install, which default-off plus a count delivers exactly -- they set
    1 -- while leaving every other install's bytes where they are. A default would
    have generalised one operator's choice into everyone's deletion policy.

    On the RFC's status: draft (line 3): conceded, and it does not change the above. The
    citation is used for SCOPE, not as an approval -- the shape it predicts is the shape
    shipped, and a draft recording that framing is still the repository's framing rather
    than a reviewer's or an author's. Note the "Blocks neither milestone" clause elsewhere
    in that file belongs to O1, which is about the bundle's secret policy, so it is not
    quoted here as covering O5.

  • Every unusable stored value reads as off, not as a fallback number. Absent, a
    string, a float, a bool: none is a count somebody chose, and a fallback would be a
    permanent delete performed on a value the operator never wrote. bool is screened
    before int because True IS an int in Python and would otherwise read as
    keep=1, the most destructive value available. Zero and negatives DO clamp to
    [1, 50] rather than switching retention off, because they are counts somebody
    wrote and reading a typo as "off" would silently stop doing what was asked for.

  • Both roads to off keep everything; only their audit differs. Nothing configured
    is the ordinary state of an unconfigured install, so the sweep succeeded at having
    nothing to do. A state file this process could not read keeps everything too but is
    audited failed, because an operator who DID configure a count is silently not
    getting it. keep is reported as "off" in both, never as a number the sweep did
    not act on. Neither is ever derived from nightly in either direction: authorizing
    unattended uploads is not authorizing permanent deletes, and turning the nightly
    off does not withdraw a configured count.

  • The newest is never retired, and that is separate from the count. The key this
    run just uploaded is dropped from the candidates by name, whatever the count says,
    including 1. The floor at 1 also spares the first entry of the age order, but
    that is a different claim: the run's own key is only first while nothing else
    carries a later timestamp, and a co-writer with a skewed clock is enough to move
    it -- which is exactly the case the by-name guard exists for. Removing that guard
    while keeping the floor reddens a test on its own. And if the listing does not show
    that key, nothing is pruned at all: a view missing the newest object cannot be trusted
    about which objects are old. This is the cloud form of the guard --keep already
    carries locally, where an incomplete bundle does not prune.

  • Per kind and per install. snapshots/ and sessions/ are separate histories
    with separate cadences, so counted together a burst of one would evict the other's
    only copy. One drive is reachable by several installs by design, so the listing and
    every delete are anchored on this install's own prefix.

  • Best-effort and last. The upload is the point of the run. Every failure is
    caught and logged as one line, so a backup whose archive is safely off-host is
    never reported as failed over a cleanup that was not.

  • Audited either way. This is the only path in the app that erases object
    versions permanently, and being best-effort is exactly why the audit entry
    matters: a failure that only logs is one nobody reviewing the trail can see. Each
    terminal outcome files one SEL_OP_RETENTION event carrying its counts, and the
    gate's own refusal is filed once, where the decision is made, rather than twice.

  • Two things it deliberately leaves alone: the label sidecar (not an archive, so it
    holds no keep slot and is never deleted) and a key whose current version is a
    delete marker (its noncurrent bytes belong to the manual delete path, whose
    documented recoverability this must not silently revoke).

The sweep runs _authorize_upload TWICE, under its own audit operation: once before
the listing, and again immediately before the delete. These are deletes of the
owner's data running after a build that may have taken minutes, so consent,
app-enabled state and the account behind the profile are re-checked rather than
inherited from before the push. The second gate exists because a listing round trip
separates the two, and consent withdrawn in that window must not reach an
irreversible call. A refused sweep
is not filed as a denied upload, because this run's upload succeeded.

4. What tests we did

test_aws_control_backup_retention.py is new: 74 tests, no network, no profile, no
bucket. Two fake layers, and which one a test uses is that test's point. The
retention tests fake the storage calls and assert what retention DECIDES (keep the
newest under every bad configured value, per-kind and per-install scoping, the
sidecar and delete-marker exclusions, the refusal when the listing omits the
upload, one SEL event per terminal outcome, and the best-effort contract). The
storage tests fake _checked, the package's single AWS CLI chokepoint, and assert
the argv that would have gone to the CLI, so "every delete carries a VersionId" is
pinned on the wire rather than inferred.

212 tests pass across the three affected suites with -n0. flake8, isort, the
baselined black gate and docs-lint are clean; mypy's 4 errors are in
transcribe.py and ops_mission_control, untouched here.

One finding came out of running the suites rather than reading them. Adding the
sweep made four existing push-path tests attempt a live aws s3api list-object-versions, and because the sweep swallows Exception by design, three
of them still PASSED while doing it. It was found by re-running under a throwaway
plugin that raised a BaseException the sweep cannot catch, then fixed in the
three autouse fixtures so future push-path tests are covered too, and re-verified:
zero live version-listing calls remain.

5. Any other suggestions on the work

  • The existing pile is never cleaned, not even gradually. Retention bounds the
    drive going forward, from the next successful push of each kind. Ownership is
    proven by the VersionId recorded at upload, and an archive pushed before that
    record existed has no id in it, so the 10.1 GB already sitting in that bucket is
    kept permanently. That is not only history: the version record is trimmed with the
    upload record under one bound, so an install that keeps pushing with no count
    configured drops its oldest recorded version once it passes that bound, and those
    archives stop being retirable too. Retention ships off, which makes this the ordinary
    path rather than an edge case. Tracked as issue 12274, because raising the bound only
    moves the cliff and letting the sweep adopt an archive it has no record for would
    weaken the one property the ownership test exists for. Draining it needs either a one-shot reclaim or a migration that
    adopts those versions by matching each against the body fingerprint uploads
    already stores. Both are separate changes, and this PR does not reduce the
    measured pile that motivated it.
  • Turning it on is an API call; there is still no panel control.
    POST /backup/{account}/retention sets or clears the count, beside the nightly
    toggle and guarded the same way, and the status read reports the effective count as
    retentionKeep. It takes the count rather than a flag because a count is on and
    null is off, and null is what turns it back off -- a switch that could only be
    turned on would leave an operator editing state by hand to stop it. What remains
    missing is a control in the console UI, which is a front-end change and a separate
    surface; with the feature off by default, its absence means "nothing is deleted"
    rather than deletion nobody asked for.
  • Noncurrent bytes from the manual delete path are still billed. delete_key
    writes a marker on purpose, for recoverability. Retention does not erase those
    versions, so that cost predates this change and remains; a lifecycle
    noncurrent-expiry rule on the drive would be the natural home for it.

@chenmingwei23
chenmingwei23 requested review from a team and dwu96 September 20, 2026 02:39
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 18523e2ac54529d311a66cfa0c988cb42ace5de9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Description says keep clamps to [1, 50]; _clamp_retention_keep and the route explicitly ship no ceiling — reconcile that sentence before merge.

[DESIGN-REVIEWED] 18523e2

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 18523e2ac54529d311a66cfa0c988cb42ace5de9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify one thing: retention reclaims nothing already on the drive — the measured 10.1 GB pile is kept permanently (pre-record archives have no version id; deferred to issue 12274).

What this change ships

Inventory (9 items) — 9 justified

Intent: Stop each successful cloud-backup push from growing the operator's bucket forever, by retiring this install's oldest archives once an operator opts in with a keep count. ADDITION.

  1. After a successful push, oldest cloud archives of that kind are permanently erased by version — only once a count is configured; ships off — justified
  2. New POST /backup/{account}/retention sets or clears the count; null turns it off — justified
  3. Backup status now reports retentionKeep (effective count or null) — justified
  4. Every upload now persists the S3 VersionId it created (upload_versions in backup.json), on every install — justified
  5. Two new storage primitives list and delete object versions; retention is their one counted caller — justified
  6. Every push now files one SEL retention audit event, even with retention off — justified
  7. A non-UTF-8 backup.json now reads as unreadable instead of crashing readers or being overwritten by the next settings write — justified
  8. Partial delete-objects failures now carry how many entries succeeded, so audits never report an erased batch as zero — justified
  9. Archives with no recorded version are never reclaimed; the sweep reports their count and bytes — justified

Checks run: RFC rfc-s3-backup.md O5 (base lines 370–383) matches the description's quotes verbatim; it is draft on the base list, but a default-off opt-in count is explicitly not product shape, so lens 9 does not fire. Website consumers of retentionKeep/the route: 0 (grep retentionKeep|retention_keep in website/src) — the route is the feature's only switch, so its zero option leaves operators hand-editing state. Existing mechanisms counted: delete_key/delete_prefix write markers only (base docstring itself named the version purge as missing); S3 lifecycle cannot express keep-newest-N across distinct keys, per the RFC. Local sibling snapshot_main([tmp, "--keep", "1"]) confirmed at backup.py:2075. Forward-only reclaim is stated plainly in the spec text this PR adds and tracked in issue 12274.

[FIRST-PRINCIPLES-REVIEWED] 18523e2

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 18523e2ac54529d311a66cfa0c988cb42ace5de9 — this comment is updated in place on each push.

Review details

Both candidates fail falsification:

Candidate 1 (_put_version_id crashes on non-dict JSON): The input to _put_version_id is stdout from s3api put-object --output json, which the AWS CLI always renders as a JSON object ({"ETag":..., "VersionId":...}), never a top-level array or scalar. The parsed.get(...) AttributeError path requires json.loads(out) to return a truthy non-dict, which does not occur in practice from this command. The candidate itself concedes the branch is "essentially unreachable." Condition (a) — a concrete input that occurs in practice — is not met. This is defensive-parity, not a demonstrated defect. Drop.

Candidate 2 (spec paragraph inaccuracy): The technical observation is plausible — _merge_uploads trims upload_versions to the keys still in uploads, so a key evicted from uploads leaves ours, never enters by_key, and thus cannot appear in unclaimed. But (c) is a claim that a doc sentence is imprecise about an acknowledged limitation (tracked in issue 12274), not an observable wrong outcome of executed code. It is a spec-accuracy nit, which does not meet the executed-code bar, and the candidate rates it low. Drop.

No new grounded finding survives the same bar.

No findings.

[OPUS-REVIEWED] 18523e2

Verdict parsed from the review's SHA-scoped output markers for commit 18523e2ac54529d311a66cfa0c988cb42ace5de9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 18523e2ac54529d311a66cfa0c988cb42ace5de9: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 18523e2ac54529d311a66cfa0c988cb42ace5de9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 18523e2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 18523e2ac54529d311a66cfa0c988cb42ace5de9: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from 1fccce8 to bdd0454 Compare September 20, 2026 03:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from bdd0454 to e5883a0 Compare September 20, 2026 04:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from e5883a0 to 2ece38b Compare September 20, 2026 04:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition of the First Principles advisory items (the Item 1 BLOCK is not addressed here and needs a maintainer).

Item 3, zero consumers - ACTED ON. Verified independently before removing it: grep -rc retention website/src/apps/aws-control/ returns no file with a non-zero count, and BackupStatus in types.ts is unchanged. The retention line is deleted from _handle_backup_status, so routes.py is now byte-identical to base and has left the change set (the black gate's scope went from 7 files to 6). An API field nothing reads is not a feature, and shipping one invites a reader to trust a number no surface displays.

Item 2, the knob's only setter is a hand-edited file - KEPT, and the gap is real. Configurability with a default of 3 is a stated requirement of the task this PR implements, not something the diff reached for: a single retained copy means one corrupt upload tonight leaves no history at all, which is why the count is a value and not a constant. So the subtraction is declined on that ground rather than on a code argument.

The finding is accurate about the consequence, and removing the status field above makes it more so: after this push the key genuinely has no reader or setter outside the sweep. That is a disclosed gap, not a solved one. Wiring a control is a separate change with its own surface to design, and folding a settings UI into a retention PR would be exactly the scope growth this review exists to catch.

Item 6, the unreadable-state abort rides along - KEPT, conditional on Item 2 as the review says. The dependency is stated correctly: if the count were a constant there would be no state read to fail, and the machinery would have nothing to guard. It is retained because the count is retained. Its own justification is independent of that, though: reading an unreadable state file as the default silently converts a configured 50 into 3 and permanently erases the other 47, so on this path a default is not a safe fallback but a destructive one.

GPT's two findings on the previous head are fixed, not overridden. Per-version ownership: presence in the upload record proved a KEY while retirement erases every version under it, so a second object version on a recorded key now leaves that key alone entirely - only a foreign write produces that state, and the record stores a body fingerprint per key rather than a version id, so nothing can say which version is ours. Mixed batches: Quiet is set on every delete-objects call, so a 200 response names only the entries it could not remove and the successes inside a failing batch are now counted into the reported total.

Both are mutation-verified. The first mutation run exposed one of my own tests as vacuous - a foreign overwrite dated after the newer archive made its key sort newest, so it never reached the retirement list and the test passed with the guard removed. The fixture is corrected and the test now fails when the guard is loosened.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 18523e2ac54529d311a66cfa0c988cb42ace5de9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from eba4139 to 85a46ae Compare September 20, 2026 05:45
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from 46477c8 to ce82f35 Compare September 20, 2026 08:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Retention now ships OFF (head ce82f35)

This head changes the shipped default, which answers the outstanding product-shape
item rather than working around it. The reasoning is in the PR body so a reviewer can
weigh it there; this comment records what moved and what proves it.

Four parts:

  1. An unconfigured install keeps every archive. Fresh or upgraded, no first run after
    an upgrade deletes anything.
  2. The count key IS the switch. Absent, or holding anything that is not an integer,
    means keep everything. There is no longer any constant carrying a default count,
    and a test asserts the symbol does not exist, because re-adding it is the one
    change that would silently restore default-on deletion.
  3. The newest complete archive is never a candidate, whatever the count says. That is
    a by-name guard and it is now documented as SEPARATE from the count floor rather
    than a consequence of it.
  4. Turning it on is the whole consent. Once a usable count is set, the sweep prunes to
    it without asking again.

Two details that were choices rather than mechanics:

  • Zero and negatives still CLAMP to the floor instead of reading as off. They are
    counts somebody wrote, and reading a typo as "off" would silently stop doing the
    thing the operator asked for. Only values that are not counts at all (absent, a
    string, a float, a bool) mean off.
  • The two roads to off are audited differently. Nothing configured is the ordinary
    state of an unconfigured install, so the sweep SUCCEEDED at having nothing to do.
    A state file this process could not read keeps everything too but stays failed,
    because an operator who did configure a count is silently not getting it, and
    filing that as success would bury it among every unconfigured install. A test
    asserts the two reasons stay distinguishable.

The count is never derived from nightly in either direction, and both directions
are pinned: the grant does not enable retention, and turning the nightly off does not
withdraw a configured count.

Mutation verified, two runs:

  • restore a deleting fallback in the clamp: 8 of 93 red. That is the exact regression
    the change exists to prevent, and it is caught by the absent-key reader, the
    symbol-absence pin, the three not-a-count pins, the nightly pin, the unconfigured
    sweep, and the audit outcome.
  • remove the by-name newest guard while leaving the count floor: 1 red, and it is the
    case where the archive just uploaded sorts old. That is the evidence clause 3 is a
    real independent guarantee rather than a restatement of the floor.

One sibling assertion moved as a consequence: the snapshot push now takes three
authorization gates rather than four, because an off sweep declines before asking for
anything and makes no listing call at all. It is pinned as the ABSENCE of a sweep
gate, so a fourth appearing there would mean the sweep had started doing cloud work
on an install that never asked for it.

364 tests green across the retention, backup, job and storage suites. Six gates green.
Spec and PR body both updated; the body no longer claims a default of 3.

Note for anyone reading an earlier board: the newest head is ce82f357b1. Verdicts
naming 48f22b7694 or 46477c8dd9 predate this change.

@chenmingwei23
chenmingwei23 force-pushed the feat/cloud-backup-retention branch from ce82f35 to 6cfd767 Compare September 20, 2026 08:58
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head 6cfd767

GPT's verdict on 46477c8dd9 carried two items. Both are real and both are fixed.
Note its adjudication stage FAILED TO RUN on that head, so the blocking verdict stood
unreviewed for proportionality; I treated it as a finding to answer on the merits
rather than as a reason to discount it.

BLOCKING: repaired or garbled account state triggering a destructive default

The chain it names is: malformed account state, the record write repairs it, retention
falls back to a default of 3, older versions are permanently erased.

That hazard no longer exists, and not because it was patched at the site GPT pointed
at. The previous head removed the deleting default entirely: there is no fallback count,
so a state file whose count is absent or garbled resolves to OFF and the sweep deletes
nothing. The prescribed fix was to preserve the pre-write unreadable state so this one
sweep skips deletion; removing the fallback is strictly stronger, because it also covers
every other route to an unusable count rather than the repair path alone.

The dangerous variant is worth separating out, because it is the one where the account
dict SURVIVES: if only the count is garbled, the upload records are intact, so every
older archive is genuinely owned and reachable and a fallback number would have erased
it. Ownership is not what makes that pile safe; off is.

Verified rather than argued. A new test drives the sweep end to end with five owned
archives and a garbled count, asserts all five are owned so the case could delete four
of them, and asserts nothing is deleted. Restoring the fallback in the clamp reddens it
with real deletions in the failure output, which is the finding reproduced and closed.

FINDING: a partial purge made "nothing was deleted" false

Correct, and the code already knew it: the comment two lines under that log call says a
failure after the listing may already have erased versions, which is why the audit entry
carries the count. The log line beside it still claimed nothing was deleted, so a reader
comparing the two was being pointed at a purge that did happen, about bytes that cannot
be recovered.

The line is now conditional on the erased count. Only the version count is named, never
a number of keys: batches are filled to the API's limit without regard to key boundaries,
so retired deliberately stays 0 on that path and a key figure is not something the
failure measured.

Both directions are pinned and mutation verified, so the fix cannot have replaced one
false message with another:

  • force the unconditional message back: the erased-count pin reddens, showing "nothing
    was deleted" returning while two versions were gone.
  • force the new branch always: the other pin reddens, because a listing failure erases
    nothing and saying so is correct.

Board

210 tests green across the retention and backup suites, 367 across all four. Six gates
green. Spec updated for both items.

Design and First Principles both last reviewed 48f22b7694, which predates the
default-off change, so their current bodies describe code that has moved. GPT's own
latest run on this PR reports it did not produce a completed verdict, which is the
absence of a review rather than a clean one, so no override was issued for it.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The Windows shard red is not from this PR

Backend Tests (Windows) (1) failed on an earlier head here. It is main-owned and
already tracked as #12247. Recording the attribution so nobody re-derives it from the
board.

The four failing tests are in test/test_acp_runtime.py, about runtime kill and reap
summaries. They assert [returncode=-15] and get [returncode=<not reaped>]. Minus
fifteen is a POSIX signal value; Windows cannot produce it.

Why it is not this PR's, in the order the checks discriminate:

  1. Zero diff overlap: this branch changes nothing under acp_runtime.
  2. git diff <merge-base> HEAD -- test/test_acp_runtime.py is EMPTY, so the branch
    never touched the file.
  3. Main ADDED those 258 lines after this branch's base, in fix(windows): reclaim runtime trees after root exit #12005 at 01:17 and fix(acp): say who killed a runtime and never log a bare returncode=None #12188
    at 01:24 today. They reach this PR only through the merge ref.
  4. The failing test names do not exist in this branch at all: zero occurrences here,
    two in main.

A rebase is not the remedy. It would import main's version of the test, which is the
broken one, so the red would follow rather than clear.

Note on reading the board: shard INDEX is not comparable across revisions, because
assignment is by file set. That other pull requests show Backend Tests (Windows) (1)
green is consistent with this, since a run triggered before 01:24 has a merge ref
without the new tests.

One hypothesis worth closing off, because it is the usual cause of a Windows-only
setup failure: a bare Path.write_text(body) uses the platform locale codec, cp1252 on
Windows, and dies on any non-ASCII byte. Every write_text in this PR's changed test
files already passes encoding="utf-8", so that is not what is happening here.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head c415926

First Principles moved from BLOCK to CONCERNS on 6cfd767dad: the product-shape
objection about default-on deletion is answered by the change rather than by an
argument. Its two remaining items are advisory and both are now fixed.

Item 10: two stale claims, deleted

The module docstring still said retention keeps "the newest retention_keep (default
3, never fewer than 1)". That default does not exist any more, so the sentence was
false. It now says retention is opt-in and keeps everything without a count.

The second half was worse than reported. When the default constant was removed, two
lines of ITS comment were left behind and adopted by RETENTION_KEEP_MIN, so the floor
carried a header about a different constant plus a dangling "Three rather than one"
clause. Both lines are gone.

Worth naming the gap that let this through: after removing the default I grepped for
the SYMBOL and found no readers, which is why the constant went cleanly. I did not grep
for the CLAIM, and "default 3" appears in prose that names no symbol at all.

Item 2: the switch now has a shipped writer

The finding was sharp and correct: a feature defaulted off with no way to turn it on
removes the named harm for nobody. POST /backup/{account}/retention now sets or clears
the count, registered beside the nightly toggle with the same guard and mutation
wrapper, and the status read reports retentionKeep.

Shape decisions, each with a reason:

  • It takes the COUNT, not a flag. There is no separate enable bit, so a count is on and
    null is off. null is also what turns it back off; a switch that could only be
    turned on would leave an operator editing state by hand to stop pruning, which is the
    problem this item raised in reverse.
  • It validates and never coerces. True IS an int in Python, so a coercing handler
    would store keep=1 -- the most destructive value available -- for a caller that
    believed it sent a flag. Refused at the route AND at the writer, because this key's
    whole safety rests on that screen.
  • Out of range is REFUSED, not clamped, so nobody configures 0 and is later told they
    configured 1. The sweep's clamp still covers state edited by hand, which the route
    does not police. The route is deliberately stricter than the reader.
  • A missing field is refused rather than read as off, so an omission cannot silently
    disable pruning.
  • A failed state write returns state_persist_failed and does not echo the value,
    because reporting a setting the next read contradicts is worse than an error.
  • The status read reports the EFFECTIVE count, not the raw stored value, so the panel
    cannot show a number the sweep would clamp or ignore.

Mutation verified on both validation sites:

  • neuter the route's type check: the True pin reddens with assert 200 == 400, which
    is the route accepting a flag and storing keep=1.
  • neuter the writer's: the direct-call pin reddens with DID NOT RAISE.

The writer is also pinned end to end rather than in isolation, since reachability was
the finding: off first (asserted), then set a count and the very next sweep prunes to
it; clear it and the next sweep keeps everything; clearing REMOVES the key rather than
storing a second spelling of off; and the nightly grant is untouched either way.

What is still missing is a control in the console UI. That is a front-end change and a
separate surface, and with the feature off by default its absence means nothing is
deleted rather than deletion nobody asked for. The PR body says so.

550 tests green across the retention, routes, backup, job and storage suites. Six gates
green. Spec updated for the route, the writer and the corrected claims.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head 2974c93

Opus raised one blocking finding on 6cfd767dad. It is real, it was mine, and it is
fixed. I verified both halves of its claim in code before accepting it rather than
taking the diagnosis on trust.

The finding

_read_state_checked caught FileNotFoundError and OSError around
read_text(encoding="utf-8"). A state file whose bytes are not valid UTF-8 raises
UnicodeDecodeError, which is a ValueError and not an OSError, so it escaped the
very function whose docstring promises ({}, False) for "exists but could not be read".

The consequence is worse than a wrong return value, and that is the part worth
confirming. _prune_remote_archives resolves its keep count as its FIRST statement,
outside its own best-effort handler, and the comment directly above its call site in
run_snapshot_backup promises retention "cannot fail the run". Both are true, so the
decode error propagated out of a backup that had already landed off-host and reported it
as failed, while also skipping the audited unreadable branch that exists for exactly
this case.

What changed

Two readers, two different branches, because their contracts differ:

  • _read_state_checked now catches (OSError, UnicodeDecodeError) and answers
    ({}, False). Non-UTF-8 bytes ARE "could not be read", and for the sweep that means
    keep everything and audit the failure.
  • The read inside _locked_state_update catches the same error into its PARSE branch,
    alongside JSONDecodeError, returning {} so the document is repaired on write. That
    is the documented split: a document that decoded to nothing usable carries nothing to
    lose, while a file the OS refuses is state nobody read and the mutation is abandoned.
    Left uncaught there it would also have escaped as a bare exception, bypassing the
    _StateUnreadable contract that function's callers are written against.

Neither catches the wider ValueError. A surprising one is a bug and should stay loud
rather than be folded into "unreadable"; a test pins that by raising a plain ValueError
from the read and asserting it still propagates.

Evidence

Five new pins, and the fixture plants genuinely invalid UTF-8 (a 0x80 continuation byte
with no lead byte) rather than merely invalid JSON, so the decode fails before any parse
is attempted.

Mutation verified separately per site, which is what shows the two branches are
independently load-bearing:

  • revert the checked reader to except OSError: 3 red, each with UnicodeDecodeError
    escaping. The sweep pin failing that way IS the reported defect reproduced.
  • revert the write path to except (FileNotFoundError, json.JSONDecodeError): 1 red,
    only its own pin.

446 tests green across the retention, routes, backup and job suites. Six gates green.
Spec updated for both branches.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 2974c93d4f6da531f161ab302342a7be727b73ce. Adjudication produced no [GPT-ADJUDICATED] marker for 2974c93: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/apps/builtins/aws_control/backend/backup.py:216 -- Invalid UTF-8 state is overwritten
except (FileNotFoundError, json.JSONDecodeError, UnicodeDecodeError):
non-UTF-8 state -> settings update treats it as empty -> existing account settings and backup records are replaced
Anchor: residual/crash-data-loss-corruption
Fix: Raise _StateUnreadable for UnicodeDecodeError instead of returning {}.
[BLOCK-MERGE] 2974c93
[GPT-REVIEWED] 2974c93
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2974c93d4f6da531f161ab302342a7be727b73ce: <one-sentence reason>

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head 4763170

GPT blocked 2974c93d4f over the write path's handling of non-UTF-8 state. The finding
is correct and is now fixed. It also contradicted the previous cycle's Opus finding on
the same line, so both were re-derived from the code rather than resolved by recency.

The finding

_read_state_for_update caught UnicodeDecodeError into its PARSE branch and returned
{}. Because _locked_state_update writes the WHOLE document back, an empty base there
does not mean "no fields to carry forward", it means "replace everything". So a state
file that stopped being valid UTF-8, plus any settings write, would have replaced every
account's nightly toggle, retention count and run history -- including the
upload_versions records the sweep's ownership test reads to decide what is safe to
delete. That is a worse loss than the settings: those records are what keep the sweep
from touching another install's archives.

Why GPT is right and the previous placement was wrong

The deciding line is in that function's own docstring, which predates this PR: repair-on
-write is justified because "a document that parsed to nothing usable carries nothing to
lose". A file that cannot be DECODED never reached the parser, so that justification does
not cover it. The docstring's other rule -- an unreadable file "is state we still have",
so the mutation is abandoned -- is the one that applies. The previous cycle's comment
asserted decode failures were the first kind; that assertion was the error.

Both findings now hold at once

They are different functions with different callers, which is why there is no conflict:

  • _read_state_checked, read by the sweep, still answers ({}, False). Opus's point
    stands: the sweep resolves its count outside its own best-effort handler and its call
    site promises retention cannot fail the run, so an escaping error there reported a
    backup already off-host as failed. Unchanged.
  • _read_state_for_update, read by set_nightly and set_retention_keep, now raises
    _StateUnreadable(EILSEQ). Aborting a settings write on state nobody read is what this
    function already did for EACCES.

Both fail closed, neither overwrites the file. That is more consistent than the split it
replaces, not a compromise between two reviewers.

_StateUnreadable is an OSError subclass, so set_nightly's existing handler needs no
change; a test pins that the decode case arrives as an OSError and not a bare
ValueError.

Evidence

Three pins replace the one that asserted the old behaviour. The data-loss pin reads the
file's BYTES before and after and asserts they are identical, rather than only asserting
the raise. A third pin writes text that decodes but does not parse and asserts it is
still repaired, so the change does not quietly widen into abandoning every corrupt file.

Mutation: putting UnicodeDecodeError back in the parse tuple reddens exactly 2 tests,
both with "DID NOT RAISE", and the decoded-but-unparseable pin stays green.

273 tests green across the three suites in scope. Black, flake8, isort, docs-lint and
comment-history green. Spec corrected.

One note on the adjudication stage

This comment records that adjudication FAILED TO RUN for 2974c93d4f, so the blocking
verdict was unreviewed for proportionality. It did not need review: the finding was
accurate on its own terms.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head 09e6ff1

First Principles raised one advisory item on 89ae81b8a4 (item 4, retentionKeep ships
with zero consumers). It is correct, it was a false statement in my own docstring, and it
is fixed.

Verified, not accepted on authority

Both halves checked here:

  • retention_keep's docstring did say "The read the console renders beside the nightly
    toggle."
  • grep -rn "retentionKeep\|/retention" website/src --include=*.ts --include=*.tsx
    returns zero hits.

So the docstring asserted a consumer that does not exist. That is worse than an
undocumented gap: it sends the next reader looking for a panel that is not there, and it
makes the shipped surface sound larger than it is.

What changed, and what did not

The claim is gone. The field stays, and the reason is in the docstring now rather than
implied: without a status read the route is write-only, so an operator who sets a count
has no way to confirm what was stored. Deleting the field would make the shipped route
less verifiable, not smaller.

The offered subtraction was "delete that docstring sentence (and the matching spec
wording), or defer the field". I took the first. On the spec: its wording for
retentionKeep was already accurate and claimed no renderer, so there was nothing false
to remove -- I added the absence explicitly instead, since a spec is where someone
deciding whether to build the panel will look.

Two pins added. One asserts the docstring does not claim a console renderer AND does
state the absence, so the sentence cannot quietly return. The other round-trips the count
through set and clear, which is the reason the field exists.

Scope note

No console control for the retention count remains a real, disclosed limitation of this
change. It is a frontend surface. It is stated in the PR body, in the module spec, and
now in the docstring of the function a renderer would call.

275 tests green in the three suites in scope. Black, flake8, isort, docs-lint and
comment-history green.

Also noted: this review independently reached the same reading of RFC O5 that the body
cites -- scope only, draft on base, and the item is not product shape because it ships
off and changes no default.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head b0fd1b0

GPT blocked 4763170e42 with a TOCTOU race at the deletion, and adjudication UPHELD it
after reading the code. Both are right. Fixed.

The race

keep is read once (backup.py:1550). The sweep then does a list_object_versions round
trip, then a re-authorization, then the irreversible delete_object_versions. No lock
spans that window: _run_lock is taken only at 254, 709, 1022 and 2820. So an owner who
switched retention off while the listing was in flight still got a permanent delete of
the versions they had just chosen to keep.

Verified here, not accepted on authority: the count read is at 1550, the delete is ~240
lines later, and every _run_lock acquisition is outside that span.

Why this one stings

The same call site ALREADY closes this window for consent, with a comment that states
the general rule: "a gate is good for the call that FOLLOWS it, not for a later one."
I wrote that, applied it to authorization, and never applied it to the count the whole
sweep turns on.

The fix

The count is re-read at the delete gate, beside the consent re-auth, and the pending set
must still be authorized:

  • unreadable -> refuse (same fail-closed rule as the first read)
  • count GREW -> refuse; it now protects keys the candidate set was built to delete, so
    the set is stale
  • count SHRANK -> proceed; it authorizes every key in the set and more, so the set is
    still valid. Refusing here would mean a sweep that never runs whenever the setting is
    touched at all.

An owner changing their mind audits successful; an unreadable setting audits failed.
That is the split the first read already files.

Not the suggested remedy, and why

The suggestion was to hold _run_lock through the deletion. I did not, because that lock
also serializes last_runs, so holding it across network calls would stall a status read
for the length of a purge. The re-read gets the property that matters -- the owner's
change is seen -- without putting network I/O inside a lock a status read waits on. A
test pins that with _run_lock does not appear in the sweep, so the rejected remedy
cannot arrive later by accident.

Evidence

Four pins. Three drive the race directly by flipping the stored count from inside the
patched lister, which is the actual interleaving rather than an approximation of it: off
mid-listing, raised mid-listing, lowered mid-listing.

Mutation (guard disabled): exactly 2 red, both asserting on real deletions that happened
after the owner withdrew the count -- the defect reproduced. The lowered-count test stays
green, which is what shows the guard does not over-refuse.

279 tests green in the three suites in scope. Black, flake8, isort, docs-lint and
comment-history green. Spec documents the re-read and the reason the lock was not used.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head 1ee04de

First Principles item 5 is correct and the ceiling's downward clamp is gone. Item 4 from
the previous round now reads as justified, so that one is settled.

The finding, verified

The claim was that RETENTION_KEEP_MAX's stated purpose is backwards and that the clamp
only adds a silent deletion. All three parts check out:

  • The comment did read "so a mistyped configuration cannot turn one nightly push into
    thousands of list and delete rounds on the owner's bill."
  • That cannot happen. A LARGER keep protects more and deletes FEWER. The listing does not
    take keep at all; its size is bounded separately by _VERSION_ROWS_MAX.
  • So the clamp's only effect was to turn a hand-edited keep=100 into 50 and delete
    archives 51 and older -- which the stored number explicitly named as keepers.

The sharpest part is that this module's own docstring already forbids it, two paragraphs
up from the clamp: "a fallback here would be a permanent delete performed on a value the
operator never wrote." Fifty is a value the operator never wrote.

What changed

_clamp_retention_keep no longer clamps down. Above the ceiling now reads as off.

The floor is untouched and that asymmetry is the point: clamping 0 UP to 1 moves
toward deleting FEWER archives than the stored number named, so a typo does not silently
stop retention. Clamping 100 DOWN moves the other way. Same operation, opposite safety,
and the old docstring claimed they were "for the same reason".

RETENTION_KEEP_MAX stays, because set_retention_keep validates against it and refuses
out-of-range. That is also what now makes the two paths agree: a count the supported way
in would reject is not one a hand-edited file gets acted on. A test pins the constant and
the refusal so removing the clamp cannot drift into removing the bound.

One thing I changed beyond the finding

Chasing this surfaced a reporting bug next to it. _retention_keep_for_sweep collapsed
"key absent" and "key present but unusable" into the same reason, "retention is not
enabled", which audits as a SUCCESSFUL sweep. So a string, a float, a bool, or an
over-range number kept everything and filed a clean audit -- an operator would never
learn the count they wrote was not in force.

Now only an ABSENT key is the ordinary off state. A key that is there and does not resolve
reports the anomaly reason and audits as failed. Nothing about what gets deleted changes
in any of those cases; this is only what the operator is told. Four existing pins moved to
the new expectation and each says why in a comment.

Evidence

Mutation, restoring the downward clamp: 2 red. The end-to-end one fails on a real
deletion list, not merely a reported number.

Getting there took a correction worth naming: my first end-to-end pin used keep=10000
with three archives, which cannot delete anything under a clamp to 50 either, so it was
vacuous. It now patches the ceiling down to 2 with four archives, and a companion test
asserts the same fixture DOES delete at a count of 2 -- so the empty deleted list is the
guard working rather than a fixture that could never delete.

283 tests green in the three suites in scope, 457 including routes. Black, flake8, isort,
docs-lint and comment-history green. Spec states the asymmetry and the reporting split.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head 99f966d

Three findings from the round on b0fd1b06b9, answered in ONE head rather than one push
each: GPT F1 (upheld, fenced), GPT F2 (downgraded), and Design's Watch.

F1 -- the race was still open, and the fix was mine

Correct, and it is a defect in my own previous fix. I had put the count re-read BEFORE
_authorize_upload, reasoning that a refusal should not pay for an STS call first. That
choice left the STS round trip sitting between the check and the irreversible delete, so
a POST keep:null landing inside it still lost versions the owner chose to keep. The
window went from "the listing" to "the STS call" and was never closed.

_delete_under_the_retention_gate now performs the re-read and the delete with one lock
held, and set_retention_keep takes the same lock. That leaves two possible orderings
and no third: either the write lands first and the re-read sees it, or the delete
completes and the write applies to the next sweep.

One deliberate difference from the suggested remedy. It asked for a per-account lock; this
is a single retention lock. A per-account map would be unbounded state keyed by a
caller-supplied string, which is what F2's own anchor family objects to, and the cost of
the coarser lock is only that two accounts' sweeps queue at their final step. It is also
NOT _run_lock, which serializes last_runs and would stall a status read for the length
of a purge; a test pins that the two are different objects.

Evidence: the writer's half is pinned by BEHAVIOUR, not by grepping for a lock name -- the
test holds the gate and asserts set_retention_keep cannot complete, then releases and
asserts it does. Removing with _RETENTION_GATE from the writer reddens exactly that
test. A one-sided lock is the failure mode that pin exists for.

F2 -- downgraded, and fixed anyway, because my comment was false

Adjudication downgraded this as disproportionate and that reasoning is sound. I fixed the
narrow part regardless, because the comment on _MAX_VERSION_ID_LEN claimed the bounds
cover "every unbounded-length field a row retains" and modified was retained without
one. The claim was false, which is worth fixing independently of the harm.

modified is now screened at the same line that already screens key and version, so this
adds a third field to an existing rule rather than the "permanent mechanism" the
adjudicator priced. It DROPS the row rather than emptying the field: an empty timestamp
sorts as oldest, which would make the row a likelier deletion candidate -- the wrong
direction for a value that arrived malformed. A second test asserts the bound leaves room
for a real ISO-8601 instant, so it cannot silently start dropping good rows.

Design's Watch -- filed as #12274

Verified and accurate, and it corrects something I wrote. upload_versions is trimmed
with uploads under MAX_REMEMBERED_UPLOADS = 200, and the sweep can only retire a key
whose version it recorded. With retention shipping off, an install pushing nightly drops
its oldest record at push 201 -- so the unclaimed class is not legacy-only as this PR
describes it. The bound keeps minting unclaimed archives on every install that has not
turned retention on.

Filed as #12274 rather than fixed here, which is the clearing condition Design offers. The
reason it belongs in its own change is in the issue: the two candidate directions are
"bound the version record independently" (which moves the cliff and needs a number argued
against a real horizon) and "let the sweep adopt an archive it has no record for" (which
weakens the one property the ownership design exists for). Neither should ride along with
a retention count.

Still outstanding from this item, and not silently dropped: the PR body and the module
spec both still describe the unclaimed class as legacy-only. That wording correction and
Design's status-read suggestion are queued for the next push rather than split into
another head now.

Verification

396 tests green across retention, backup, job and storage; 571 including routes. Black,
flake8, isort, docs-lint and comment-history green. One commit.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head dfbd12a

GPT's block on 1ee04de is already answered

That comment names backup.py:1812 and asks to "serialize the final retention check and
deletion against set_retention_keep". That is what 99f966d7ae shipped:
_delete_under_the_retention_gate performs the re-read and the delete under one lock,
and set_retention_keep takes the same lock. The marker is bound to a sha that predates
the fix. No new work; noted rather than skipped, because a stale blocking marker is easy
to mistake for a live one.

First Principles item 4 -- the ceiling is gone

This one quotes my own docstring back at me, correctly. Last round I removed the ceiling's
downward clamp and wrote that the constant "does NOT exist to bound cost". Having conceded
that, the constant's only remaining effects were a route refusal and a special case in the
reader -- so the finding asks what harm it removes, and the answer is none.

I tried to name one and could not. A count larger than the number of archives that exist
selects nothing for deletion; the listing's size never depended on it; the memory bound is
_VERSION_ROWS_MAX and is independent. The bound does not even catch the dangerous typo:
mistyping 50 as 5 deletes MORE, and no upper bound helps that.

So RETENTION_KEEP_MAX is deleted, along with the route's upper-bound refusal and the
reader's over-ceiling branch. A large count is now honoured as written.

That is strictly safer than either previous behaviour and resolves both findings in this
area at once: clamping 100 down to 50 deleted archives the stored number named, reading it
as off ignored a configured intent, and honouring it as 100 does neither.

RETENTION_KEEP_MIN stays, because its harm IS nameable: a stored 0 means "keep
nothing", and clamping it up deletes fewer archives than the number named.

Pins: a large count resolves to itself and reaches the sweep as itself; the same fixture
DOES delete under a small count, so the empty deletion list is the count working rather
than an inert fixture; not hasattr(backup, "RETENTION_KEEP_MAX") so the constant cannot
return quietly; the route accepts a large count and still refuses zero.

Design's Watch -- the wording I owed

The body and the module spec both described the unreclaimable class as archives written
before the version record existed. That was incomplete in the way Design set out: the
record is trimmed with the upload record under one bound, so an install pushing with no
count configured keeps producing unreclaimable archives once past it, and retention
shipping off makes that the ordinary path. Both now say so and both cite issue 12274,
which is where the choice between raising the bound and adopting unrecorded archives is
tracked.

Design's other suggestion -- carrying the last sweep's unclaimed figures into the status
read -- is not in this head. It is a real improvement and I would rather add it with its
own pins than tack it onto a head that is already changing validation behaviour.

Verification

571 tests green across retention, backup, job, routes and storage. Black, flake8, isort,
docs-lint and comment-history green. One commit. Body edited before the push, so its run
set is not the one this push replaces.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head 1adbc46

GPT's finding on 99f966d7ae is correct. set_retention_keep wrote the state TWICE: once
inside _RETENTION_GATE and once after it. The second call is gone.

How it got there, since that matters more than the line

When I added the gate two rounds ago I wrapped the existing write and did not remove the
original, so the function had both. I did try to check for exactly this: the command I ran
was grep -n "_locked_state_update(mutate)" chained with the test run. The chained command
was refused by the path resolver, I re-ran only the test half, and the grep never
executed. The tests passed, because with nothing racing them both writes store the same
value -- so the suite could not see it either.

Two things follow, and I have taken both:

  • A verification that was refused is not a verification that passed. The refusal and the
    green test arrived in the same breath and I read the pair as confirmation.
  • Asserting the final stored value cannot catch a duplicate write. The pin has to count
    the writes.

Why the duplicate was a real hazard, not just redundancy

The second write was outside the gate, so the ordering the gate establishes did not apply
to it. With a concurrent clear -- two overlapping requests on the config endpoint, which
is ordinary -- the ungated write could land after the clear and restore the count.
Retention would be back on without anyone turning it on, and the next push's sweep would
delete permanently. The adjudicator reached the same reading independently and upheld it
fenced.

Evidence

Two pins, and they COUNT the writes rather than checking the result: one for setting a
count, one for clearing it. Restoring the duplicate reddens both with assert 2 == 1.
Before the fix they would have been the only thing in the suite that noticed.

289 tests green across the three suites in scope; 127 in the retention suite. Black,
flake8, isort, docs-lint and comment-history green.

Board note

Design and First Principles are both PASS on this round, and PR Readiness has appeared
for the first time and is pending. The three older BLOCK-MERGE markers name superseded
shas.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for head 270be6e

Both findings on 1adbc46ff8 are correct. Before treating the red as a verdict I checked
the job's step list: every step succeeded including "Extract blocking findings for
adjudication", the adjudication itself, and "Post/update review comment", with only "Gate
on findings" failing. That is a published verdict rather than an incomplete review, so it
gets fixed, not re-run.

F1 -- my lock was the wrong KIND of lock

_RETENTION_GATE is a threading.Lock. It orders threads inside one process and nothing
else. Meanwhile set_retention_keep writes through _locked_state_update, which holds the
state file's sidecar lock -- a CROSS-process lock -- and this module's own docstrings
describe a second install finding the first one's bucket and writing into the same state
"BY DESIGN".

So the case the gate was built for had a hole exactly where the module says concurrency is
expected: a clear issued by another process was never ordered against the sweep's
read-check-delete, and the delete is irreversible. The adjudicator reached that by reading
those docstrings and upheld it fenced. I verified the same two facts before agreeing: the
gate is threading.Lock(), and _locked_state_update takes open_lock_file plus
file_lock on the state path.

The final check and the delete now hold BOTH: _RETENTION_GATE for threads here, and the
state file's own sidecar lock for other processes. Either alone leaves a real hole -- a
thread lock cannot see another process, and a file lock alone would not serialize two
threads here because each opens its own descriptor.

Two things I decided rather than copied from the suggested fix:

  • It is still not _run_lock, which also serializes last_runs; sharing it would stall a
    status read for the length of a purge. A test pins that they are different objects.
  • The cost is stated in the code rather than left for a reader to discover: a state write
    in ANY process now waits for the purge. That is the price of ordering an irreversible
    remote delete against a local withdrawal, and what waits is one batched delete.

Pinned by asserting the delete path opens the SAME lock path _locked_state_update uses,
because the property that matters is that they are one lock, not that two locks exist.

F2 -- stale claims, the third time this class has bitten me

Correct, and the diagnosis is embarrassing in a useful way. Removing the ceiling left three
sentences saying a count above it "reads as off", in backup.py and twice in the spec,
while the code now accepts and honours 10_000. All three are gone.

That is the third finding in this PR of the same shape: a comment or doc asserting a
behaviour the code does not have (the console renderer, the ceiling's cost justification,
and now this). In each case the code was right and my prose was stale. I am treating the
pattern as the signal rather than the individual lines.

Verification

290 tests green across the three suites in scope, 128 in the retention suite. No deadlock
from nesting a file lock inside the sweep, which the suite would have hung on. Black,
flake8, isort, docs-lint and comment-history green. One commit.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 270be6eddd88ff001d33b3fc5c3f206939df2c15. Adjudication produced no [GPT-ADJUDICATED] marker for 270be6e: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/apps/builtins/aws_control/backend/backup.py:1416 -- authorization can expire while waiting for retention locks
return storage.delete_object_versions(
Authorization succeeds -> purge waits on either lock while consent is revoked -> versions are permanently deleted without current authorization.
Anchor: backend-security-controls
Fix: Reauthorize inside both retention locks immediately before deletion.
[BLOCK-MERGE] 270be6e
[GPT-REVIEWED] 270be6e
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 270be6eddd88ff001d33b3fc5c3f206939df2c15: <one-sentence reason>

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head 50a35f1

Two lanes went red on 270be6eddd. They are not the same kind of red, and the step lists
are what separate them.

GPT: a published verdict, and it is right

The job's "Gate on findings" was SKIPPED rather than failed, because "Opus 4.8 adjudication"
failed first. But the comment itself is a completed GPT verdict and says so: it carries
[BLOCK-MERGE] 270be6eddd and states that adjudication FAILED TO RUN, so the verdict
stands UNREVIEWED for proportionality. So I weighed proportionality myself instead of
assuming it, and the finding holds.

The finding: authorization could expire while waiting for the retention locks. It is the
rule I had already applied to the count, not applied to consent. Acquiring the gate and
the file lock can wait on another purge, and a consent check taken before that wait is a
gate for a call on the far side of it. My own comment two lines away says a gate is good
for the call that FOLLOWS it.

Consent is now re-checked inside both locks, as the last thing before the irreversible
call. I did not add a third authorization: the one that already sat before the gate moved
inside it, so the sweep still makes exactly two -- one before the listing, one immediately
before the delete -- and the second is now where it counts.

The cost, stated in the code: an STS round trip now happens inside the critical section,
so other state writers wait slightly longer. That is the same trade already accepted for
the delete itself.

Pinned by asserting the gate is ALREADY held when authorization runs. The test also
asserts the FIRST authorization in the same run observes the gate UNHELD, which is what
makes it non-vacuous: both states occur in one run, so True is not what every call
returns.

Opus: an incomplete run, not a verdict

The Opus lane failed at "Opus 4.8 validation" -- a review step, not a gate -- and published
no verdict for this head. Per the same discriminator that made GPT's a real finding, this
one is a re-run rather than something to fix or override. Both failures are at Opus-backed
steps, which is also why GPT's adjudication did not run, so the likely cause is one backend
problem rather than two findings.

I am not requesting an override for either. GPT's is fixed; Opus's needs a re-run.

Verification

291 tests green across the three suites in scope, 129 in the retention suite. Black,
flake8, isort, docs-lint and comment-history green. One commit.

Measured rather than assumed on the way: moving the authorization one indent deeper risked
crossing the 100-column limit, so I checked every line in both changed files. The only line
over 100 is a pre-existing docstring line, and origin/main has exactly the same single
long line, so this change adds none.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for head f7a9b06

Fully adjudicated this time: every step of the GPT job succeeded except "Gate on findings",
and adjudication ran and UPHELD the finding. So it is a verdict, and it is correct.

The finding

_put_version_id retained parsed.get("VersionId") as a string with no length check. The
sibling READ path at the version listing already bounds the same field against
_MAX_VERSION_ID_LEN, and the comment on that constant claims every retained
variable-length field is bounded. The value reaches _record_run and is persisted in
backup.json, so it is retained in the sense that claim is about.

Two rounds ago I extended that same invariant to cover modified, and wrote that the
pairing "bounds every unbounded-length field a row retains". I closed it on the read path
and left it open on the write path -- the side that actually stores the value.

The fix

An over-long version id now reads as ABSENT rather than being truncated. The function's own
docstring already states what absent costs, and it is the safe direction: retention will not
retire a key it has no recorded version for. A truncated id would be worse than none,
because it names a different version while looking like proof of ownership.

Two pins: one that an over-long id comes back empty, one that an id exactly AT the bound is
kept. The second matters as much as the first -- a bound tight enough to reject real ids
would silently stop retention ever claiming an upload, which is a quiet failure rather than
a safe one.

The pattern, which is the real finding

This is the fourth item in this PR of one shape: a comment or doc asserting a property the
code does not have. Console renderer, the ceiling's cost justification, the above-ceiling
claims, and now this bound. In every case the code was right and my prose was ahead of it,
and this one is worse than the others because the prose was mine from two rounds ago and the
gap was in the half I had just edited.

What I take from it: when I write that an invariant holds across a set, the next thing to do
is enumerate the set rather than the instance in front of me. Here the set was "places a
version id is retained", and there were two.

Board note

The Opus lane, which failed its own validation step last round with no verdict, came back
clean on this head -- consistent with reading that failure as an incomplete run rather than a
finding.

Verification

291 tests green across the three suites in scope, 240 across storage and retention. Black,
flake8, isort, docs-lint and comment-history green. One commit.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Correction and fix for head f983b75

Both shard-2 failures were MINE, and I previously reported the Windows one as inherited.
Correcting that on the record.

What actually failed

test/test_aws_control_app.py::TestRouteGates::test_registrar_installs_exactly_the_p0_contract

It asserts the registrar installs EXACTLY a pinned set of routes. I added
POST /backup/{account}/retention and never added it to that set, so the assertion failed
with one extra item on the left -- my route. Reproduced locally: the set difference is
exactly ('POST', '/backup/{account}/retention') and nothing else.

The test is platform-independent, which is why it failed on Linux shard 2 AND Windows
shard 2. Both reds had one cause.

The attribution I got wrong

I reported Backend Tests (Windows) (2) as inherited and pointed at the missing ACP fix.
That was wrong. What the ACP fix (954fb69185) actually cleared was shard 1 -- the
[returncode=-15] kill-summary tests -- and shard 1 is green after the rebase. Shard 2 was
never the ACP failure; it was this route contract, from the moment I added the route.

The rebase was still the right move: it cleared a real inherited failure in shard 1. But it
also removed the noise that had been masking my own break.

Why my own testing missed it for sixteen cycles

My test scope was three files: retention, backup, backup_job. test_aws_control_app.py is
not among them, so nothing I ran ever exercised the route contract. The route landed in
cycle 31 and the pin has been red ever since, invisible to me.

Fixed, and the scope with it: I now run all seven test_aws_control_* suites, 781 tests,
which is the correct scope for a change that touches a route registrar rather than the
minimum I had been treating as complete.

I also enumerated the set rather than fixing the one file that failed. Exactly two test
files pin this surface -- test_aws_control_routes.py and test_aws_control_app.py -- and
both are green.

One thing the fix confirmed for free

MUTATIONS in that file derives from the pinned route list by filtering for POST, so adding
my route also subjected it to every mutation-contract assertion. Those pass, which is
independent confirmation that registering it beside nightly with the guard and mutation
wrappers was right.

Verification

781 tests green across all seven aws_control suites. Black, flake8, isort, docs-lint and
comment-history green. One commit; the changed-file count is now 10, up from 9, because the
pinned contract lives in a file this PR had not touched before.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on the current head: CI green, readiness passed, no security-sensitive path in the diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants