Repository navigation
feat(aws-control): retire old cloud backups by version after each push - #12159
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Description says keep clamps to [DESIGN-REVIEWED] 18523e2 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify one thing: retention reclaims nothing already on the drive — the measured 10.1 GB pile is kept permanently (pre-record archives have no version id; deferred to issue 12274). What this change shipsInventory (9 items) — 9 justifiedIntent: Stop each successful cloud-backup push from growing the operator's bucket forever, by retiring this install's oldest archives once an operator opts in with a keep count. ADDITION.
Checks run: RFC [FIRST-PRINCIPLES-REVIEWED] 18523e2 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsBoth candidates fail falsification: Candidate 1 ( Candidate 2 (spec paragraph inaccuracy): The technical observation is plausible — No new grounded finding survives the same bar. No findings. [OPUS-REVIEWED] 18523e2 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
1fccce8 to
bdd0454
Compare
bdd0454 to
e5883a0
Compare
e5883a0 to
2ece38b
Compare
|
Disposition of the First Principles advisory items (the Item 1 BLOCK is not addressed here and needs a maintainer). Item 3, zero consumers - ACTED ON. Verified independently before removing it: Item 2, the knob's only setter is a hand-edited file - KEPT, and the gap is real. Configurability with a default of 3 is a stated requirement of the task this PR implements, not something the diff reached for: a single retained copy means one corrupt upload tonight leaves no history at all, which is why the count is a value and not a constant. So the subtraction is declined on that ground rather than on a code argument. The finding is accurate about the consequence, and removing the status field above makes it more so: after this push the key genuinely has no reader or setter outside the sweep. That is a disclosed gap, not a solved one. Wiring a control is a separate change with its own surface to design, and folding a settings UI into a retention PR would be exactly the scope growth this review exists to catch. Item 6, the unreadable-state abort rides along - KEPT, conditional on Item 2 as the review says. The dependency is stated correctly: if the count were a constant there would be no state read to fail, and the machinery would have nothing to guard. It is retained because the count is retained. Its own justification is independent of that, though: reading an unreadable state file as the default silently converts a configured 50 into 3 and permanently erases the other 47, so on this path a default is not a safe fallback but a destructive one. GPT's two findings on the previous head are fixed, not overridden. Per-version ownership: presence in the upload record proved a KEY while retirement erases every version under it, so a second object version on a recorded key now leaves that key alone entirely - only a foreign write produces that state, and the record stores a body fingerprint per key rather than a version id, so nothing can say which version is ours. Mixed batches: Both are mutation-verified. The first mutation run exposed one of my own tests as vacuous - a foreign overwrite dated after the newer archive made its key sort newest, so it never reached the retirement list and the test passed with the guard removed. The fixture is corrected and the test now fails when the guard is loosened. |
UX Review (Fable 5) — ⏭️ skippedRevision |
eba4139 to
85a46ae
Compare
46477c8 to
ce82f35
Compare
Retention now ships OFF (head ce82f35)This head changes the shipped default, which answers the outstanding product-shape Four parts:
Two details that were choices rather than mechanics:
The count is never derived from Mutation verified, two runs:
One sibling assertion moved as a consequence: the snapshot push now takes three 364 tests green across the retention, backup, job and storage suites. Six gates green. Note for anyone reading an earlier board: the newest head is |
ce82f35 to
6cfd767
Compare
Dispositions for head 6cfd767GPT's verdict on BLOCKING: repaired or garbled account state triggering a destructive defaultThe chain it names is: malformed account state, the record write repairs it, retention That hazard no longer exists, and not because it was patched at the site GPT pointed The dangerous variant is worth separating out, because it is the one where the account Verified rather than argued. A new test drives the sweep end to end with five owned FINDING: a partial purge made "nothing was deleted" falseCorrect, and the code already knew it: the comment two lines under that log call says a The line is now conditional on the erased count. Only the version count is named, never Both directions are pinned and mutation verified, so the fix cannot have replaced one
Board210 tests green across the retention and backup suites, 367 across all four. Six gates Design and First Principles both last reviewed |
The Windows shard red is not from this PR
The four failing tests are in Why it is not this PR's, in the order the checks discriminate:
A rebase is not the remedy. It would import main's version of the test, which is the Note on reading the board: shard INDEX is not comparable across revisions, because One hypothesis worth closing off, because it is the usual cause of a Windows-only |
Dispositions for head c415926First Principles moved from BLOCK to CONCERNS on Item 10: two stale claims, deletedThe module docstring still said retention keeps "the newest The second half was worse than reported. When the default constant was removed, two Worth naming the gap that let this through: after removing the default I grepped for Item 2: the switch now has a shipped writerThe finding was sharp and correct: a feature defaulted off with no way to turn it on Shape decisions, each with a reason:
Mutation verified on both validation sites:
The writer is also pinned end to end rather than in isolation, since reachability was What is still missing is a control in the console UI. That is a front-end change and a 550 tests green across the retention, routes, backup, job and storage suites. Six gates |
Dispositions for head 2974c93Opus raised one blocking finding on The finding
The consequence is worse than a wrong return value, and that is the part worth What changedTwo readers, two different branches, because their contracts differ:
Neither catches the wider EvidenceFive new pins, and the fixture plants genuinely invalid UTF-8 (a 0x80 continuation byte Mutation verified separately per site, which is what shows the two branches are
446 tests green across the retention, routes, backup and job suites. Six gates green. |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/apps/builtins/aws_control/backend/backup.py:216 -- Invalid UTF-8 state is overwritten |
Dispositions for head 4763170GPT blocked The finding
Why GPT is right and the previous placement was wrongThe deciding line is in that function's own docstring, which predates this PR: repair-on Both findings now hold at onceThey are different functions with different callers, which is why there is no conflict:
Both fail closed, neither overwrites the file. That is more consistent than the split it
EvidenceThree pins replace the one that asserted the old behaviour. The data-loss pin reads the Mutation: putting 273 tests green across the three suites in scope. Black, flake8, isort, docs-lint and One note on the adjudication stageThis comment records that adjudication FAILED TO RUN for |
Disposition for head 09e6ff1First Principles raised one advisory item on Verified, not accepted on authorityBoth halves checked here:
So the docstring asserted a consumer that does not exist. That is worse than an What changed, and what did notThe claim is gone. The field stays, and the reason is in the docstring now rather than The offered subtraction was "delete that docstring sentence (and the matching spec Two pins added. One asserts the docstring does not claim a console renderer AND does Scope noteNo console control for the retention count remains a real, disclosed limitation of this 275 tests green in the three suites in scope. Black, flake8, isort, docs-lint and Also noted: this review independently reached the same reading of RFC O5 that the body |
Disposition for head b0fd1b0GPT blocked The race
Verified here, not accepted on authority: the count read is at 1550, the delete is ~240 Why this one stingsThe same call site ALREADY closes this window for consent, with a comment that states The fixThe count is re-read at the delete gate, beside the consent re-auth, and the pending set
An owner changing their mind audits Not the suggested remedy, and whyThe suggestion was to hold EvidenceFour pins. Three drive the race directly by flipping the stored count from inside the Mutation (guard disabled): exactly 2 red, both asserting on real deletions that happened 279 tests green in the three suites in scope. Black, flake8, isort, docs-lint and |
Disposition for head 1ee04deFirst Principles item 5 is correct and the ceiling's downward clamp is gone. Item 4 from The finding, verifiedThe claim was that
The sharpest part is that this module's own docstring already forbids it, two paragraphs What changed
The floor is untouched and that asymmetry is the point: clamping
One thing I changed beyond the findingChasing this surfaced a reporting bug next to it. Now only an ABSENT key is the ordinary off state. A key that is there and does not resolve EvidenceMutation, restoring the downward clamp: 2 red. The end-to-end one fails on a real Getting there took a correction worth naming: my first end-to-end pin used 283 tests green in the three suites in scope, 457 including routes. Black, flake8, isort, |
Dispositions for head 99f966dThree findings from the round on F1 -- the race was still open, and the fix was mineCorrect, and it is a defect in my own previous fix. I had put the count re-read BEFORE
One deliberate difference from the suggested remedy. It asked for a per-account lock; this Evidence: the writer's half is pinned by BEHAVIOUR, not by grepping for a lock name -- the F2 -- downgraded, and fixed anyway, because my comment was falseAdjudication downgraded this as disproportionate and that reasoning is sound. I fixed the
Design's Watch -- filed as #12274Verified and accurate, and it corrects something I wrote. Filed as #12274 rather than fixed here, which is the clearing condition Design offers. The Still outstanding from this item, and not silently dropped: the PR body and the module Verification396 tests green across retention, backup, job and storage; 571 including routes. Black, |
Dispositions for head dfbd12aGPT's block on 1ee04de is already answeredThat comment names First Principles item 4 -- the ceiling is goneThis one quotes my own docstring back at me, correctly. Last round I removed the ceiling's I tried to name one and could not. A count larger than the number of archives that exist So That is strictly safer than either previous behaviour and resolves both findings in this
Pins: a large count resolves to itself and reaches the sweep as itself; the same fixture Design's Watch -- the wording I owedThe body and the module spec both described the unreclaimable class as archives written Design's other suggestion -- carrying the last sweep's unclaimed figures into the status Verification571 tests green across retention, backup, job, routes and storage. Black, flake8, isort, |
Disposition for head 1adbc46GPT's finding on How it got there, since that matters more than the lineWhen I added the gate two rounds ago I wrapped the existing write and did not remove the Two things follow, and I have taken both:
Why the duplicate was a real hazard, not just redundancyThe second write was outside the gate, so the ordering the gate establishes did not apply EvidenceTwo pins, and they COUNT the writes rather than checking the result: one for setting a 289 tests green across the three suites in scope; 127 in the retention suite. Black, Board noteDesign and First Principles are both PASS on this round, and |
Dispositions for head 270be6eBoth findings on F1 -- my lock was the wrong KIND of lock
So the case the gate was built for had a hole exactly where the module says concurrency is The final check and the delete now hold BOTH: Two things I decided rather than copied from the suggested fix:
Pinned by asserting the delete path opens the SAME lock path F2 -- stale claims, the third time this class has bitten meCorrect, and the diagnosis is embarrassing in a useful way. Removing the ceiling left three That is the third finding in this PR of the same shape: a comment or doc asserting a Verification290 tests green across the three suites in scope, 128 in the retention suite. No deadlock |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/apps/builtins/aws_control/backend/backup.py:1416 -- authorization can expire while waiting for retention locks |
Disposition for head 50a35f1Two lanes went red on GPT: a published verdict, and it is rightThe job's "Gate on findings" was SKIPPED rather than failed, because "Opus 4.8 adjudication" The finding: authorization could expire while waiting for the retention locks. It is the Consent is now re-checked inside both locks, as the last thing before the irreversible The cost, stated in the code: an STS round trip now happens inside the critical section, Pinned by asserting the gate is ALREADY held when authorization runs. The test also Opus: an incomplete run, not a verdictThe Opus lane failed at "Opus 4.8 validation" -- a review step, not a gate -- and published I am not requesting an override for either. GPT's is fixed; Opus's needs a re-run. Verification291 tests green across the three suites in scope, 129 in the retention suite. Black, Measured rather than assumed on the way: moving the authorization one indent deeper risked |
Disposition for head f7a9b06Fully adjudicated this time: every step of the GPT job succeeded except "Gate on findings", The finding
Two rounds ago I extended that same invariant to cover The fixAn over-long version id now reads as ABSENT rather than being truncated. The function's own Two pins: one that an over-long id comes back empty, one that an id exactly AT the bound is The pattern, which is the real findingThis is the fourth item in this PR of one shape: a comment or doc asserting a property the What I take from it: when I write that an invariant holds across a set, the next thing to do Board noteThe Opus lane, which failed its own validation step last round with no verdict, came back Verification291 tests green across the three suites in scope, 240 across storage and retention. Black, |
Correction and fix for head f983b75Both shard-2 failures were MINE, and I previously reported the Windows one as inherited. What actually failed
It asserts the registrar installs EXACTLY a pinned set of routes. I added The test is platform-independent, which is why it failed on Linux shard 2 AND Windows The attribution I got wrongI reported The rebase was still the right move: it cleared a real inherited failure in shard 1. But it Why my own testing missed it for sixteen cyclesMy test scope was three files: retention, backup, backup_job. Fixed, and the scope with it: I now run all seven I also enumerated the set rather than fixing the one file that failed. Exactly two test One thing the fix confirmed for free
Verification781 tests green across all seven aws_control suites. Black, flake8, isort, docs-lint and |
iamwhatever
left a comment
There was a problem hiding this comment.
Approved on the current head: CI green, readiness passed, no security-sensitive path in the diff.
1. What is the problem?
Cloud backup uploads an archive and never deletes an old one. Both push paths mint
a key carrying a timestamp (
sessions-<stamp>.tar.gz,kirocrew-snapshot-<stamp>.tar.gz),so nothing is ever overwritten: every nightly run adds one object and removes none.
The word
deletedid not appear anywhere inbackup.py.Measured on a real drive: 15 snapshot archives, 10.1 GB, the newest 2.49 GB, the
oldest three weeks old. The bucket's size had never once gone down.
The local side already bounds itself.
snapshot_main([tmp, "--keep", "1"])prunesafter each local snapshot. The cloud side had no counterpart.
2. Why this issue matters to the user
It is the user's own AWS bill, growing at roughly one archive a night forever, for
history nobody asked to keep. Nothing in the UI says it is happening and nothing
stops it: the drive is the user's bucket, so the cost lands on them and the only
remedy available today is deleting objects by hand in the console.
3. How our fix solves it
The symptom is a bucket that only grows. Walking it back:
newest;
objects.
That last step is the one worth reading. The drive has versioning enabled and
no lifecycle rule. A
delete-objectwithout a version id writes a delete markerand turns the bytes into a noncurrent version that goes on being billed, so a
retention pass built that way empties the listing and saves nothing. A lifecycle
noncurrent-expiry rule does not solve it either: the 15-object pile is 15 current
versions under distinct keys, and a time-based rule cannot express "keep the newest
three". So the sweep is version-aware end to end. Two new storage primitives,
list_object_versions(complete-or-raised, folder-scoped) anddelete_object_versions(version-pinned, with no prefix argument that could widenthe blast radius), erase the bytes and leave no marker behind.
storage.pyalreadynamed this purge as missing in its own docstring; retention is its one caller.
backup._prune_remote_archivesruns as the LAST step of a successful push:It ships OFF, and there is no count that applies without being configured. A
fresh install and an upgraded one both keep every archive, exactly as before, so no
first run after an upgrade deletes anything. Retention turns on only when this
account's state holds a usable count under
retention_keep; that key IS theswitch. Deleting an object version cannot be undone, and these are the operator's
bytes in the operator's bucket, so the two ways of being wrong do not compare:
shipping off costs storage they can see in a listing and reclaim with one command,
while shipping on silently destroys archives somebody was deliberately keeping and
leaves them nothing to restore from. That asymmetry is not this PR's invention: the
repository already recorded it.
docs/request-for-change/rfc-s3-backup.mdO5("retention, partly answered", lines 370-383 on main) states that a lifecycle rule
"can express
delete older than N daysbut neverkeep the newest N", that on a hostwhich stopped backing up such a rule "would delete the last surviving copy of its
memory exactly when it is needed", that "unbounded growth is the cheaper failure", and
that what remains open is "whether operators want an opt-in, count-based remote prune,
which needs a lister and a deleter rather than a lifecycle rule". Opt-in, count-based,
a lister and a deleter: that is this change, in the RFC's own words, including the
reason the newest archive is protected by name. What the RFC leaves open is whether
operators WANT it, and shipping off is precisely what leaves that open -- an install
that sets no count is in exactly the state the RFC describes today. It is also the
house shape rather than a new
policy --
nightlyships off, reads fail-closed, and is never inferred from anadjacent grant, and a retention default that deleted would be the first capability
here to act destructively on operator data unasked. The switch costs one key read,
so there is no engineering case for omitting it. This does not ignore the request
that started the work: "keep only the most recent one" is someone choosing a value
for their own install, which default-off plus a count delivers exactly -- they set
1-- while leaving every other install's bytes where they are. A default wouldhave generalised one operator's choice into everyone's deletion policy.
On the RFC's
status: draft(line 3): conceded, and it does not change the above. Thecitation is used for SCOPE, not as an approval -- the shape it predicts is the shape
shipped, and a draft recording that framing is still the repository's framing rather
than a reviewer's or an author's. Note the "Blocks neither milestone" clause elsewhere
in that file belongs to O1, which is about the bundle's secret policy, so it is not
quoted here as covering O5.
Every unusable stored value reads as off, not as a fallback number. Absent, a
string, a float, a bool: none is a count somebody chose, and a fallback would be a
permanent delete performed on a value the operator never wrote.
boolis screenedbefore
intbecauseTrueIS anintin Python and would otherwise read askeep=1, the most destructive value available. Zero and negatives DO clamp to[1, 50]rather than switching retention off, because they are counts somebodywrote and reading a typo as "off" would silently stop doing what was asked for.
Both roads to off keep everything; only their audit differs. Nothing configured
is the ordinary state of an unconfigured install, so the sweep succeeded at having
nothing to do. A state file this process could not read keeps everything too but is
audited
failed, because an operator who DID configure a count is silently notgetting it.
keepis reported as"off"in both, never as a number the sweep didnot act on. Neither is ever derived from
nightlyin either direction: authorizingunattended uploads is not authorizing permanent deletes, and turning the nightly
off does not withdraw a configured count.
The newest is never retired, and that is separate from the count. The key this
run just uploaded is dropped from the candidates by name, whatever the count says,
including
1. The floor at1also spares the first entry of the age order, butthat is a different claim: the run's own key is only first while nothing else
carries a later timestamp, and a co-writer with a skewed clock is enough to move
it -- which is exactly the case the by-name guard exists for. Removing that guard
while keeping the floor reddens a test on its own. And if the listing does not show
that key, nothing is pruned at all: a view missing the newest object cannot be trusted
about which objects are old. This is the cloud form of the guard
--keepalreadycarries locally, where an incomplete bundle does not prune.
Per kind and per install.
snapshots/andsessions/are separate historieswith separate cadences, so counted together a burst of one would evict the other's
only copy. One drive is reachable by several installs by design, so the listing and
every delete are anchored on this install's own prefix.
Best-effort and last. The upload is the point of the run. Every failure is
caught and logged as one line, so a backup whose archive is safely off-host is
never reported as failed over a cleanup that was not.
Audited either way. This is the only path in the app that erases object
versions permanently, and being best-effort is exactly why the audit entry
matters: a failure that only logs is one nobody reviewing the trail can see. Each
terminal outcome files one
SEL_OP_RETENTIONevent carrying its counts, and thegate's own refusal is filed once, where the decision is made, rather than twice.
Two things it deliberately leaves alone: the label sidecar (not an archive, so it
holds no keep slot and is never deleted) and a key whose current version is a
delete marker (its noncurrent bytes belong to the manual delete path, whose
documented recoverability this must not silently revoke).
The sweep runs
_authorize_uploadTWICE, under its own audit operation: once beforethe listing, and again immediately before the delete. These are deletes of the
owner's data running after a build that may have taken minutes, so consent,
app-enabled state and the account behind the profile are re-checked rather than
inherited from before the push. The second gate exists because a listing round trip
separates the two, and consent withdrawn in that window must not reach an
irreversible call. A refused sweep
is not filed as a denied upload, because this run's upload succeeded.
4. What tests we did
test_aws_control_backup_retention.pyis new: 74 tests, no network, no profile, nobucket. Two fake layers, and which one a test uses is that test's point. The
retention tests fake the storage calls and assert what retention DECIDES (keep the
newest under every bad configured value, per-kind and per-install scoping, the
sidecar and delete-marker exclusions, the refusal when the listing omits the
upload, one SEL event per terminal outcome, and the best-effort contract). The
storage tests fake
_checked, the package's single AWS CLI chokepoint, and assertthe argv that would have gone to the CLI, so "every delete carries a VersionId" is
pinned on the wire rather than inferred.
212 tests pass across the three affected suites with
-n0.flake8,isort, thebaselined
blackgate anddocs-lintare clean;mypy's 4 errors are intranscribe.pyandops_mission_control, untouched here.One finding came out of running the suites rather than reading them. Adding the
sweep made four existing push-path tests attempt a live
aws s3api list-object-versions, and because the sweep swallowsExceptionby design, threeof them still PASSED while doing it. It was found by re-running under a throwaway
plugin that raised a
BaseExceptionthe sweep cannot catch, then fixed in thethree autouse fixtures so future push-path tests are covered too, and re-verified:
zero live version-listing calls remain.
5. Any other suggestions on the work
drive going forward, from the next successful push of each kind. Ownership is
proven by the
VersionIdrecorded at upload, and an archive pushed before thatrecord existed has no id in it, so the 10.1 GB already sitting in that bucket is
kept permanently. That is not only history: the version record is trimmed with the
upload record under one bound, so an install that keeps pushing with no count
configured drops its oldest recorded version once it passes that bound, and those
archives stop being retirable too. Retention ships off, which makes this the ordinary
path rather than an edge case. Tracked as issue 12274, because raising the bound only
moves the cliff and letting the sweep adopt an archive it has no record for would
weaken the one property the ownership test exists for. Draining it needs either a one-shot reclaim or a migration that
adopts those versions by matching each against the body fingerprint
uploadsalready stores. Both are separate changes, and this PR does not reduce the
measured pile that motivated it.
POST /backup/{account}/retentionsets or clears the count, beside the nightlytoggle and guarded the same way, and the status read reports the effective count as
retentionKeep. It takes the count rather than a flag because a count is on andnullis off, andnullis what turns it back off -- a switch that could only beturned on would leave an operator editing state by hand to stop it. What remains
missing is a control in the console UI, which is a front-end change and a separate
surface; with the feature off by default, its absence means "nothing is deleted"
rather than deletion nobody asked for.
delete_keywrites a marker on purpose, for recoverability. Retention does not erase those
versions, so that cost predates this change and remains; a lifecycle
noncurrent-expiry rule on the drive would be the natural home for it.