Repository navigation
fix(aws-control): restore the version this install recorded when a key is overwritten - #12565
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify #12644 stays tracked: a recoverable overwrite now restores silently, so a co-writer clobbering this install's keys is visible only in backend logs. [DESIGN-REVIEWED] e9399cb |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo findings. The sole candidate — a leak of the staged No additional grounded defect surfaced in the changed lines. [OPUS-REVIEWED] e9399cb Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Two spellings of one authorization now coexist — What this change shipsInventory (9 items) — 9 justifiedIntent: FIX — a restore refused the operator's own backup after a co-writer overwrote the key, though the archive sat on the drive as the recorded noncurrent version.
[FIRST-PRINCIPLES-REVIEWED] e9399cb |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
e5e7186 to
2f466cc
Compare
2f466cc to
d6e7de6
Compare
GPT 5.6 Review —
|
1 similar comment
GPT 5.6 Review —
|
d6e7de6 to
4d0dc07
Compare
GPT 5.6 Review —
|
1 similar comment
GPT 5.6 Review —
|
4d0dc07 to
712468f
Compare
712468f to
ac4e6c1
Compare
7a8fda6 to
b57be8a
Compare
GPT 5.6 Review —
|
b57be8a to
c6f08e4
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/apps/builtins/aws_control/backend/backup.py:3981 -- Corrupt consent state escapes recovery as a 500 |
…y is overwritten One backup drive is reachable by every install pointed at the account, and versioning is on for that reason. When a co-writer overwrites a key this install recorded, our archive stays on the drive as a noncurrent version. The restore read named no version, so it fetched the overwriting object, failed the recorded body fingerprint and refused -- with our archive present and unnameable. storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails the fingerprint backup._recover_recorded_version makes exactly one further read, of the VersionId uploaded_versions records for that key. It is accepted only on the same evidence the current-version path uses -- that fingerprint, re-taken over the bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only find bytes that already pass; it can never widen what a restore accepts. The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller has already said it will take whatever is current without proof, so there is no refusal to rescue, and reaching past the current object would hand that caller different bytes than it accepted, relabelled as a proven self archive. The reply carries recovered, because the alternative outcome for this case is a refusal, which is loud: an operator who silently receives a recovered archive would never learn a co-writer is overwriting their backup chain. Version ids are validated as data. They are opaque and drawn from the full base64 alphabet, so the shape is printable ASCII with no whitespace and a bounded length, applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security rule is that a leading '-' is refused, because get_file passes the id as a bare argv element and the AWS CLI's own option grammar would read one as another option. There is no shell in the path. apps/aws-control/data is neither agent-readable nor agent-writable -- it sits behind the agent file-tool floor and is bind-masked from every agent sandbox -- so this check stands between a stored value and an argv element, not between an agent and the CLI. A fingerprint match is the whole test, and nothing else is asked of the recovered bytes. Whether they still open as a tar.gz is not checked, because the current-version read does not check it either and the upload side pushes payloads it cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive can legitimately be malformed. Refusing one only on the recovery path would hand the operator their own file when nobody overwrote the key and a refusal when somebody did, for identical bytes, at the cost of decompressing a multi-gigabyte archive to get there. Retention is deliberately untouched. _current_version_is_ours stays a question about the current version, so a recoverable-but-noncurrent key still holds no keep slot -- which errs toward retaining more. Closes #12391 Refs #12381
c6f08e4 to
e9399cb
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Verified the load-bearing claims against main myself rather than reading the body's summary.
restore_downloadreaches the recovery only insideif origin == ORIGIN_SELF:(backup.py:83of that function), and the pre-transferorigin != ORIGIN_SELF and not foreign_okrefusal still stands ahead of it, so the new path cannot widen what an unproven archive reaches.recovered is Noneleavesorigin = ORIGIN_UNVERIFIEDuntouched._authorize_recovery_read's single-snapshot grant read is not a weakening.aws_consent.is_grantedis exactlyread_grantplus the profile/region comparison and nothing else, so the new gate asks the same local questions off one snapshot and then addsgrant.account != account, which the upload gate does not ask at all. Strictly tighter, and the STS-first ordering matches_authorize_upload._VERSION_ID_REwithfullmatchand no anchor is the right fix for the$-before-trailing-newline hole, and the leading--check is what actually closes the argv-option case._is_provable_version_idalready existed and is what keeps"null"out, so the syntax/identity split is real rather than a restatement.apps/aws-control/datais insandbox.py:267's hidden leaves, so "not agent-writable" holds; validating on the way out is still correct for a truncated read-back.
Approving. The four disclosed non-goals are all the conservative direction, and #12644 already carries the dashboard warning gap with the path a fix needs, so nothing here is left implicit.
…y is overwritten (kirodotdev#12565) One backup drive is reachable by every install pointed at the account, and versioning is on for that reason. When a co-writer overwrites a key this install recorded, our archive stays on the drive as a noncurrent version. The restore read named no version, so it fetched the overwriting object, failed the recorded body fingerprint and refused -- with our archive present and unnameable. storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails the fingerprint backup._recover_recorded_version makes exactly one further read, of the VersionId uploaded_versions records for that key. It is accepted only on the same evidence the current-version path uses -- that fingerprint, re-taken over the bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only find bytes that already pass; it can never widen what a restore accepts. The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller has already said it will take whatever is current without proof, so there is no refusal to rescue, and reaching past the current object would hand that caller different bytes than it accepted, relabelled as a proven self archive. The reply carries recovered, because the alternative outcome for this case is a refusal, which is loud: an operator who silently receives a recovered archive would never learn a co-writer is overwriting their backup chain. Version ids are validated as data. They are opaque and drawn from the full base64 alphabet, so the shape is printable ASCII with no whitespace and a bounded length, applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security rule is that a leading '-' is refused, because get_file passes the id as a bare argv element and the AWS CLI's own option grammar would read one as another option. There is no shell in the path. apps/aws-control/data is neither agent-readable nor agent-writable -- it sits behind the agent file-tool floor and is bind-masked from every agent sandbox -- so this check stands between a stored value and an argv element, not between an agent and the CLI. A fingerprint match is the whole test, and nothing else is asked of the recovered bytes. Whether they still open as a tar.gz is not checked, because the current-version read does not check it either and the upload side pushes payloads it cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive can legitimately be malformed. Refusing one only on the recovery path would hand the operator their own file when nobody overwrote the key and a refusal when somebody did, for identical bytes, at the cost of decompressing a multi-gigabyte archive to get there. Retention is deliberately untouched. _current_version_is_ours stays a question about the current version, so a recoverable-but-noncurrent key still holds no keep slot -- which errs toward retaining more. Closes kirodotdev#12391 Refs kirodotdev#12381
…y is overwritten (kirodotdev#12565) One backup drive is reachable by every install pointed at the account, and versioning is on for that reason. When a co-writer overwrites a key this install recorded, our archive stays on the drive as a noncurrent version. The restore read named no version, so it fetched the overwriting object, failed the recorded body fingerprint and refused -- with our archive present and unnameable. storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails the fingerprint backup._recover_recorded_version makes exactly one further read, of the VersionId uploaded_versions records for that key. It is accepted only on the same evidence the current-version path uses -- that fingerprint, re-taken over the bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only find bytes that already pass; it can never widen what a restore accepts. The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller has already said it will take whatever is current without proof, so there is no refusal to rescue, and reaching past the current object would hand that caller different bytes than it accepted, relabelled as a proven self archive. The reply carries recovered, because the alternative outcome for this case is a refusal, which is loud: an operator who silently receives a recovered archive would never learn a co-writer is overwriting their backup chain. Version ids are validated as data. They are opaque and drawn from the full base64 alphabet, so the shape is printable ASCII with no whitespace and a bounded length, applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security rule is that a leading '-' is refused, because get_file passes the id as a bare argv element and the AWS CLI's own option grammar would read one as another option. There is no shell in the path. apps/aws-control/data is neither agent-readable nor agent-writable -- it sits behind the agent file-tool floor and is bind-masked from every agent sandbox -- so this check stands between a stored value and an argv element, not between an agent and the CLI. A fingerprint match is the whole test, and nothing else is asked of the recovered bytes. Whether they still open as a tar.gz is not checked, because the current-version read does not check it either and the upload side pushes payloads it cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive can legitimately be malformed. Refusing one only on the recovery path would hand the operator their own file when nobody overwrote the key and a refusal when somebody did, for identical bytes, at the cost of decompressing a multi-gigabyte archive to get there. Retention is deliberately untouched. _current_version_is_ours stays a question about the current version, so a recoverable-but-noncurrent key still holds no keep slot -- which errs toward retaining more. Closes kirodotdev#12391 Refs kirodotdev#12381
What is the problem?
A backup drive can be shared by several installs pointed at the same account.
When another writer overwrites a key this install recorded, the archive this
install uploaded stays on the drive as a noncurrent version.
Restore only ever read the CURRENT version of the key. So it fetched the other
writer's bytes, failed them against the body fingerprint this install recorded,
and refused -- with this install's own archive sitting on the same drive, intact
and unreachable, because no code path could name a version. The recorded version
id was already persisted; nothing read it.
Why this issue matters to the user
The operator cannot get their own backup back. The refusal is correct about the
bytes it saw and wrong about what is available.
Their only escape makes it worse: passing the override hands them the OTHER
install's archive, labelled unverified, which is not the file they asked for.
How our fix solves it
Symptom: restoring an own key refuses. Immediately above it: the bytes handed to
the fingerprint came from the current object. Root cause: the read had no way to
name a version, so the recorded id was dead data.
storage.get_filetakes an optionalversion. Every existing caller isunchanged, and the parameter is the only new capability.
backup._recover_recorded_versionmakes ONE further read, pinned to theversion the record names, and only on the path that was about to refuse. It
accepts only when the same body fingerprint is re-taken over the bytes that
actually arrive.
delete marker, unreadable, mismatched, and a staged copy that cannot be hashed.
Reading the bytes back is part of fetching them, so the fingerprint read sits
inside the same guard as the transfer: an error there refuses and unlinks the
staged file instead of escaping the helper as a 500. The recovery can only find
bytes that pass; it can never accept bytes that failed.
_authorize_recovery_read: the same four questions_authorize_uploadasks ofthe paid upload, in the same order. The live caller identity must still name the
requested account, the app must still be enabled, S3 consent must still hold for
this profile and region, and the recorded grant must name THIS account. The
network round-trip runs first and the cheap local decisions last, so no window
sits between a check and the read it guards. The stored grant is read ONCE and
its profile, region and account all checked against that one snapshot: grant
reads are unlocked while writes take the consent lock, so checking profile and
region against one read and the account against a second would let a re-grant
landing between them satisfy each half from a different record, turning a refusal
into an allow. A profile repointed during the first download therefore cannot
reach AWS under a consent the owner never gave for this account.
A refusal means the recovery does not run, leaving exactly the refusal the caller
already had -- the two gates differ only in that the upload raises and this one
declines.
foreign_okthe recovery never runs. That override means "I accept thismay not be my machine's", so there is no refusal to rescue, and reaching past
the current object would hand back different bytes than the caller asked for,
labelled a proven self archive instead of the unverified one they accepted.
whitespace, printable ASCII, and a leading
-refused as its own rule. Thatlast one is the security rule: the id lands as a bare argv element, where the
AWS CLI reads a leading dash as another option and a planted
--profilewouldsilently repoint the call.
s3:GetObjectVersion, nots3:GetObject. S3 authorizesa version-pinned
GetObjectagainst a different action, andengine._checkedrenders the action name as its
AccessDeniedremediation hint -- so reportingthe unversioned one would tell a denied operator to add a permission they
already hold.
get_filereports whichever action matches the request it made.What tests we did
40 new tests, 23 in
test/test_aws_control_backup.pyand 17 intest/test_aws_control_storage.py.Scoped local runs with an explicit
-n0, all passing: backup 181, and 554 acrossstorage, retention, unchanged and routes together.
Every guard was mutation-verified -- the guard broken by hand, the reddened set
recorded, the marker reverted -- and the sets are disjoint, which is what
separates real assertions from a harness that would pass either way:
matchforeign_okgateGates clean on the rebased tree: black, flake8, isort, mypy, docs lint, and the
comment-history gate scoped with a base ref (without one it exits 0 vacuously).
Any other suggestions on the work
Four things are disclosed rather than folded in.
A malformed own archive is handed back. The recovery accepts a
fingerprint-matching archive that does not open, symmetric with the
current-version read, which has always done so. A readability check existed here
and was removed on review: bytes matching the recorded fingerprint ARE the bytes
this install uploaded, so provenance is settled and whether they open is a
usability question the product answers "hand it back" everywhere else. The check
also cost two full decompressions of a possibly multi-gigabyte archive to turn
the operator's own file back into the refusal this PR exists to remove. A test
pins the accepting behaviour, so the check cannot return quietly.
The dashboard's overwritten warning disappears for the recoverable case.
Both existing sentences are driven by the refusal this PR removes -- a
selfrow reaches the confirm strip only after the backend refuses, and the refusal
sentence renders on the error alone. Filed as #12644 with the whole path a fix
needs. A
recoveredreply flag was shipped here and removed again: with noreader it was decoration, and adding the reader means one new catalog key in
every locale, which is its own change rather than a rider on this one. The
backend still logs the event.
The
ORIGIN_UNVERIFIEDmessage wording is left alone. It says there is "norecord of uploading it", which is already inaccurate for the overwrite case on
base. The refusal had to be preserved exactly, so correcting the sentence
belongs to a change that owns it.
Retention still declines a now-recoverable key. Untouched on purpose: it
retains more and never less, and counting a recoverable-but-noncurrent copy
toward
keepis a decision about what may be DELETED.Pattern harvest
Rule candidate: semgrep
Pattern:
re.match()on a$-anchored pattern used as a validator.$alsomatches immediately before a trailing newline, so
"abc\n"passes a check meantto reject whitespace.
re.fullmatch()with no anchor is the correct shape, andthe defect is mechanically detectable wherever a validator pairs
.match(with a$-terminated pattern.Three more observations from this fix, generalizable but not lintable:
recoveredtravelledto the JSON boundary and stopped; the operator saw the same success card
either way. Ship the reader with the field or ship neither.
to be the ones this install uploaded, a second gate asking whether they are
any good is answering a question the rest of the product answers by handing
the file over.
tarfilemember iterator does not detect truncation. A header cutat a non-zero offset reads as a normal end of archive, so a valid gzip wrapping
an incomplete tar walks clean. This was measured while implementing a reviewer
remedy, and it is why that remedy was insufficient.
Closes #12391
Refs #12644