Skip to content

feat(aws-control): stop the nightly re-uploading a tree that has not changed - #12381

Merged
iamwhatever merged 1 commit into
mainfrom
fix/backup-unchanged-check-12239
Sep 21, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/backup-unchanged-check-12239

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

The aws-control nightly built a complete archive and pushed it under a new
timestamped key every run, with nothing comparing it against what the drive already
held. On one real install that produced 15 snapshot archives totalling 10.1 GB, one
per night, growing from 232 MB to 2.5 GB each. The growth was not the operator adding
data at that rate; it was the same tree being packed and sent again.

Retention (#12159, merged) bounds how many copies are KEPT. It does not stop a copy
being sent, so the transfer and the storage write happened every night regardless.

Two things that look like the fix and are not:

  • Overwriting one key does not reclaim the bytes. The drive bucket is versioned,
    so writing the same key twice retains the previous version and keeps billing for it
    behind a listing that no longer shows it.
  • Comparing archive bytes cannot work. A tar.gz embeds per-entry mtimes and a
    gzip header stamp, so two runs over a byte-identical tree produce different bytes.
    Measured in this branch: identical input, differing archives, every time. An
    archive-level check would report "changed" every night and the skip could never
    fire.

Why it matters

The cost is a ratchet rather than a level. Today's figure is small in dollars and it
has no ceiling. Carrying transcripts in the nightly makes it sharply worse: the two
session trees measure 22 GB across 28,947 files on the same host, and a full
re-upload of that per night is a different order of cost from the memory snapshot.

What changed

A run now computes what its archive CARRIES and sends nothing when the drive already
holds it.

The comparison is over the entry set, not the artifact. _tree_fingerprint
digests one row per archive member - path, kind, size and a SHA-256 of its bytes - in
sorted path order. That is the per-entry source manifest the decision needs, read
from the packed payload. Reading it from the packed copy rather than walking the
source a second time is deliberate and is the stronger of the two options:

  • It cannot drift. A second walk would have to re-derive which paths each kind
    packs, knowledge that lives in snapshot.COMPONENTS and in _add_tree. The day the
    two disagreed, the check would answer "unchanged" for a tree whose content had
    moved. A backup that silently stops backing up is a worse failure than a local
    rebuild.
  • It sees redaction. The snapshot path may upload a redacted copy, so flipping that
    switch changes the bytes that leave while the source tree is untouched. Taken over
    the payload this notices; taken over the source it would not.

Two normalizations, both measured rather than assumed. Building two bundles from
one untouched home and diffing every member shows exactly two things move: the
bundle's root directory is named kirocrew-snapshot-<stamp>, and inside
MANIFEST.json the single field created_at. So the root segment is stripped for the
snapshot payload (volatile_root=True; the sessions archive passes False because
its crew and cli roots are meaningful), and the manifest is compared with
created_at dropped. Only that one field: purpose, staging (pinned vs unpinned)
and version are not derivable from the file set at all, so dropping the whole
member - the obvious shortcut - would silently stop noticing a bundle that switched to
an unpinned staging walk. mtime is excluded on purpose; a content change already
moves the content hash, while a restore or a checkout bumps mtime without changing a
byte and would spend the very upload this removes.

The skip requires the previous archive to be PROVEN present, at its recorded
length.
A record proves only that this install once wrote that key. Retention
deletes by design, and this drive is reachable by several installs by design - the
same reason _body_fingerprint exists - so a co-writer can overwrite a name.
_unchanged_baseline therefore HEADs the recorded key and compares ContentLength
against the recorded size. Every uncertain branch uploads: no baseline, no recorded
fingerprint (an install upgrading into this feature has none, and unknown is not a
pass), a moved tree, no recorded key, a missing object, a length mismatch, a HEAD that
could not be answered at all, or an archive this cannot read.

A skip records itself honestly and does not strand the loop. The record carries
uploaded: false plus the matched run's key, body fingerprint and version, so the
baseline survives for tomorrow's comparison and retention keeps the version it needs.
It takes a FRESH stamp because due_for_nightly reads at: keeping the old stamp
would read as due on the very next wake and rebuild the archive every few minutes,
turning the saving into a busy loop. Neither the label publish nor the retention sweep
runs on a skip; both exist to follow a push, and running retention here would let a
stretch of unchanged nights walk the keep window down and delete the archive the next
skip has to prove is present.

The module docstring's "the per-session incremental integration with the storage
inventory is future work" sentence is replaced by a section stating what the code now
does, and saying that per-session incremental transfer remains out of scope - the RFC
lists incremental and deduplicating transfer among its non-goals.

Backwards compatibility

Additive. Run records gain tree and uploaded; a record written before this change
carries no tree, which never matches, so an existing install uploads once and gets a
baseline. The uploads and version maps are written only on a real upload, so retention
reads exactly what it read before. An archive that cannot be read as a tar.gz yields
no fingerprint and is pushed exactly as it was pushed before, so this adds no new
refusal path: validating the payload is a separate question from deciding whether to
send it. One extra head-object per run is the only added cloud call.

Tests

test/test_aws_control_backup_unchanged.py, 35 cases in three layers:

  • The mechanism, on hand-built archives. Includes the acceptance property - the
    same tree packed twice yields differing bytes and an identical fingerprint - plus a
    falsifying case for every exclusion, so each normalization is shown to ignore
    exactly what it claims and nothing more. created_at is ignored while staging is
    not; the bundle root is ignored while content beneath it is not; a MANIFEST.json
    deeper in the tree is ordinary content whose every byte counts.
  • The assumption about snapshot.py, which this module does not own. Two real
    bundles are built from one unchanged tree and their fingerprints must match. The day
    a second volatile field appears, this goes red instead of the skip quietly never
    firing again.
  • The decision, in both directions. Every refusing branch of
    _unchanged_baseline has a case, alongside the one accepting case that stops them
    all passing on a function that returns None; and each end-to-end skip test has a
    sibling proving the same setup uploads when something really moved.

Mutation-verified, 15/15 killed, with non-uniform blast radii (1 to 10 failures)
rather than one number repeated - which is what distinguishes per-behaviour
assertions from a harness reporting the same breakage each time. Every kill was read
rather than counted: each is an AssertionError about a value, not an import or type
error. Disabling either skip, making the baseline unconditional, dropping the
presence proof, the length proof, the created_at exclusion, the root
normalization, the fresh stamp, the non-file members, or recording a skip as an
upload each reddens its own tests.

Manual verification

Measured against the real snapshot engine on an isolated KIROCREW_HOME:

  • Two builds of one untouched tree: archive bytes differ, normalized member name sets
    are equal, and exactly one member differs in content (MANIFEST.json) in exactly
    one field (created_at).
  • Member mtimes: source files keep theirs across rebuilds; MANIFEST.json's is
    rewritten each build. This is why mtime is out of the digest.
  • storage.head_object_meta raises rather than returning None for a throttle, a
    timeout or an owner-pin refusal, so all three of present / absent / unanswerable
    land on the safe side.

Related Issues

Closes #12239

Out of scope by design, not oversight:

Pattern harvest

A check whose input is nondeterministic has to be taken one level in from the
artifact: the bytes of a tar.gz cannot answer "did anything change", while the entry
set can. And when a guard rests on an assumption about a module you do not own, pin
the assumption with its own test rather than a comment - here, that only created_at
is volatile. Prose cannot fail, so without that test the skip would one day stop
firing silently and the nightly would quietly go back to a full upload every night.

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 20, 2026 22:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 2781c4b37a510a07bc4884091196d56f106ce295 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the fingerprint pass's runtime on the multi-GB sessions archive it cites — every claim was measured except the nightly decompress-and-hash cost itself.

[DESIGN-REVIEWED] 2781c4b

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ human override accepted

@chenmingwei23 overrode this lane for 2781c4b37a510a07bc4884091196d56f106ce295 via /ai-review override (this commit only).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 2781c4b37a510a07bc4884091196d56f106ce295 — this comment is updated in place on each push.

Review details

The candidate discovery pass produced no candidates. I've read the full diff and worked through the core logic myself: _tree_fingerprint/_archive_entries (surrogatepass encoding, volatile-root stripping, manifest normalization), _unchanged_baseline (every non-match branch returns None → upload, version+length+presence proofs, probe authorization placed after local checks), the (process, sequence) compare-and-set in _record_run_locked/_record_skip, and the hooks._run_once uploaded is False branch. Every uncertain branch falls toward uploading, the skip record carries the matched baseline forward, and the audit label is told apart from a push. I could not ground a concrete input → call path → wrong outcome at 80+ on any changed line, and nothing in the diff crosses a security boundary or removes a guard without compensation.

No findings.

[OPUS-REVIEWED] 2781c4b

Verdict parsed from the review's SHA-scoped output markers for commit 2781c4b37a510a07bc4884091196d56f106ce295.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2781c4b37a510a07bc4884091196d56f106ce295: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 2781c4b37a510a07bc4884091196d56f106ce295 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2781c4b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2781c4b37a510a07bc4884091196d56f106ce295: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/backup-unchanged-check-12239 branch from e38a789 to fb57e1f Compare September 20, 2026 22:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head fb57e1fbcd adds a fourth test layer. The body's "35 cases in three layers" is
now 41 in four; the body itself stands, since editing it would re-roll the review
lanes for a figure a comment can carry.

New: TestRetentionComposition, six cases driving the REAL _prune_remote_archives
against a fake versioned drive that grows on upload.
Retention bounds how many
copies are kept while this change stops one being sent, and the two meet at a single
point: the next skip proves its baseline by HEADing the key the last run recorded, and
retention is the thing in this module that deletes keys. So the composition is
asserted rather than argued:

  • The sweep never deletes the archive the next skip proves, at the most aggressive
    setting (keep 1) with real candidates it recorded uploading.
  • The run's own key survives even when it sorts OLDEST. Retention drops it by NAME,
    separately from the count, and this pins that the by-name drop is the guarantee
    doing the work: removing it reddens this test. A future-dated object or a skewed
    co-writer clock is enough to put the recorded key inside the slice the count alone
    would delete.
  • A skip consumes no keep slot: four consecutive unchanged runs leave the drive's
    contents, deletions and uploads identical.
  • A skip spends no authorization round trip. Retention and each label PUT take their
    own, and a skip reaches none of them.
  • A baseline removed from the drive forces a full upload rather than a skip.
  • A changed tree after a skip uploads and sweeps again, sparing the new recorded key
    and retiring the old one.

Two facts shaped how those tests had to be built, and both were found by a test
failing rather than by reading:

  1. Retention only retires an archive whose recorded version is still the key's CURRENT
    version, so fabricated listing rows this install never recorded uploading are not
    candidates at all. A sweep over seeded rows deletes nothing and every assertion
    passes while proving nothing. The tests therefore drive real uploads.
  2. candidates = [key for key in by_age[keep:] if key != newest_key], so keep counts
    archives besides the run's own, and a fake handing every upload one modified
    stamp leaves the ordering tied and the slice arbitrary -- producing the same
    silent no-delete outcome.

Both traps are now guarded inside the tests by an explicit "retention deleted nothing,
so this proves nothing" assertion, so a future change that makes the sweep inert
cannot leave these passing.

One mutation deliberately left uncovered, disclosed rather than papered over.
Removing the if uploaded: guard around _merge_uploads in _record_run_locked
changes nothing observable today: a skip passes the matched run's own key, fingerprint
and version, so the merge would rewrite identical values. The guard is therefore
defensive, not behavioural, and the comment now says so. It is kept because the
equality is a property of _record_skip rather than of _record_run_locked -- a later
skip carrying any other key would otherwise write a map entry for an upload that never
happened, and uploaded_versions is what retention reads before it erases object
versions. Writing a test for a provable no-op would have manufactured evidence.

Scope unchanged: retention itself is untouched, no seal list is extended, and the
conversation-database gap stays out.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/backup-unchanged-check-12239 branch from fb57e1f to 5bef29a Compare September 20, 2026 23:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Review findings dispositioned at 5bef29ae181f4c21fd6cb4c1beb70d550e29b5a6

Three findings from the review lanes on the previous head. All three were verified
against the source before acting, all three were real, and all three are fixed in
code rather than argued.

1. The baseline probe reached the account without passing the consent gate (GPT, security-fenced)

_unchanged_baseline called storage.head_object_meta directly. Every other reach
out to the account in this module goes through _authorize_upload first, so the
probe was a network round trip on the operator's credentials that consent had not
approved. It now calls _authorize_upload(..., operation=SEL_OP_BASELINE_PROBE)
before the HEAD, under its own SEL operation so the audit trail distinguishes a
probe from a push.

Placement matters and is deliberate: the gate sits after the local early
returns (no last run, no recorded key, no recorded fingerprint, fingerprint
mismatch). A run with no baseline to prove therefore spends no authorization round
trip at all, which is also what keeps the "a skip spends no extra authorization"
property in TestRetentionComposition true.

A gate that raises is not swallowed. test_a_refused_probe_uploads_rather_than_skipping
pins that a withdrawn consent propagates and the run uploads, and asserts
head_object_meta was never called -- a skip must never rest on an authorization
the account has revoked.

2. Key plus length did not identify the object (GPT, security-fenced)

The skip proved the baseline by HEADing the recorded key and comparing
ContentLength. That is not an identity: this drive is reachable by more than one
install, so a co-writer can overwrite the same name with different content of the
same length, and the skip would then treat a foreign object as its own baseline --
a backup silently not backing up.

The recorded version is now compared against the version the HEAD reports. A drive
that reports no version is treated as unprovable and uploads, which is the
fail-safe direction: absence of the identifier is not evidence of identity.

test_a_drive_that_reports_no_version_uploads is the discriminating case here. It
was added after a mutation survived: removing the "no version reported" branch
changed nothing observable, because every other test's fake returned a version.
That mutation now reddens.

3. The nightly consumer could not tell a skip from a push (Opus, advisory)

hooks.py read the run record and reported every completed run as a push, so a
skip would have been announced and audited as an upload that never happened. It now
reads the uploaded field. A record without the field is treated as a push, which
is correct for every record written before this change and is pinned by
test_a_record_without_the_field_is_treated_as_a_push.

Evidence

Check Result
flake8 / isort / mypy 0 / 0 / 0
black, comment-history (base origin/main) 0 / 0
subprocess-encoding, sync-io, loop-bound-locks, brand 0 / 0 / 0 / 0
5 owned test files, -n0 363 passed
Mutation runs on the three fixes 7/7 killed
ASCII on added lines and commit message 0 / 0 non-ASCII

The mutation kills are AssertionErrors about values with non-uniform blast radii,
read rather than counted. Disabling the consent gate, moving it ahead of the local
early returns, dropping the version comparison, or accepting a missing version each
reddens its own tests.

One disclosure kept rather than papered over: the if uploaded: guard around
_merge_uploads is defensive, not load-bearing. Removing it changes nothing
observable today, because a skip passes the matched run's own key, fingerprint and
version, so the merge rewrites identical values. There is no test for it, and
writing one would have manufactured evidence for a provable no-op. The comment in
the source says so.

Scope

Unchanged from the description: retention stays out (it is already merged and owns
that half), and the conversation database stays out. The local build still runs
every night; only the upload is skipped.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/backup-unchanged-check-12239 branch from 5bef29a to 94b038e Compare September 20, 2026 23:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

94b038e5ac8f5e409fb5731f67ce791124711262

The Windows shard was production refusing correctly, and the test was wrong to ask

Two cases in TestRealBundleAssumption failed on Windows only, both as
AssertionError: assert 1 == 0 from snapshot_main. The log names the cause:

refusing to stage the data home: this platform cannot open a directory relative to a
descriptor, so every component would be re-opened by name and an ancestor swapped
mid-walk could redirect the copy into a credential store. Re-run with
--allow-unpinned-staging to accept a by-name traversal.

Windows has no dir_fd, so _staging_is_pinned refuses rather than quietly walking by
name, and the refusal is the documented behaviour with a documented way to proceed. The
tests were asking for a snapshot without saying they accepted that walk.

Both now build through one helper that adds --allow-unpinned-staging only where
pinned_fs.supports_pinned_tree_walk() is false. The assertions stay universal and
there is no platform skip: the same property is measured on every platform, in
whichever staging mode that platform runs.

Three things make that more than a flag sprinkled on a red test.

A new case reports the platform as unable to pin and measures the property in the
by-name mode, because staging is a manifest field the fingerprint keeps -- so the
result does not carry over from the pinned mode for free. It asserts its own
precondition, staging == "unpinned" on both bundles, since a simulation that failed to
reach the by-name path would satisfy everything else (a pinned walk agrees with itself
too). Locally the two modes measurably differ: staging='pinned' against
staging='unpinned'.

The helper's contract is pinned rather than described. The unchanged-tree case asserts
the recorded mode equals what this platform's capability implies, so a helper that
always asked for the by-name walk -- leaving the pinned walk untested -- turns it red.

And the diagnosis is confirmed by mutation rather than by reading. Removing the flag
branch reproduces the Windows failure exactly, same assertion and same refusal text, on
Linux under the simulated platform. That is the evidence that this is the cause and that
this is the fix, without needing a Windows host to iterate on.

The skip already proves identity, and it does not use the ETag

A review finding on the earlier head read the skip as deciding on ContentLength alone.
That was true of the head it reviewed and is not true now: the check compares the
recorded VersionId against the version the HEAD reports, and refuses the skip when the
drive names none. A same-length overwrite is caught, because an overwrite is a new
version.

The ETag is deliberately not the instrument, and this module already records why: S3's
ETag stops equalling the body MD5 under multipart and under SSE-KMS. These archives run
from 232 MB to 2.5 GB, so they are multipart, and a recorded body MD5 compared against a
HEAD's ETag would mismatch every night -- the skip would never fire and the feature
would be dead on arrival, the same failure mode as a fingerprint that never matches.

Comparing version to version has neither caveat and is strictly stronger for the actual
threat: it catches an overwrite even when the overwriting content is byte-identical,
which an ETag comparison cannot.

The cost of that choice is stated rather than hidden: on an unversioned bucket
nothing can be proven ours, so the skip never fires and every night uploads in full.
That is the fail-safe direction, it matches how retention already reads a missing
version, and the app creates its drive with versioning on.

The recovery half is deferred, not dropped

The same finding's second half is real and separable: restore_download takes a key and
no version, so it fetches whatever is current, and a fingerprint mismatch ends at
ORIGIN_UNVERIFIED and a refusal. The version we recorded is used by retention and now
by this check, but no code path can fetch by it -- so an overwritten key has no automated
route back to our bytes even though the bucket still holds them.

Filed as #12391 with the ingredients that already exist and the one question worth
settling first, which is what should happen when the recorded version is itself gone. It
is out of scope here because it is the recovery half of the problem this PR detects, and
folding it in would make this two changes.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The blocking finding on this head names a value that has a second instance
outside this diff, so recording where each one is handled:

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head 2781c4b37a. Correcting my previous comment, which dispositioned the wrong failure.

Withdrawal

My comment above quoted the failure on 1142750756 as assert '<ts>' != '<ts>' and called it a precondition that did not hold on a coarse clock. That was the failure on 2f1a35cc29, the head before it. The annotation on 1142750756 reads:

test_a_second_skip_against_the_same_baseline_refuses
AssertionError: a baseline already consumed by a skip must refuse a second

That is the behaviour assertion, not the premise. The test reached its subject and the skip was accepted when it had to be refused. I read the older annotation and did not check it against the head I was fixing, so the fix I shipped -- rewriting the test to construct the stamp by hand -- narrowed the test away from the defect instead of addressing it. Withdraw that reading.

The real defect was a contract one

_record_skip's docstring claimed the write lands only while the run slot still carries both the baseline's at and its key. Neither term can establish that the slot is still the record the baseline came from:

  • at comes from datetime.now, which resolves to the platform's clock tick. Two successive writes on a coarse clock share one value to the microsecond.
  • a skip copies the matched run's key, so after one skip the slot's key still equals the baseline's by construction.

So both terms compare equal while the slot holds a different record, and a second skip overwrites the first. Windows produced both collisions at once. The docstring asserted a property the code did not have.

The fix uses a token the module already writes

Every run record already carries sequence, bumped on each write, and process, which carries the pid. _run_is_newer already pairs them, for the same reason a bare sequence is not enough: it counts one process's own writes, so two processes can both sit at the same number. The compare-and-set now compares that pair, and the docstring's claim is literally true on every platform.

at and key are no longer compared at all. That is measured rather than assumed: with the pair in place, no mutation of either term is observable, because the pair already refuses every state either could catch. Keeping an unreachable term would read as depth it does not provide.

A record predating these fields carries neither, so the type guards refuse and a full copy uploads. That is the direction every other proof in this design fails toward, and it is why there is no fallback to comparing at alone -- a fallback would reopen the collision this closes.

The test that was removed is restored

test_a_second_skip_against_the_same_baseline_refuses is back, unchanged in intent, and now passes on every platform because the sequence separates the records regardless of clock granularity. Four more cases were added, each isolating a term no other term can see:

Case Term it isolates
a second skip whose fresh stamp lands in the same clock tick the sequence comparison
a matching sequence carried by another process the process comparison
an empty process on both sides the process presence guard
a record carrying a process but no sequence the two sequence type guards, as a pair

Evidence

357 tests across the four owned backup test files pass with -n0, exit 0. Seven mutations over the guard: five killed with radii 1, 1, 1, 7 and 10, and every kill an assertion about a value.

The two remaining survivors are the sequence type guards taken one at a time, and that is reported rather than papered over: each covers the other on a record with no sequence, so dropping either alone is not observable while dropping both accepts an absent sequence as proof of identity. The pair is pinned jointly, the code comment says so, and the joint mutation is the seventh above.

The first mutation run was itself informative. It reported the sequence comparison as surviving, which was true and not a false alarm: on a fine clock the at term refused first and masked it. That is what prompted removing the masking terms and building the coarse-clock case by hand.

flake8, isort, mypy, the black gate, comment-history, brand, subprocess-encoding and docs-lint all report 0. Zero non-ASCII in added lines and in the commit message. One commit, five files, working tree clean at push. The owning spec and the commit message both state the identity the code now compares.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head 2781c4b37a. Answering Design Review's watch item, which asked for the one cost this change had not measured: the fingerprint pass's own decompress-and-hash time. No code change; this is the measurement.

Measured, on two archive shapes

Shape Members Uncompressed Packed Best of 3
few large files 45 319.1 MB 99.3 MB 1.356 s
many small files 20000 82.3 MB 0.9 MB 1.113 s

The cost has two terms, not one rate

Solving the two shapes together gives roughly:

time  ~=  uncompressed_bytes / 235 MB/s  +  members * 38 us

The large-file case is entirely byte-bound: 319.1 MB at 235 MB/s predicts 1.358 s against 1.356 s measured, leaving no room for per-member cost. The small-file case is the opposite: its bytes account for 0.35 s of the 1.113 s, so the remaining 0.76 s over 20000 members is the per-member overhead.

One caveat worth more than the number

A packed-bytes rate does not carry across shapes and must not be extrapolated. The small-file tree compresses 91x, because every member is nearly identical, so its 0.8 MB/s packed rate would predict 50 minutes for a 2.5 GB packed archive when the real cost is bounded by uncompressed bytes and member count. Extrapolate from those two terms, never from a packed rate.

For a 2.5 GB packed sessions archive at a realistic 4x ratio, roughly 10 GB uncompressed and say 50000 members, the model gives about 43 s of hashing plus about 2 s of per-member overhead. Tens of seconds, and stated as a model rather than as a measurement of that archive, because I do not have one that size to time.

Why that is cheap here rather than merely small

The archive is built on every run whether or not the upload is skipped, which this PR's body already discloses as a scope boundary. So building has already read and compressed the entire tree, and the fingerprint pass is one additional sequential read of a file written moments earlier, which is the case most likely to still be in page cache. What it buys on an unchanged night is not sending 10.1 GB over the network.

The probe drove the real _tree_fingerprint, asserted a non-empty digest before trusting any timing, and took the best of three runs. It was a throwaway and is not part of the diff; the tree is clean at 2781c4b37a.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 2781c4b: Default accepted as the product decision: the nightly skips the upload when the tree is unchanged. rfc-s3-backup.md is the covering RFC and its draft status is a documentation lag, not an unmade decision. Fail-closed is preserved: any comparison that cannot be proven uploads anyway. Accepting the recorded-RFC gap.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 2781c4b37a510a07bc4884091196d56f106ce295.

Default accepted as the product decision: the nightly skips the upload when the tree is unchanged. rfc-s3-backup.md is the covering RFC and its draft status is a documentation lag, not an unmade decision. Fail-closed is preserved: any comparison that cannot be proven uploads anyway. Accepting the recorded-RFC gap.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Branch frozen: the first-principles override recorded above is pinned to 2781c4b. Any further push voids it and the block returns. Do not push this branch.

@iamwhatever
iamwhatever merged commit 4e8af7b into main Sep 21, 2026
93 of 94 checks passed
@iamwhatever
iamwhatever deleted the fix/backup-unchanged-check-12239 branch September 21, 2026 06:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 21, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 2781c4b37a.

I read _tree_fingerprint, _archive_entries, _manifest_digest, _unchanged_baseline, _record_skip and the _record_run_locked CAS rather than taking the description.

The comparison is taken at the right level. A tar.gz embeds per-entry mtimes and a gzip stamp, so a body-level check would report "changed" every night and the skip could never fire. Digesting the entry set read back out of the packed payload is the stronger of the two options for the reason given: a second source walk would have to re-derive what snapshot.COMPONENTS and _add_tree pack, and the day the two disagreed the answer would be "unchanged" for a tree that had moved. It also sees a redacted upload, which a source walk would not.

Every uncertain branch uploads, and I checked each one. No tree fingerprint, no recorded tree, fingerprints differ, no recorded key, object absent, recorded version not provable, stored version not ours, length mismatch, HEAD unanswerable, archive unreadable — all return None. _is_provable_version_id rejecting "null" is the detail that matters most: on a suspended-versioning bucket both sides report "null", and a length-only check would let a co-writer's overwrite pass while restore_download would then refuse the foreign object as ORIGIN_UNVERIFIED with no automated path back. Declining to skip there is right.

The two normalizations are bounded and justified. Only created_at is dropped from the manifest, not the whole member, so a switch to an unpinned staging walk is still noticed — and the assumption about snapshot.py is pinned by its own two-real-bundles test rather than a comment, which is the part that keeps this from silently ceasing to fire.

The skip does not strand the loop or the baseline. Fresh at so due_for_nightly does not rebuild every wake; matched key/fingerprint/version copied forward so tomorrow has something provable and retention has a version to retire; (process, sequence) compare-and-set so a concurrent run's record is not replaced by a stale baseline; neither label publish nor retention runs on a skip.

mtime being out of the digest is correct — a restore or checkout bumps it without changing a byte and would spend the very upload this removes.

Advisory lanes are clean at this head and the first-principles override is an accepted RFC-recording gap with fail-closed behaviour preserved, so I am not re-litigating it. Landing this before #12190 so the SHA-pinned override is not voided by a rebase.

bolichen97 added a commit that referenced this pull request Sep 21, 2026
…e-checks

`Backend Lint & Type Check (3.12)` is red on main and therefore on every open
PR:

    backup.py:2526: error: Missing named argument "payload_kind" for
    "_authorize_upload"  [call-arg]

Two merges eleven seconds apart did it. #12190 (619c10f, 06:11:07Z) made
`payload_kind` a required keyword-only parameter; #12381 (4e8af7b,
06:11:18Z) added this call site without it. The second PR's CI ran against a
base where the parameter did not yet exist, so it merged textually clean and
semantically stale -- neither PR is wrong on its own.

The value is not a mypy silencer. The parameter's own docstring says `None` is
a real answer meaning this write carries no kind's payload, and the baseline
probe is a HEAD of an archive already in the bucket: it writes nothing, so no
per-kind grant governs it. Line 2413 states `None` at the comparable site.

Naming a kind here would be a behaviour change, not a cosmetic one: it would
subject a read to that kind's grant, so a withdrawn snapshot consent would
refuse the PROBE -- which this module answers by uploading, making the nightly
re-upload an unchanged tree every night, the exact cost the baseline exists to
avoid.

Nothing covered that. With the wrong kind passed, all 357 tests across the four
aws-control backup suites still passed, so the fix arrives with a regression
test on the assertion that was missing: the probe's authorization carries
`payload_kind is None`. Negative control fires -- `assert 'snapshot' is None`.

511 tests pass across five aws-control suites; mypy, black and flake8 are clean
on both touched files.

## Pattern harvest

Rule candidate: none for the typo class; the type checker already catches it,
and it did.

Not generalizable: the missing argument itself is a one-site omission.

The RECURRENCE is generalizable and is the third time this session has met it:
two PRs merged seconds apart, each green against a base that lacked the other,
combining into a break neither one contains. A required status check cannot see
it, because both were checked before the other existed. The candidate fix is a
merge-time re-check rather than a rule in any source file: re-run the type
check on the merge result when a PR's base moved after its last green, which is
the only moment the combination first exists.
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length;
the security rule is that a leading '-' is refused, because get_file passes the id
as a bare argv element and the AWS CLI's own option grammar would read one as
another option. There is no shell in the path.

_archive_is_readable walks every member AND requires the tar's end-of-archive
marker. Walking alone is not enough: tarfile swallows a header truncated anywhere
but the first byte and reports a normal end of archive, so a valid gzip wrapping an
incomplete tar reads as fine.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length,
applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored
pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security
rule is that a leading '-' is refused, because get_file passes the id as a bare argv
element and the AWS CLI's own option grammar would read one as another option. There
is no shell in the path. apps/aws-control/data is neither agent-readable nor
agent-writable -- it sits behind the agent file-tool floor and is bind-masked from
every agent sandbox -- so this check stands between a stored value and an argv
element, not between an agent and the CLI.

_archive_is_readable walks every member AND requires the tar's end-of-archive
marker. Walking alone is not enough: tarfile swallows a header truncated anywhere
but the first byte and reports a normal end of archive, so a valid gzip wrapping an
incomplete tar reads as fine.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length,
applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored
pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security
rule is that a leading '-' is refused, because get_file passes the id as a bare argv
element and the AWS CLI's own option grammar would read one as another option. There
is no shell in the path. apps/aws-control/data is neither agent-readable nor
agent-writable -- it sits behind the agent file-tool floor and is bind-masked from
every agent sandbox -- so this check stands between a stored value and an argv
element, not between an agent and the CLI.

A fingerprint match is the whole test, and nothing else is asked of the recovered
bytes. Whether they still open as a tar.gz is not checked, because the
current-version read does not check it either and the upload side pushes payloads it
cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive
can legitimately be malformed. Refusing one only on the recovery path would hand the
operator their own file when nobody overwrote the key and a refusal when somebody did,
for identical bytes, at the cost of decompressing a multi-gigabyte archive to get
there.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length,
applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored
pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security
rule is that a leading '-' is refused, because get_file passes the id as a bare argv
element and the AWS CLI's own option grammar would read one as another option. There
is no shell in the path. apps/aws-control/data is neither agent-readable nor
agent-writable -- it sits behind the agent file-tool floor and is bind-masked from
every agent sandbox -- so this check stands between a stored value and an argv
element, not between an agent and the CLI.

A fingerprint match is the whole test, and nothing else is asked of the recovered
bytes. Whether they still open as a tar.gz is not checked, because the
current-version read does not check it either and the upload side pushes payloads it
cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive
can legitimately be malformed. Refusing one only on the recovery path would hand the
operator their own file when nobody overwrote the key and a refusal when somebody did,
for identical bytes, at the cost of decompressing a multi-gigabyte archive to get
there.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length,
applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored
pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security
rule is that a leading '-' is refused, because get_file passes the id as a bare argv
element and the AWS CLI's own option grammar would read one as another option. There
is no shell in the path. apps/aws-control/data is neither agent-readable nor
agent-writable -- it sits behind the agent file-tool floor and is bind-masked from
every agent sandbox -- so this check stands between a stored value and an argv
element, not between an agent and the CLI.

A fingerprint match is the whole test, and nothing else is asked of the recovered
bytes. Whether they still open as a tar.gz is not checked, because the
current-version read does not check it either and the upload side pushes payloads it
cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive
can legitimately be malformed. Refusing one only on the recovery path would hand the
operator their own file when nobody overwrote the key and a refusal when somebody did,
for identical bytes, at the cost of decompressing a multi-gigabyte archive to get
there.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
chenmingwei23 added a commit that referenced this pull request Sep 21, 2026
…y is overwritten

One backup drive is reachable by every install pointed at the account, and
versioning is on for that reason. When a co-writer overwrites a key this install
recorded, our archive stays on the drive as a noncurrent version. The restore read
named no version, so it fetched the overwriting object, failed the recorded body
fingerprint and refused -- with our archive present and unnameable.

storage.get_file now takes an optional version, and when an ORIGIN_SELF key fails
the fingerprint backup._recover_recorded_version makes exactly one further read, of
the VersionId uploaded_versions records for that key. It is accepted only on the
same evidence the current-version path uses -- that fingerprint, re-taken over the
bytes that arrive -- plus _archive_is_readable. Absent, deleted, malformed or
mismatched returns the existing ORIGIN_UNVERIFIED refusal, so the recovery can only
find bytes that already pass; it can never widen what a restore accepts.

The recovery runs only where the mismatch would REFUSE. Under foreign_ok the caller
has already said it will take whatever is current without proof, so there is no
refusal to rescue, and reaching past the current object would hand that caller
different bytes than it accepted, relabelled as a proven self archive. The reply
carries recovered, because the alternative outcome for this case is a refusal, which
is loud: an operator who silently receives a recovered archive would never learn a
co-writer is overwriting their backup chain.

Version ids are validated as data. They are opaque and drawn from the full base64
alphabet, so the shape is printable ASCII with no whitespace and a bounded length,
applied with fullmatch: '$' also matches BEFORE a trailing newline, so an anchored
pattern accepts "abc\n" and sends a whitespace-bearing id to the CLI. The security
rule is that a leading '-' is refused, because get_file passes the id as a bare argv
element and the AWS CLI's own option grammar would read one as another option. There
is no shell in the path. apps/aws-control/data is neither agent-readable nor
agent-writable -- it sits behind the agent file-tool floor and is bind-masked from
every agent sandbox -- so this check stands between a stored value and an argv
element, not between an agent and the CLI.

A fingerprint match is the whole test, and nothing else is asked of the recovered
bytes. Whether they still open as a tar.gz is not checked, because the
current-version read does not check it either and the upload side pushes payloads it
cannot read (_tree_fingerprint returns "" for an unreadable tar.gz), so an own archive
can legitimately be malformed. Refusing one only on the recovery path would hand the
operator their own file when nobody overwrote the key and a refusal when somebody did,
for identical bytes, at the cost of decompressing a multi-gigabyte archive to get
there.

Retention is deliberately untouched. _current_version_is_ours stays a question about
the current version, so a recoverable-but-noncurrent key still holds no keep slot --
which errs toward retaining more.

Closes #12391
Refs #12381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants