Repository navigation
fix(aws-control): claim the nightly backup through the Job SDK - #17526
bolichen97 merged 1 commit into
Conversation
4f69df2 to
b033c5e
Compare
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (9 items) — 5 justifiedIntent: make the scheduled backup show up on the Backup row and dedupe against a manual click. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 56712d6 Description read[DESCRIPTION-READ] 2cd594131feafb128a770004dc8ca9c538f72934c51d8e30d589efe86818cf9a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that [DESIGN-REVIEWED] 56712d6 Description read[DESCRIPTION-READ] 90e2f4cecc551640eaa705703ab021ec502ee825e2faf6e82e4050d0e0eb21f5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
b033c5e to
b25579f
Compare
|
Both blocking findings are addressed at No-drive / no-connection became a recorded failure (Design Review): fixed. In the runner, a scheduled run that finds no working connection or no drive yet now returns as a clean skip — it records no Dropped shared-drive notice and unchanged/succeeded audit (both findings): restored. The The description's compatibility section no longer claims "removes nothing" and now states that these records move into the runner, and |
b25579f to
a6bf55a
Compare
|
Addressed at Tests reached an unstubbed AWS LIST (GPT 6.1, blocking): fixed. The scheduled runner tests that drive A no-drive wake left a The one-paid-LIST pin was deleted (First Principles): restored, pointed at The shared-drive LIST ran for every due kind (Opus): fixed. It now runs once, on the snapshot kind only, so a both-due wake pays one LIST (not two) and a sessions-only wake pays none. The description and |
Security Scope Review (fork) — 🔴 BLOCK (blocking)Which legitimate operations does no leg reported, and this was not the nothing-to-adjudicate short circuit No |
a6bf55a to
ce2fe0a
Compare
ce2fe0a to
99cbf81
Compare
|
This routes the scheduled AWS backup through the same |
99cbf81 to
7fb1f30
Compare
|
Rebased onto current main and pushed 1. Adopted-run caller. Because a scheduled and a manual backup of one account share one 2. Stale profile before The diff size and the credentialed-path concern are the maintainer's call to make. Unrelated CI note: the The advisory First Principles lane reports a premise/scope concern (the diff size and the credentialed-path rewrite). That is the same maintainer-reserved question raised above, so it is left for the maintainer rather than coded around — the change does what the issue asks; whether it should take this shape is the maintainer's call. The GPT 6.1 lane raised one blocking test-only finding — a determinism race in a helper that let a second same-account run adopt the first before its slot was released — which is now fixed by having the helper wait for the run to leave the live set; no product behaviour changed. Rebased onto current main ( Pushed Pushed |
7fb1f30 to
24f493b
Compare
24f493b to
b20a153
Compare
The nightly loop ran the backup directly on a thread, so a scheduled run
claimed no Job SDK run: it shared no (kind, dedupe_key) with a manual click
(both could upload, both paid) and the Backup row read idle while a nightly
ran. _run_once now fires start_async(kind, dedupe_key=account,
params={caller: scheduled}) per due kind -- the same call the manual route
makes -- so the two callers dedupe and the row sees the run. The one
registered runner reads the caller from params and, for a scheduled run,
records a failure into the backoff due_for_nightly reads (minus teardown), so
a deterministic nightly fault still is not retried every half hour.
Two refinements keep the shared-caller path honest:
Adopted-run attribution. Because both callers share one (kind, dedupe_key),
an owner's "Back up now" click during a nightly adopts the in-flight
scheduled run -- the SDK returns the first caller's run and params do not
merge, so the adopted run stays CALLER_SCHEDULED and runs the scheduled-only
re-checks. A scheduled-only precondition the owner controls (the unattended
grant withdrawn, or scheduled transcripts blocked because redaction is on)
now raises ScheduledUploadWithheld; the runner audits it withheld and does
NOT feed the nightly backoff. Without this an owner who clicked during a
nightly and surfaced one of those gates would see a failure they did not
cause and watch the backoff grow. An unregistered-kind fail-closed stays a
fault that backs off.
Stale profile fallback. resolve_account_profile_cached serves only from the
warm snapshot and returns None once it is past _PROBE_TTL_SECS, and
start_async is fire-and-forget, so the runner resolves later than the loop's
live probe; a gap past the TTL audited a just-verified nightly as failed. The
loop now names the (profile, region) it freshly probed under
JOB_PARAM_PROFILE / JOB_PARAM_REGION, and a scheduled run falls back to that
pair when the cache has expired. The live sts:GetCallerIdentity re-check still
guards the account, and the bucket is still re-discovered per run.
Fixes kirodotdev#7783
- Restore the per-kind "cancelled" SEL audit on a setup (drive-lookup) cancel
(dropped in the rewrite): teardown still leaves a trail it was interrupted,
without backing off. The loop-side failure and cancel audits route through
backup._nightly_audit / _nightly_subject (one audit helper, not a second
inline copy). Pinned by
test_a_cancel_during_the_lookup_audits_cancelled_but_does_not_back_off.
b20a153 to
56712d6
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approved at this head. Both named findings are fixed: the scheduled path now falls back to the profile it just verified (no TTL race before start_async), and an adopted run's refusal no longer feeds the nightly backoff. Credential and consent checks are kept, and _authorize_upload still does a live STS check. Small follow-up: docs/system-specs/modules/aws-control.md still mentions hooks._push_nightly, which this PR removes.
Posted by an AI assistant at bolichen97's request, after a read-only review of this head.
Problem / Motivation
Goal: make the scheduled nightly backup go through the same Job SDK call the
manual button uses, so the Backup row shows the nightly run and a nightly and a
manual click cannot both run (and both pay) at the same time.
The nightly loop ran the backup directly instead of starting a Job SDK run. Because
it shared no
(kind, dedupe_key)with the manual route, the two callers neverdeduped: a nightly and a hand-clicked backup could run concurrently and each pay for
its own upload. And since no SDK run was started, the Backup row read idle the
whole time a nightly was uploading. This is pre-existing on
origin/main.Why it matters
The AWS backup runs unattended, off-host, every night. With the nightly bypassing the
Job SDK, the operator's Backup row says idle while a nightly is actually uploading, so
there is no visible record of the run, and a manual click that happens to overlap
shares no dedupe key with it — both uploads proceed and both are charged. Routing the
nightly through the same
start_async(kind, dedupe_key=account, ...)call the manualroute makes means the two callers dedupe on one key and the row shows the run. Anyone
who has the scheduled backup enabled is on this path.
Not a goal
gates. The scheduled run still goes through the same authorization path with the
scheduled caller; only the entry point changes from a direct call to an SDK run.
What changed (motivation → approach → change)
Symptom: the nightly loop ran the backup directly, so it claimed no SDK run — the
Backup row read idle during a nightly and a manual click shared no dedupe key with it
(both could upload, both paid). Root cause: the scheduled path did not start a Job
SDK run. Change: the loop now starts the run through the same SDK call the manual
route uses.
hooks.py:_run_once(sdk)firessdk.start_async(kind, dedupe_key=account, params={caller: scheduled})per due kind — the exact call the manual route makes —so the scheduled and manual callers dedupe on one key and the Backup row sees the
run.
_loopfetches the live SDK each wake (so it survives a re-enable). Thedue-check and consent gating are unchanged, and the loop still checks for the drive
before claiming — a wake with no drive yet claims no run, exactly as before. The old
direct-run helpers the loop no longer uses are removed.
backup.pyrunner (make_job_runner): the one registered runner reads the callerfrom
handle.params(defaulting to the owner caller), passes it to theauthorization gate and the work so the scheduled run keeps its own caller instead of
being treated as an owner-initiated upload. For a scheduled run it records a failure
into the backoff
due_for_nightlyreads — excluding teardown — so a deterministicnightly fault is still not retried every half hour. The loop checks for the drive
BEFORE claiming a run (below), so when there is nowhere to push no run is claimed at
all — an account whose drive is merely not created yet does not get a
donerun onthe Backup row every wake (which would clear its last real failure) nor accrue a
backoff. The runner also carries the unattended SEL trail (invoked / succeeded /
unchanged/ failed) and the shared-drive notice (backup_shared_drive, checkedonce on the snapshot kind only), which have no HTTP handler to supply them on the
scheduled path — the SDK's generic run record cannot tell an unchanged night from an
upload.
uploads.py: adds thecallerparam key the runner reads.This is the smallest change that matches the existing manual route: the loop makes the
same SDK call, and the runner reads one extra param. Only the two names the runner
actually needs are promoted to real globals; the rest of the facade is unchanged.
Why the backoff stays: dropping it would be a regression — a nightly that fails the
same way every time would retry every ~30 min, because
due_for_nightlycomputes thebackoff from the recorded-failure state, not from SDK run records; the smallest form
that still feeds that record from the scheduled run's outcome is kept.
Backwards compatibility
Removes no behaviour: the backup is built, authorized and uploaded exactly as before,
and the unattended SEL audit trail (invoked / succeeded /
unchanged/ failed) and theshared-drive notice are preserved — they move from the loop into the run's runner so
they still fire on the scheduled path, now wrapped around the SDK run rather than an
inline call. No field, kind, key, or public function is removed, renamed, or tightened;
the owner caller stays the default when no caller param is present, so existing callers
are unaffected.
docs/system-specs/modules/aws-control.mdis updated to say thescheduled backup claims a Job SDK run and that the runner carries that audit.
Tests
test/test_aws_control_hooks.py: the loop claims each due kind through the SDK(
start_async(kind, dedupe_key=account, params={caller: scheduled})), and both duekinds are each claimed once and independently; the early-return guards and the
loop supervisor / startup-shutdown paths are preserved.
test/test_aws_control_backup_job.py: the runner reads the caller fromparamsand applies the scheduled caller; a failed scheduled run records the backoff and a
second failure withholds the next wake (
due_for_nightlygoes False), a completingrun clears the backoff, and neither an owner failure nor a teardown feeds it; a
scheduled run with no connection or no drive is a clean skip that records no backoff;
and a scheduled run emits its SEL trail — invoked, then
succeededorunchangedfrom the run record, plus
backup_shared_drivewhen another install shares thedrive — while an owner run emits none (its audit comes from the HTTP layer).
test/test_aws_control_backup_composition_contract.py: the new param key is addedto the frozen public-name set.
test/test_aws_control_app.pyandtest/test_aws_control_nightly_sessions.py:updated to the
_run_once(sdk)surface; loop-claim and withheld-transcriptcoverage retained.
Tests removed and why: the deleted cases in
test_aws_control_hooks.pyandtest_aws_control_nightly_sessions.pyexercised the loop's old direct-run internals(the inline push/cancel paths and the shared-setup failure attribution that only
existed while the loop ran the backup itself) — behavior this change removes, now that
the loop only starts an SDK run. The behaviours that still exist were moved to the
runner's tests rather than dropped: the backoff on a repeated failure, the clear on
success, the no-drive/no-connection skip, and the invoked/succeeded/
unchangedauditsplit plus the shared-drive notice.
Locally: the directly-testing suites pass (
-n 2), composition contract passes, andblack / isort / flake8 / mypy / comment-history are clean.
Manual verification
N/A — unit coverage sufficient; no UI or external-service path changed.
Reproduction
The loop-claim regression tests fail on
origin/mainand pass on this branch; thefailure on
mainisTypeError: _run_once() takes 0 positional arguments but 1 was given, because onmainthe loop takes no SDK and runs the backup inline rather thanstarting an SDK run. A dedicated test pins that a repeated scheduled failure records
the backoff and withholds the next wake.
Pattern harvest
Rule candidate: review-prompt
Pattern: a scheduled/background path that does the work inline instead of through the
same job/queue entry point its interactive counterpart uses will not dedupe against
that counterpart and will leave the status surface showing idle — route both through
one call with a shared dedupe key.
Related Issues
Fixes #7783
CI note
Review lanes fail on infra, run 37722951415.
Why not a smaller change
A review raised whether this is larger than the fix needs. It is not: the issue is that a nightly backup ran on a bare thread, outside the Job SDK the manual "Back up now" click uses. Routing the nightly through the SDK is the fix, not an elaboration of it.
What a smaller patch would leave broken:
(kind, dedupe_key=account)index is what makes one adopt the other — that is the whole point of the change, not a side feature.The credential and consent re-checks are preserved, not loosened: the loop still resolves the account/profile under the recorded grant key (strict resolution, no registry-default fallback), still runs
probe_identity(use_cache=False)so the livests:GetCallerIdentityguards the account each wake, and the claimed run namesCALLER_SCHEDULEDso the shared runner applies the unattended-only gates and attributes spend to the schedule. Pinned byTestClaimsTheRunThroughTheSdkand the scheduled-fallback testsTestScheduledPathFallsBackToTheVerifiedProfile.On the two pins the review read as dropped:
cancelledSEL audit was dropped in the rewrite and is now restored: a cancel during the drive lookup auditscancelledper due kind (teardown stays visible) without backing off. Pinned bytest_a_cancel_during_the_lookup_audits_cancelled_but_does_not_back_off.test_the_notice_costs_exactly_one_paid_list_calllives intest/test_aws_control_backup_job.py(classTestSharedDriveNoticeCostsOneListCall), re-anchored to the unchangedother_install_ids, which still makes exactly one paid LIST for the snapshot prefix.