Skip to content

fix(aws-control): claim the nightly backup through the Job SDK - #17526

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/7783-nightly-backup-through-job-sdk
Oct 11, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:fix/7783-nightly-backup-through-job-sdk

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: make the scheduled nightly backup go through the same Job SDK call the
manual button uses, so the Backup row shows the nightly run and a nightly and a
manual click cannot both run (and both pay) at the same time.

The nightly loop ran the backup directly instead of starting a Job SDK run. Because
it shared no (kind, dedupe_key) with the manual route, the two callers never
deduped: a nightly and a hand-clicked backup could run concurrently and each pay for
its own upload. And since no SDK run was started, the Backup row read idle the
whole time a nightly was uploading. This is pre-existing on origin/main.

Why it matters

The AWS backup runs unattended, off-host, every night. With the nightly bypassing the
Job SDK, the operator's Backup row says idle while a nightly is actually uploading, so
there is no visible record of the run, and a manual click that happens to overlap
shares no dedupe key with it — both uploads proceed and both are charged. Routing the
nightly through the same start_async(kind, dedupe_key=account, ...) call the manual
route makes means the two callers dedupe on one key and the row shows the run. Anyone
who has the scheduled backup enabled is on this path.

Not a goal

  • Changing what a backup uploads, where it uploads, or the consent/authorization
    gates. The scheduled run still goes through the same authorization path with the
    scheduled caller; only the entry point changes from a direct call to an SDK run.
  • Re-tiering, disabling, or re-platforming the backup.
  • Reworking the Job SDK's run model or the Backup row UI.

What changed (motivation → approach → change)

Symptom: the nightly loop ran the backup directly, so it claimed no SDK run — the
Backup row read idle during a nightly and a manual click shared no dedupe key with it
(both could upload, both paid). Root cause: the scheduled path did not start a Job
SDK run. Change: the loop now starts the run through the same SDK call the manual
route uses.

  • hooks.py: _run_once(sdk) fires sdk.start_async(kind, dedupe_key=account, params={caller: scheduled}) per due kind — the exact call the manual route makes —
    so the scheduled and manual callers dedupe on one key and the Backup row sees the
    run. _loop fetches the live SDK each wake (so it survives a re-enable). The
    due-check and consent gating are unchanged, and the loop still checks for the drive
    before claiming — a wake with no drive yet claims no run, exactly as before. The old
    direct-run helpers the loop no longer uses are removed.
  • backup.py runner (make_job_runner): the one registered runner reads the caller
    from handle.params (defaulting to the owner caller), passes it to the
    authorization gate and the work so the scheduled run keeps its own caller instead of
    being treated as an owner-initiated upload. For a scheduled run it records a failure
    into the backoff due_for_nightly reads — excluding teardown — so a deterministic
    nightly fault is still not retried every half hour. The loop checks for the drive
    BEFORE claiming a run (below), so when there is nowhere to push no run is claimed at
    all — an account whose drive is merely not created yet does not get a done run on
    the Backup row every wake (which would clear its last real failure) nor accrue a
    backoff. The runner also carries the unattended SEL trail (invoked / succeeded /
    unchanged / failed) and the shared-drive notice (backup_shared_drive, checked
    once on the snapshot kind only), which have no HTTP handler to supply them on the
    scheduled path — the SDK's generic run record cannot tell an unchanged night from an
    upload.
  • uploads.py: adds the caller param key the runner reads.

This is the smallest change that matches the existing manual route: the loop makes the
same SDK call, and the runner reads one extra param. Only the two names the runner
actually needs are promoted to real globals; the rest of the facade is unchanged.

Why the backoff stays: dropping it would be a regression — a nightly that fails the
same way every time would retry every ~30 min, because due_for_nightly computes the
backoff from the recorded-failure state, not from SDK run records; the smallest form
that still feeds that record from the scheduled run's outcome is kept.

Backwards compatibility

Removes no behaviour: the backup is built, authorized and uploaded exactly as before,
and the unattended SEL audit trail (invoked / succeeded / unchanged / failed) and the
shared-drive notice are preserved — they move from the loop into the run's runner so
they still fire on the scheduled path, now wrapped around the SDK run rather than an
inline call. No field, kind, key, or public function is removed, renamed, or tightened;
the owner caller stays the default when no caller param is present, so existing callers
are unaffected. docs/system-specs/modules/aws-control.md is updated to say the
scheduled backup claims a Job SDK run and that the runner carries that audit.

Tests

  • test/test_aws_control_hooks.py: the loop claims each due kind through the SDK
    (start_async(kind, dedupe_key=account, params={caller: scheduled})), and both due
    kinds are each claimed once and independently; the early-return guards and the
    loop supervisor / startup-shutdown paths are preserved.
  • test/test_aws_control_backup_job.py: the runner reads the caller from params
    and applies the scheduled caller; a failed scheduled run records the backoff and a
    second failure withholds the next wake (due_for_nightly goes False), a completing
    run clears the backoff, and neither an owner failure nor a teardown feeds it; a
    scheduled run with no connection or no drive is a clean skip that records no backoff;
    and a scheduled run emits its SEL trail — invoked, then succeeded or unchanged
    from the run record, plus backup_shared_drive when another install shares the
    drive — while an owner run emits none (its audit comes from the HTTP layer).
  • test/test_aws_control_backup_composition_contract.py: the new param key is added
    to the frozen public-name set.
  • test/test_aws_control_app.py and test/test_aws_control_nightly_sessions.py:
    updated to the _run_once(sdk) surface; loop-claim and withheld-transcript
    coverage retained.

Tests removed and why: the deleted cases in test_aws_control_hooks.py and
test_aws_control_nightly_sessions.py exercised the loop's old direct-run internals
(the inline push/cancel paths and the shared-setup failure attribution that only
existed while the loop ran the backup itself) — behavior this change removes, now that
the loop only starts an SDK run. The behaviours that still exist were moved to the
runner's tests rather than dropped: the backoff on a repeated failure, the clear on
success, the no-drive/no-connection skip, and the invoked/succeeded/unchanged audit
split plus the shared-drive notice.

Locally: the directly-testing suites pass (-n 2), composition contract passes, and
black / isort / flake8 / mypy / comment-history are clean.

Manual verification

N/A — unit coverage sufficient; no UI or external-service path changed.

Reproduction

The loop-claim regression tests fail on origin/main and pass on this branch; the
failure on main is TypeError: _run_once() takes 0 positional arguments but 1 was given, because on main the loop takes no SDK and runs the backup inline rather than
starting an SDK run. A dedicated test pins that a repeated scheduled failure records
the backoff and withholds the next wake.

Pattern harvest

Rule candidate: review-prompt
Pattern: a scheduled/background path that does the work inline instead of through the
same job/queue entry point its interactive counterpart uses will not dedupe against
that counterpart and will leave the status surface showing idle — route both through
one call with a shared dedupe key.

Related Issues

Fixes #7783

CI note

Review lanes fail on infra, run 37722951415.

Why not a smaller change

A review raised whether this is larger than the fix needs. It is not: the issue is that a nightly backup ran on a bare thread, outside the Job SDK the manual "Back up now" click uses. Routing the nightly through the SDK is the fix, not an elaboration of it.

What a smaller patch would leave broken:

  • Double upload. The nightly and a manual click for the same account would both do the paid upload, because nothing makes them dedupe. The SDK's (kind, dedupe_key=account) index is what makes one adopt the other — that is the whole point of the change, not a side feature.
  • Invisible run. The Backup row reads claimed SDK runs. A bare-thread nightly leaves the row idle while it uploads, and a nightly that failed is never visible there. Claiming the run is what puts the nightly on the same status surface as every other run.

The credential and consent re-checks are preserved, not loosened: the loop still resolves the account/profile under the recorded grant key (strict resolution, no registry-default fallback), still runs probe_identity(use_cache=False) so the live sts:GetCallerIdentity guards the account each wake, and the claimed run names CALLER_SCHEDULED so the shared runner applies the unattended-only gates and attributes spend to the schedule. Pinned by TestClaimsTheRunThroughTheSdk and the scheduled-fallback tests TestScheduledPathFallsBackToTheVerifiedProfile.

On the two pins the review read as dropped:

  • The setup-cancel cancelled SEL audit was dropped in the rewrite and is now restored: a cancel during the drive lookup audits cancelled per due kind (teardown stays visible) without backing off. Pinned by test_a_cancel_during_the_lookup_audits_cancelled_but_does_not_back_off.
  • The one-LIST notice pin was moved, not deleted: test_the_notice_costs_exactly_one_paid_list_call lives in test/test_aws_control_backup_job.py (class TestSharedDriveNoticeCostsOneListCall), re-anchored to the unchanged other_install_ids, which still makes exactly one paid LIST for the snapshot prefix.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 6, 2026 21:01
@chenmingwei23
chenmingwei23 requested a review from buluoray October 6, 2026 21:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from 4f69df2 to b033c5e Compare October 8, 2026 02:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 56712d67fba7d2f2c3b47aff8771661c90217de6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 56712d67fba7d2f2c3b47aff8771661c90217de6 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

ScheduledUploadWithheld is new, undeclared surface. It guards a corner case this PR creates, and it also quietly changes backoff for every scheduled run.

Not justified as shipped

  • Item 5 — undeclared, oversized: the description never mentions it. Its stated harm is only the adopted-click case: "an owner surfacing one of these gates would see a failure they did not cause." Yet withheld=True now stops a plain nightly whose grant was withdrawn from feeding record_nightly_failure, where base _failed_attempt did record it.
    Subtraction: delete ScheduledUploadWithheld, the withheld kwarg on _refuse_upload and the withheld branch in _run, so every scheduled fault backs off as on base.
    Clears when: the class is removed, or a reproduced adopted-run backoff shows it is needed.
  • Item 3 — undeclared: an owner's "Back up now" during a nightly now runs under CALLER_SCHEDULED, with the unattended-only gates. Only the spec diff says so.
  • Item 4 — undeclared: the description says uploads.py "adds the caller param key". It actually adds JOB_PARAM_PROFILE/JOB_PARAM_REGION and a fallback branch as well.
  • Item 9 — undeclared: the description says "behaviours that still exist were moved to the runner's tests rather than dropped". But test_neither_due_asks_nobody_to_spend and test_a_transcripts_only_wake_does_not_file_its_failure_as_a_snapshot are deleted, and both behaviours still exist in _run_once.

What this change ships

Inventory (9 items) — 5 justified

Intent: make the scheduled backup show up on the Backup row and dedupe against a manual click. This is a FIX.

  1. A nightly backup now shows as a run on the Backup row — justified
  2. A nightly and a manual click for one account no longer both upload — justified
  3. A manual click during a nightly adopts the scheduled run and its gates — undeclared
  4. The scheduled run falls back to the profile/region the loop verified when the cache has expired — undeclared
  5. A withdrawn grant or redaction refusal is audited withheld and no longer backs off — undeclared, oversized
  6. The unattended SEL trail and shared-drive notice move from the loop into the runner — justified
  7. A drive lookup that raises still backs off per due kind; a cancel is audited — justified
  8. New caller job param key — justified
  9. Two loop pins dropped although their behaviour remains — undeclared

[FIRST-PRINCIPLES-REVIEWED] 56712d6

Description read

[DESCRIPTION-READ] 2cd594131feafb128a770004dc8ca9c538f72934c51d8e30d589efe86818cf9a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 56712d67fba7d2f2c3b47aff8771661c90217de6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that docs/system-specs/modules/aws-control.md:1548 is updated: it still says each kind runs in hooks._push_nightly, which this PR deletes.

[DESIGN-REVIEWED] 56712d6

Description read

[DESCRIPTION-READ] 90e2f4cecc551640eaa705703ab021ec502ee825e2faf6e82e4050d0e0eb21f5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 56712d67fba7d2f2c3b47aff8771661c90217de6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 56712d6

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 56712d67fba7d2f2c3b47aff8771661c90217de6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 56712d6

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from b033c5e to b25579f Compare October 8, 2026 03:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both blocking findings are addressed at b25579f47211df7683dc0f6b39a6d4aba8662b94.

No-drive / no-connection became a recorded failure (Design Review): fixed. In the runner, a scheduled run that finds no working connection or no drive yet now returns as a clean skip — it records no record_nightly_failure and does not fail the SDK run — matching the previous "skip absent drives rather than creating resources" behaviour. So an account with the schedule on but no drive created yet no longer gets a FAILED run or an accruing backoff every night, and a drive created later is not held back. An owner-triggered run still raises (a person is present). Pinned by test_a_scheduled_run_with_no_drive_yet_does_not_feed_the_backoff and test_a_scheduled_run_with_no_connection_does_not_feed_the_backoff.

Dropped shared-drive notice and unchanged/succeeded audit (both findings): restored. The backup_shared_drive observation and the invoked / succeeded / unchanged / failed / cancelled SEL records are back, now emitted by the run's runner for the scheduled path (the one path with no HTTP handler to supply them). The unchanged-vs-succeeded split is driven by the run record's uploaded field, so an unchanged night is not filed as a push. Pinned by test_a_succeeded_run_audits_invoked_then_succeeded, test_an_unchanged_run_audits_unchanged_not_succeeded, test_another_install_on_the_drive_is_audited_and_the_run_proceeds, and test_an_owner_run_emits_no_nightly_audit.

The description's compatibility section no longer claims "removes nothing" and now states that these records move into the runner, and docs/system-specs/modules/aws-control.md is updated to describe the scheduled backup claiming a Job SDK run and the runner carrying that audit.

@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from b25579f to a6bf55a Compare October 8, 2026 03:27
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Addressed at a6bf55a6810e593cf57afe5efac62306b590185a.

Tests reached an unstubbed AWS LIST (GPT 6.1, blocking): fixed. The scheduled runner tests that drive work() — the scheduled-caller, failure-backoff, success-clear, and teardown cases — now stub other_install_ids to [], so none of them reach _install_folders / aws s3api list-objects-v2. The explicit shared-drive audit mocks are kept where they assert the notice.

A no-drive wake left a done run that cleared lastFailed (Design, First Principles, Opus): fixed at the root. The drive check moved back into _run_once, before anything is claimed: a wake with no drive (or no resolvable account) now claims no SDK run at all, exactly as the loop behaved before this change — so no done run is written on the Backup row and nothing clears an owner's earlier failure. The runner now simply raises for both callers if it is ever reached with no drive (only possible via the generic _jobs entry point, which is an honest failure there). Pinned by a loop test that a no-drive wake claims no run.

The one-paid-LIST pin was deleted (First Principles): restored, pointed at backup, asserting other_install_ids makes exactly one _install_folders call against the snapshot prefix.

The shared-drive LIST ran for every due kind (Opus): fixed. It now runs once, on the snapshot kind only, so a both-due wake pays one LIST (not two) and a sessions-only wake pays none.

The description and docs/system-specs/modules/aws-control.md are updated to say the drive is checked before claiming and that the runner carries the unattended audit. The remaining red lanes — a chat.spec.ts streaming spec and an owner-only-boot permissions check — are pre-existing flakes on surfaces this change does not touch.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Security Scope Review (fork) — 🔴 BLOCK (blocking)

Which legitimate operations does a6bf55a6810e593cf57afe5efac62306b590185a newly refuse? Model proposes, scripts/deny_diff.py decides — updated in place on each push. A script-confirmed regression blocks PR readiness.

no leg reported, and this was not the nothing-to-adjudicate short circuit

No [SCOPE-REVIEWED] marker for this head was produced, so the model's text is withheld: it cannot be shown to describe this revision. See the job logs.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from a6bf55a to ce2fe0a Compare October 10, 2026 00:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from ce2fe0a to 99cbf81 Compare October 10, 2026 00:50
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 10, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

This routes the scheduled AWS backup through the same sdk.start_async(kind, dedupe_key=account, ...) call as the manual button. The two then dedupe, and the Backup row shows the nightly run. That does address the root cause: the nightly bypassed the Job SDK. I am holding for three reasons. The PR rewrites the unattended, off-host, credentialed backup path (hooks.py -218/+111, backup.py +161/-36) and moves where the consent and authorization re-checks run, which falls under the credential/data path the brief reserves for a maintainer. The diff is also large (+879/-967). And two current-head findings on 99cbf81 have no disposition. First, the Design watch: because both callers share one dedupe key, an owner's "Back up now" click during a nightly adopts the scheduled run. It then runs as CALLER_SCHEDULED, can be refused by the scheduled-only re-check in backup_parts/uploads.py, and that refusal feeds record_nightly_failure. The owner sees a failure they did not cause and the backoff grows. The spec should say which caller an adopted run keeps, and a test should pin it. Second, the Opus note at backup.py:1911: resolve_account_profile_cached can expire past _PROBE_TTL_SECS between _run_once's checks and start_async, so a nightly that was just verified gets audited as failed. Refresh the accounts right before start_async, or let the scheduled path fall back to the profile it verified. All checks are green now, including Sensitive Change Review. I read the description, the lane verdicts and the author's dispositions. — crew-pr-reviewer

@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from 99cbf81 to 7fb1f30 Compare October 10, 2026 08:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 10, 2026
@chenmingwei23

chenmingwei23 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and pushed 7fb1f30982 addressing both current-head findings.

1. Adopted-run caller. Because a scheduled and a manual backup of one account share one (kind, dedupe_key), an owner's "Back up now" click during a nightly adopts the in-flight scheduled run: the SDK returns the first caller's run and start_async does not merge params, so the adopted run keeps CALLER_SCHEDULED and still runs the scheduled-only re-checks. The rule that keeps that honest for the owner is that a scheduled-only precondition the owner controls — the unattended grant withdrawn, or scheduled transcripts blocked because redaction is on — now raises ScheduledUploadWithheld, which the runner records as withheld and does not feed into record_nightly_failure. So an owner whose adopting click surfaces one of those gates no longer sees the nightly backoff grow for a run they chose to stop. An unregistered-kind fail-closed stays a fault that backs off. Stated in the spec and the runner docstring; pinned by test_a_withheld_scheduled_run_does_not_feed_the_backoff (with its converse test_a_plain_scheduled_fault_still_feeds_the_backoff).

2. Stale profile before start_async. resolve_account_profile_cached serves only from the warm snapshot and returns None once it is past _PROBE_TTL_SECS, and start_async is fire-and-forget, so the runner thread resolves later than the loop's live probe — a gap past the TTL audited a just-verified nightly as failed. The loop now carries the (profile, region) it freshly probed under JOB_PARAM_PROFILE / JOB_PARAM_REGION, and a scheduled run falls back to that pair when the cache has expired. The live sts:GetCallerIdentity re-check still guards the account, and the bucket is still re-discovered per run; the owner path is unchanged. Pinned by TestScheduledPathFallsBackToTheVerifiedProfile (test_a_scheduled_run_uses_the_verified_profile_when_the_cache_expired, plus the owner-path and warm-cache cases).

The diff size and the credentialed-path concern are the maintainer's call to make.


Unrelated CI note: the Build Wheel / Build Desktop red on this head is the stale src/kiro_crew/docs/ui-index.generated.json inherited from main (gen:ui --check), not from this change — the generated index on this branch is byte-identical to main and this PR touches no frontend. Tracked in #18749 and regenerated on main by #18753; it clears here on the next rebase once that lands.

The advisory First Principles lane reports a premise/scope concern (the diff size and the credentialed-path rewrite). That is the same maintainer-reserved question raised above, so it is left for the maintainer rather than coded around — the change does what the issue asks; whether it should take this shape is the maintainer's call. The GPT 6.1 lane raised one blocking test-only finding — a determinism race in a helper that let a second same-account run adopt the first before its slot was released — which is now fixed by having the helper wait for the run to leave the live set; no product behaviour changed.


Rebased onto current main (d32c08492176e331b68058933350d491572a02b7) and pushed 24f493b587. The earlier Build Wheel / Build Desktop / E2E / Frontend Tests reds were the stale ui-index.generated.json inherited from main; main regenerated it, so the rebase clears them (and Coverage Gate, which only failed closed on that frontend red — the backend tests were green throughout). The GPT review raised one blocking, test-only finding: _await_terminal returned on the terminal record before the Job SDK released the run's dedupe-key slot, so a second start for the same account in that window could adopt the finished run instead of starting a fresh one. Fixed by having _await_terminal also wait for the run to leave the SDK live set; pinned by test_a_plain_scheduled_fault_still_feeds_the_backoff and the other sequential-start tests in test_aws_control_backup_job.py. No product behaviour changed.


Pushed b20a153cf5. The First Principles review raised one real finding: the rewrite had dropped the per-kind cancelled SEL audit on a setup (drive-lookup) cancel. Restored it — a cancel during the drive lookup now audits cancelled per due kind (teardown stays visible) without backing off; pinned by test_a_cancel_during_the_lookup_audits_cancelled_but_does_not_back_off. The premise ("why not a smaller change") is answered in the description: the Job SDK routing is the fix (a smaller patch leaves double-upload and an invisible run), credential/consent re-checks are preserved, and the one-LIST notice pin was moved to test_aws_control_backup_job.py, not deleted.


Pushed 56712d67fb (rebased onto current main). Addressed First Principles' follow-up: the setup failure/cancel audit and the backoff record now route through the one shared helper — backup._nightly_audit under backup._nightly_subject plus the existing record_nightly_failure — instead of a second inline SEL block in hooks.py, so there is one caller="aws-control-nightly" audit site again, not three.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from 7fb1f30 to 24f493b Compare October 10, 2026 09:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from 24f493b to b20a153 Compare October 10, 2026 16:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026
The nightly loop ran the backup directly on a thread, so a scheduled run
claimed no Job SDK run: it shared no (kind, dedupe_key) with a manual click
(both could upload, both paid) and the Backup row read idle while a nightly
ran. _run_once now fires start_async(kind, dedupe_key=account,
params={caller: scheduled}) per due kind -- the same call the manual route
makes -- so the two callers dedupe and the row sees the run. The one
registered runner reads the caller from params and, for a scheduled run,
records a failure into the backoff due_for_nightly reads (minus teardown), so
a deterministic nightly fault still is not retried every half hour.

Two refinements keep the shared-caller path honest:

Adopted-run attribution. Because both callers share one (kind, dedupe_key),
an owner's "Back up now" click during a nightly adopts the in-flight
scheduled run -- the SDK returns the first caller's run and params do not
merge, so the adopted run stays CALLER_SCHEDULED and runs the scheduled-only
re-checks. A scheduled-only precondition the owner controls (the unattended
grant withdrawn, or scheduled transcripts blocked because redaction is on)
now raises ScheduledUploadWithheld; the runner audits it withheld and does
NOT feed the nightly backoff. Without this an owner who clicked during a
nightly and surfaced one of those gates would see a failure they did not
cause and watch the backoff grow. An unregistered-kind fail-closed stays a
fault that backs off.

Stale profile fallback. resolve_account_profile_cached serves only from the
warm snapshot and returns None once it is past _PROBE_TTL_SECS, and
start_async is fire-and-forget, so the runner resolves later than the loop's
live probe; a gap past the TTL audited a just-verified nightly as failed. The
loop now names the (profile, region) it freshly probed under
JOB_PARAM_PROFILE / JOB_PARAM_REGION, and a scheduled run falls back to that
pair when the cache has expired. The live sts:GetCallerIdentity re-check still
guards the account, and the bucket is still re-discovered per run.

Fixes kirodotdev#7783

- Restore the per-kind "cancelled" SEL audit on a setup (drive-lookup) cancel
  (dropped in the rewrite): teardown still leaves a trail it was interrupted,
  without backing off. The loop-side failure and cancel audits route through
  backup._nightly_audit / _nightly_subject (one audit helper, not a second
  inline copy). Pinned by
  test_a_cancel_during_the_lookup_audits_cancelled_but_does_not_back_off.
@chenmingwei23
chenmingwei23 force-pushed the fix/7783-nightly-backup-through-job-sdk branch from b20a153 to 56712d6 Compare October 10, 2026 17:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at this head. Both named findings are fixed: the scheduled path now falls back to the profile it just verified (no TTL race before start_async), and an adopted run's refusal no longer feeds the nightly backoff. Credential and consent checks are kept, and _authorize_upload still does a live STS check. Small follow-up: docs/system-specs/modules/aws-control.md still mentions hooks._push_nightly, which this PR removes.

Posted by an AI assistant at bolichen97's request, after a read-only review of this head.

@bolichen97
bolichen97 merged commit 705f541 into kirodotdev:main Oct 11, 2026
150 of 152 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws-control: the nightly backup loop bypasses the Job SDK, so it can double-upload and is invisible to the row

3 participants