Repository navigation
fix(aws-control): skip the sessions upload lock only for an attended withheld run - #12980
Conversation
The sessions upload enters `_upload_lock()` unconditionally, including on the path where the lock can order nothing. The recheck it exists for reads `layer_b and not sessions_layer_b_enabled(account)`, so when `layer_b` is False the condition short-circuits on its first operand and the refusal it guards cannot fire. On that path the state lock is held across the PUT while guarding nothing, and withholding is the default, so that is the common path. The lock file is `_state_path()`'s sidecar, `backup.json` in the app data directory -- one path for every account, not one per account. Every state writer reaches it, so one account's upload puts every other account's state write behind it for up to `_STATE_LOCK_TIMEOUT_SECS`, including the toggles an operator uses to turn the nightly off or change retention. Taking no lock on the withheld path also serves `_authorize_upload`'s own invariant more directly than holding one does: with no blocking acquisition in the block, the authorization and the PUT are adjacent. The permitted path is unchanged and still holds the lock across the authorization, the recheck and the PUT, so the revocation ordering against `set_sessions_layer_b` is untouched. Concurrency does not rest on this lock: `_stamp()` carries entropy because a manual run can race the nightly loop into the same second, and `_record_run` compare-and-sets on `(process, sequence)` for same-kind runs that reach the file lock out of order. Tests: `test_a_withheld_upload_does_not_hold_the_setter_lock` probes the sidecar file lock from another thread inside `put_file` on a withheld run and asserts it is free. It fails with the unconditional `with`. Its permitted-path sibling, `test_the_upload_holds_the_setter_lock_across_the_permission_read`, still asserts the lock is held, so the two pin the scope exactly. Closes #12548
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The skip's safety rests on Watch
[DESIGN-REVIEWED] 5ee11a4 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Seven undeclared riders on a one-lock fix — four match commits already on the base tip (#12944, #12951), so the diff base looks stale; rebase. Not justified as shipped
What this change shipsInventory (9 items) — 2 justifiedIntent: stop an owner's Layer-B-off "back up now" from stalling every other account's backup-state writes — a FIX (issue #12548).
[FIRST-PRINCIPLES-REVIEWED] 5ee11a4 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…s runs The lock skip is keyed on the attended owner rather than on layer_b. The crew display half rides on every sessions run, and for a scheduled caller that half is authorized by the unattended grant, which _authorize_upload re-reads inside the gate and set_nightly_sessions writes under the same sidecar lock. Skipping the lock whenever layer_b was False left that read unordered against the PUT, so a revocation landing mid-block could ship the transcript after withdrawal, with no recovery available. Only an owner-initiated withheld run has no such read: both scheduled-only re-reads are gated on CALLER_SCHEDULED, the Layer B recheck short-circuits, and what remains lives outside this module's state file. That run keeps the cross-account stall fix; every other shape holds the lock. Refs #12548
|
Disposition for the two advisory CONCERNS on First Principles: the seven "undeclared riders" are not in this diffThis branch is two commits and three files. Measured locally against the true The two commits the review names as the duplicated sources are on
Declining the rebase. Per this repo's own guidance a rebase is for a red it can The inventory's items 1 and 2 are the whole change, and the description declares Design: the predicate's dependence on
|
…withheld run (kirodotdev#12980) * fix(aws-control): take the sessions upload lock only when Layer B rides The sessions upload enters `_upload_lock()` unconditionally, including on the path where the lock can order nothing. The recheck it exists for reads `layer_b and not sessions_layer_b_enabled(account)`, so when `layer_b` is False the condition short-circuits on its first operand and the refusal it guards cannot fire. On that path the state lock is held across the PUT while guarding nothing, and withholding is the default, so that is the common path. The lock file is `_state_path()`'s sidecar, `backup.json` in the app data directory -- one path for every account, not one per account. Every state writer reaches it, so one account's upload puts every other account's state write behind it for up to `_STATE_LOCK_TIMEOUT_SECS`, including the toggles an operator uses to turn the nightly off or change retention. Taking no lock on the withheld path also serves `_authorize_upload`'s own invariant more directly than holding one does: with no blocking acquisition in the block, the authorization and the PUT are adjacent. The permitted path is unchanged and still holds the lock across the authorization, the recheck and the PUT, so the revocation ordering against `set_sessions_layer_b` is untouched. Concurrency does not rest on this lock: `_stamp()` carries entropy because a manual run can race the nightly loop into the same second, and `_record_run` compare-and-sets on `(process, sequence)` for same-kind runs that reach the file lock out of order. Tests: `test_a_withheld_upload_does_not_hold_the_setter_lock` probes the sidecar file lock from another thread inside `put_file` on a withheld run and asserts it is free. It fails with the unconditional `with`. Its permitted-path sibling, `test_the_upload_holds_the_setter_lock_across_the_permission_read`, still asserts the lock is held, so the two pin the scope exactly. Closes kirodotdev#12548 * fix(aws-control): keep the upload lock for scheduled withheld sessions runs The lock skip is keyed on the attended owner rather than on layer_b. The crew display half rides on every sessions run, and for a scheduled caller that half is authorized by the unattended grant, which _authorize_upload re-reads inside the gate and set_nightly_sessions writes under the same sidecar lock. Skipping the lock whenever layer_b was False left that read unordered against the PUT, so a revocation landing mid-block could ship the transcript after withdrawal, with no recovery available. Only an owner-initiated withheld run has no such read: both scheduled-only re-reads are gated on CALLER_SCHEDULED, the Layer B recheck short-circuits, and what remains lives outside this module's state file. That run keeps the cross-account stall fix; every other shape holds the lock. Refs kirodotdev#12548
What is the problem?
run_sessions_backupuploads the session archive insidewith _upload_lock():,entered on every run. That lock is the sidecar file lock on
_state_path(), whichis a single
backup.jsonfor every account, not one file per account. It isheld across
_authorize_uploadand the wholestorage.put_file, a PUT allowed_PUSH_TIMEOUT_SECS.For one shape of run the hold orders nothing at all. The only thing the lock
sequences inside that block is a re-read of a permission that a setter on the same
lock can change. An owner-initiated run with Layer B withheld has no such re-read:
the recheck it would order is
layer_b and not sessions_layer_b_enabled(account),which stops at its first operand, and the remaining checks (
is_app_enabled,aws_consent, STS) are not stored in this module's state file. So that run holds anexclusive cross-account lock for the length of its upload and gains nothing by it.
While it is held, every other account's state writer waits up to
_STATE_LOCK_TIMEOUT_SECS:_record_run,set_sessions_layer_b,set_retention_keep, and the nightly loop.Why this issue matters to the user
An operator clicking "back up now" on one account with Layer B off can stall the
backup state writes of every other account for as long as that upload runs. The
toggles they reach for next, Layer B and retention, are among the writers that
wait. Nothing about the operation they started needs that exclusivity.
How our fix solves it
The lock acquisition becomes conditional, and the condition is the caller, not
the Layer B decision:
Keying it on
layer_balone is wrong, and the chain is worth stating because it isthe reason this PR is narrower than the issue asked for. The crew display half of
the archive rides on every sessions run, permitted or withheld
(
_add_tree(tar, crew_sessions, "crew")is unconditional). For a scheduled callerthat half is authorized by the unattended grant, which
_authorize_uploadre-readsinside the guarded block, for
CALLER_SCHEDULEDalone. That grant's setter,set_nightly_sessions, writes through_locked_state_update->_state_lock,which takes the very same sidecar file lock. So a scheduled withheld run does
have a permission that a setter can withdraw mid-block, and dropping the lock there
would leave the revocation unordered against the PUT: the transcript would ship
after the owner withdrew the grant, and an object already on S3 cannot be taken
back.
So the skip is scoped to the one shape with no such read. Both scheduled-only
re-reads are gated on
CALLER_SCHEDULED; the Layer B recheck short-circuits; whatis left lives outside this module's state file. The predicate is written in the
positive and tested in the negative, so a caller nobody anticipated holds the lock
rather than skipping it.
The permitted path is untouched: it still takes the lock before
_authorize_uploadand holds it across the recheck and the PUT, so the revocation guarantee that gate
exists for is unchanged. The scheduled path is also untouched.
Prose in the same commit is corrected where it stated the old, wrong reason: the
_upload_lockdocstring (which said the lock is taken "on the PERMITTED pathalone"), the call-site comment (whose list of what
_authorize_uploadreads stoppedat STS, which is what made the second re-read easy to miss), and the aws-control
module spec.
What tests we did
test_aws_control_backup.py::TestSessionsArchiveLayerBGatenow pins all threedirections, each probing the lock file from another thread inside
put_fileon afresh descriptor without blocking, so the answer is the contention a real concurrent
writer would meet:
test_the_authorization_runs_inside_the_lock_it_will_upload_under-- permitted runholds it (pre-existing).
test_an_attended_withheld_upload_does_not_hold_the_setter_lock-- owner withheldrun leaves it free.
test_a_scheduled_withheld_upload_holds_the_setter_lock-- scheduled withheld runstill holds it.
Both new tests were confirmed discriminating by restoring the predicate they reject
and watching only that test fail on its own assertion: the owner test fails with
"the withheld upload held the state lock" under the unconditional
with, and thescheduled test fails with "the scheduled withheld upload left the state lock free"
under a
layer_b-only predicate.324 passed across
test/test_aws_control_backup.py,test/test_aws_control_nightly_sessions.pyandtest/test_aws_control_backup_job.py,each run with explicit
-n0and an isolatedKIROCREW_HOME. black, isort, flake8,mypy, the comment-history gate against the base sha, and
check_loop_bound_locksare clean.
Any other suggestions on the work
The cross-account shape of this lock is the underlying cost: one
backup.jsonandone sidecar lock for every account means any per-account upload can stall every
other account's writes. Narrowing the hold, as here, reduces how often that happens
without changing it. Per-account state files would remove the class, and that is a
data-layout change well outside a fix.
Worth a reviewer's eye: the predicate names
CALLER_OWNERexplicitly. The two livecallers are
CALLER_OWNER(routes.py) andCALLER_SCHEDULED(hooks.py), sotoday the two spellings are equivalent; the fail-closed direction was chosen so a
third caller added later holds the lock by default.
Pattern harvest
Rule candidate: review-prompt
Pattern: a conditional lock acquisition whose predicate is derived from one of
the permissions re-read inside the guarded block. The guarded block here re-read two
permissions under the same lock, on different caller conditions, and a predicate
built from the more visible one silently dropped the ordering for the other. The
review question that catches it: for a lock skipped on some condition, enumerate
every state read inside the block and every setter that writes what those reads see,
and check the predicate covers all of them, not just the one the surrounding comment
names.
Fixes #12548