Skip to content

fix(aws-control): skip the sessions upload lock only for an attended withheld run - #12980

Merged
iamwhatever merged 2 commits into
mainfrom
fix/aws-control-upload-lock-12548
Sep 23, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/aws-control-upload-lock-12548

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What is the problem?

run_sessions_backup uploads the session archive inside with _upload_lock():,
entered on every run. That lock is the sidecar file lock on _state_path(), which
is a single backup.json for every account, not one file per account. It is
held across _authorize_upload and the whole storage.put_file, a PUT allowed
_PUSH_TIMEOUT_SECS.

For one shape of run the hold orders nothing at all. The only thing the lock
sequences inside that block is a re-read of a permission that a setter on the same
lock can change. An owner-initiated run with Layer B withheld has no such re-read:
the recheck it would order is layer_b and not sessions_layer_b_enabled(account),
which stops at its first operand, and the remaining checks (is_app_enabled,
aws_consent, STS) are not stored in this module's state file. So that run holds an
exclusive cross-account lock for the length of its upload and gains nothing by it.

While it is held, every other account's state writer waits up to
_STATE_LOCK_TIMEOUT_SECS: _record_run, set_sessions_layer_b,
set_retention_keep, and the nightly loop.

Why this issue matters to the user

An operator clicking "back up now" on one account with Layer B off can stall the
backup state writes of every other account for as long as that upload runs. The
toggles they reach for next, Layer B and retention, are among the writers that
wait. Nothing about the operation they started needs that exclusivity.

How our fix solves it

The lock acquisition becomes conditional, and the condition is the caller, not
the Layer B decision:

withheld_and_attended = not layer_b and caller == CALLER_OWNER
with contextlib.nullcontext() if withheld_and_attended else _upload_lock():

Keying it on layer_b alone is wrong, and the chain is worth stating because it is
the reason this PR is narrower than the issue asked for. The crew display half of
the archive rides on every sessions run, permitted or withheld
(_add_tree(tar, crew_sessions, "crew") is unconditional). For a scheduled caller
that half is authorized by the unattended grant, which _authorize_upload re-reads
inside the guarded block, for CALLER_SCHEDULED alone. That grant's setter,
set_nightly_sessions, writes through _locked_state_update -> _state_lock,
which takes the very same sidecar file lock. So a scheduled withheld run does
have a permission that a setter can withdraw mid-block, and dropping the lock there
would leave the revocation unordered against the PUT: the transcript would ship
after the owner withdrew the grant, and an object already on S3 cannot be taken
back.

So the skip is scoped to the one shape with no such read. Both scheduled-only
re-reads are gated on CALLER_SCHEDULED; the Layer B recheck short-circuits; what
is left lives outside this module's state file. The predicate is written in the
positive and tested in the negative, so a caller nobody anticipated holds the lock
rather than skipping it.

The permitted path is untouched: it still takes the lock before _authorize_upload
and holds it across the recheck and the PUT, so the revocation guarantee that gate
exists for is unchanged. The scheduled path is also untouched.

Prose in the same commit is corrected where it stated the old, wrong reason: the
_upload_lock docstring (which said the lock is taken "on the PERMITTED path
alone"), the call-site comment (whose list of what _authorize_upload reads stopped
at STS, which is what made the second re-read easy to miss), and the aws-control
module spec.

What tests we did

test_aws_control_backup.py::TestSessionsArchiveLayerBGate now pins all three
directions, each probing the lock file from another thread inside put_file on a
fresh descriptor without blocking, so the answer is the contention a real concurrent
writer would meet:

  • test_the_authorization_runs_inside_the_lock_it_will_upload_under -- permitted run
    holds it (pre-existing).
  • test_an_attended_withheld_upload_does_not_hold_the_setter_lock -- owner withheld
    run leaves it free.
  • test_a_scheduled_withheld_upload_holds_the_setter_lock -- scheduled withheld run
    still holds it.

Both new tests were confirmed discriminating by restoring the predicate they reject
and watching only that test fail on its own assertion: the owner test fails with
"the withheld upload held the state lock" under the unconditional with, and the
scheduled test fails with "the scheduled withheld upload left the state lock free"
under a layer_b-only predicate.

324 passed across test/test_aws_control_backup.py,
test/test_aws_control_nightly_sessions.py and test/test_aws_control_backup_job.py,
each run with explicit -n0 and an isolated KIROCREW_HOME. black, isort, flake8,
mypy, the comment-history gate against the base sha, and check_loop_bound_locks
are clean.

Any other suggestions on the work

The cross-account shape of this lock is the underlying cost: one backup.json and
one sidecar lock for every account means any per-account upload can stall every
other account's writes. Narrowing the hold, as here, reduces how often that happens
without changing it. Per-account state files would remove the class, and that is a
data-layout change well outside a fix.

Worth a reviewer's eye: the predicate names CALLER_OWNER explicitly. The two live
callers are CALLER_OWNER (routes.py) and CALLER_SCHEDULED (hooks.py), so
today the two spellings are equivalent; the fail-closed direction was chosen so a
third caller added later holds the lock by default.

Pattern harvest

Rule candidate: review-prompt

Pattern: a conditional lock acquisition whose predicate is derived from one of
the permissions re-read inside the guarded block. The guarded block here re-read two
permissions under the same lock, on different caller conditions, and a predicate
built from the more visible one silently dropped the ordering for the other. The
review question that catches it: for a lock skipped on some condition, enumerate
every state read inside the block and every setter that writes what those reads see,
and check the predicate covers all of them, not just the one the surrounding comment
names.

Fixes #12548

The sessions upload enters `_upload_lock()` unconditionally, including on the
path where the lock can order nothing. The recheck it exists for reads
`layer_b and not sessions_layer_b_enabled(account)`, so when `layer_b` is False
the condition short-circuits on its first operand and the refusal it guards
cannot fire. On that path the state lock is held across the PUT while guarding
nothing, and withholding is the default, so that is the common path.

The lock file is `_state_path()`'s sidecar, `backup.json` in the app data
directory -- one path for every account, not one per account. Every state
writer reaches it, so one account's upload puts every other account's state
write behind it for up to `_STATE_LOCK_TIMEOUT_SECS`, including the toggles an
operator uses to turn the nightly off or change retention.

Taking no lock on the withheld path also serves `_authorize_upload`'s own
invariant more directly than holding one does: with no blocking acquisition in
the block, the authorization and the PUT are adjacent. The permitted path is
unchanged and still holds the lock across the authorization, the recheck and the
PUT, so the revocation ordering against `set_sessions_layer_b` is untouched.

Concurrency does not rest on this lock: `_stamp()` carries entropy because a
manual run can race the nightly loop into the same second, and `_record_run`
compare-and-sets on `(process, sequence)` for same-kind runs that reach the file
lock out of order.

Tests: `test_a_withheld_upload_does_not_hold_the_setter_lock` probes the sidecar
file lock from another thread inside `put_file` on a withheld run and asserts it
is free. It fails with the unconditional `with`. Its permitted-path sibling,
`test_the_upload_holds_the_setter_lock_across_the_permission_read`, still asserts
the lock is held, so the two pin the scope exactly.

Closes #12548
@chenmingwei23
chenmingwei23 requested review from a team and patrigao September 23, 2026 03:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 5ee11a4f225b0dc142363dbc17a597ae70972e57 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The skip's safety rests on _authorize_upload's per-caller read set, which no test pins — both new tests mock that gate out entirely.

Watch

  • withheld_and_attended = not layer_b and caller == CALLER_OWNER is valid only while _authorize_upload reads nothing from this module's state file for an owner caller. Both new tests mock.patch.object(backup, "_authorize_upload"), so the invariant lives in prose alone: a later change adding an owner-path state re-read inside that gate would silently ship an upload unordered against its setter — the same unrecoverable class this PR fixes for the scheduled caller, and the branch's own first commit (layer_b-only predicate) shows how easily this coupling is gotten wrong.
    Clears when: a test exercises the real _authorize_upload on the owner-withheld path and asserts it performs no read of _state_path() (or an equivalent structural pin tying the lock-skip predicate to the gate's actual read set).

[DESIGN-REVIEWED] 5ee11a4

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 5ee11a4f225b0dc142363dbc17a597ae70972e57 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Seven undeclared riders on a one-lock fix — four match commits already on the base tip (#12944, #12951), so the diff base looks stale; rebase.

Not justified as shipped

  • Item 3 — duplicate of src/kiro_crew/acp/runtime.py on base: commit 29d7bc6 "fix(acp): … unstarve session-start gate (fix(acp): recycle idle shared runtimes; unstarve session-start gate #12944)" is the merge target's ancestor.
    Subtraction: rebase onto current main; drop the _COLLECTOR_PERMIT_HEADROOM hunks.
    Clears when: after a push or rebase the diff carries only the aws-control change.
  • Item 4 — duplicate of the same base commit 29d7bc6 ("recycle idle shared runtimes").
    Subtraction: drop the reap_idle_stale_bg_runtime/park_only hunks in that rebase.
    Clears when: same rebase.
  • Item 5 — undeclared: support surface for item 6 the description never mentions.
    Clears when: item 6 clears.
  • Item 6 — undeclared, inherited: its only provenance is the diff's own "observed on an operator host"; the description says nothing about loops.
    Subtraction: defer SESSION_START_FAILURE_REASON, consecutive_start_failures and the chat_runner hooks to their own PR.
    Clears when: the rebase shows it already on base, or it ships separately with its incident or issue linked.
  • Item 7 — undeclared, inherited: "165,975 files … on an operator host" exists only in the diff's comments.
    Subtraction: defer prune_legacy_member_pid_bindings likewise.
    Clears when: same as item 6.
  • Item 8 — duplicate of the base tip itself: ec99705 "docs(skills): catch a session that dies mid-flow (docs(skills): catch a browser session that dies mid-flow #12951)".
    Subtraction: drop both SKILL.md hunks in the rebase.
    Clears when: same rebase.
  • Item 9 — duplicate of ec99705, the web-browse half of item 8.
    Clears when: same rebase.

What this change ships

Inventory (9 items) — 2 justified

Intent: stop an owner's Layer-B-off "back up now" from stalling every other account's backup-state writes — a FIX (issue #12548).

  1. Owner-clicked withheld backup no longer blocks other accounts' state writes — justified
  2. Lock docstring, call-site comment and aws-control spec corrected in the same commit — justified
  3. Session-start gate reserves one permit from timed-out collectors — undeclared, duplicate of base commit 29d7bc6
  4. Idle-and-stale shared runtime retired by the periodic sweep — undeclared, duplicate of base commit 29d7bc6
  5. ACP timeouts carry a session_start_failed tag — undeclared, rides along with item 6
  6. Nudge loops defer after 3 failed session starts, stand down at 5, with new persisted state and stop reason — undeclared, inherited provenance
  7. Legacy member-memory pid-binding files pruned by periodic cleanup — undeclared, inherited provenance
  8. browser-auth skill gains token-mismatch, one-tab and mid-flow session-death guidance — undeclared, duplicate of base commit ec99705
  9. web-browse skill gains the one-tab attach caveat — undeclared, duplicate of base commit ec99705

[FIRST-PRINCIPLES-REVIEWED] 5ee11a4

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 5ee11a4f225b0dc142363dbc17a597ae70972e57 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5ee11a4

Verdict parsed from the review's SHA-scoped output markers for commit 5ee11a4f225b0dc142363dbc17a597ae70972e57.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5ee11a4f225b0dc142363dbc17a597ae70972e57: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 5ee11a4f225b0dc142363dbc17a597ae70972e57 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5ee11a4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5ee11a4f225b0dc142363dbc17a597ae70972e57: <one-sentence reason>

…s runs

The lock skip is keyed on the attended owner rather than on layer_b. The crew
display half rides on every sessions run, and for a scheduled caller that half is
authorized by the unattended grant, which _authorize_upload re-reads inside the
gate and set_nightly_sessions writes under the same sidecar lock. Skipping the
lock whenever layer_b was False left that read unordered against the PUT, so a
revocation landing mid-block could ship the transcript after withdrawal, with no
recovery available.

Only an owner-initiated withheld run has no such read: both scheduled-only
re-reads are gated on CALLER_SCHEDULED, the Layer B recheck short-circuits, and
what remains lives outside this module's state file. That run keeps the
cross-account stall fix; every other shape holds the lock.

Refs #12548
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@chenmingwei23 chenmingwei23 changed the title fix(aws-control): take the sessions upload lock only when Layer B rides fix(aws-control): skip the sessions upload lock only for an attended withheld run Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for the two advisory CONCERNS on 5ee11a4f225b0dc142363dbc17a597ae70972e57.

First Principles: the seven "undeclared riders" are not in this diff

This branch is two commits and three files. Measured locally against the true
merge base rather than against a diff of the moving main tip:

$ git log --oneline origin/main..HEAD
5ee11a4f22 fix(aws-control): keep the upload lock for scheduled withheld sessions runs
51e660c412 fix(aws-control): take the sessions upload lock only when Layer B rides

$ git merge-base origin/main HEAD
41082f363a1908220029720403cefcb27537b791

$ git diff --stat 41082f363a1908220029720403cefcb27537b791..HEAD
 docs/system-specs/modules/aws-control.md           |  52 ++++--
 src/kiro_crew/apps/builtins/aws_control/backend/backup.py | 104 ++++++++-----
 test/test_aws_control_backup.py                    | 128 +++++++++++++++++
 3 files changed, 247 insertions(+), 37 deletions(-)

The two commits the review names as the duplicated sources are on main, and are
in neither this branch nor its base:

29d7bc681 (#12944): ancestor of origin/main YES | of this branch no | of the base no
ec99705be (#12951): ancestor of origin/main YES | of this branch no | of the base no

git diff --name-only <base>..HEAD filtered for acp/runtime, SKILL.md,
chat_runner and member_pid returns nothing: items 3 through 9 touch files this
branch does not modify. main is 25 commits ahead of this head, and those 25
include both named commits, so a diff taken against the current main tip rather
than against the merge base shows main's own newer work alongside this change.
That also explains why every one of those items is reported as already present on
base and clearing "after the rebase": a rebase makes them vanish because it moves
the base, not because it removes anything from this branch.

Declining the rebase. Per this repo's own guidance a rebase is for a red it can
fix, and there is none here: every lane on this head is green or in flight, with
the two lanes that were red on the previous head (GPT 5.6 Review, PR Hygiene)
now passing. A push re-rolls the non-deterministic review lanes, and on this head
that means re-rolling a security-class verdict that has just cleared. Paying that
to change a diff base is the wrong trade.

The inventory's items 1 and 2 are the whole change, and the description declares
both.

Design: the predicate's dependence on _authorize_upload is real, and stays prose here

The concern is accurate as stated. withheld_and_attended = not layer_b and caller == CALLER_OWNER is valid only while _authorize_upload reads nothing from this
module's state file on an owner caller, both new tests mock that gate out, and the
first commit on this branch is evidence of how easily the coupling is misread.

Verified in source for this head rather than assumed. Inside the guarded block
_authorize_upload reads is_app_enabled, aws_consent.is_granted,
aws_consent.read_grant, an STS call, and then two further reads that are each
gated on caller == CALLER_SCHEDULED: the _NIGHTLY_CONSENT_READERS grant and
scheduled_sessions_blocked_reason. Only those last two touch _state_path(), and
an owner caller reaches neither. aws_consent is a separate module with its own
store, and is_app_enabled is not this module's state.

Not adding the structural pin in this PR, for two reasons. It guards a future edit
to _authorize_upload rather than a defect in what ships here, which puts it
outside a fix PR's scope. And landing it costs a push, which re-rolls the review
lanes on a head where a security-class finding has just cleared.

What stands in its place is the invariant written at the three sites a reader of
this code reaches: the _upload_lock docstring, the call-site comment (whose
enumeration of what _authorize_upload reads is now complete, the omission that
made the coupling easy to miss in the first place), and the aws-control module
spec. A maintainer who wants the pin enforced mechanically rather than by reading
should say so and it can ship as its own change, where a reviewer can judge the
mocking it needs on its own merits.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever
iamwhatever merged commit b83100d into main Sep 23, 2026
100 of 107 checks passed
@iamwhatever
iamwhatever deleted the fix/aws-control-upload-lock-12548 branch September 23, 2026 05:34
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 23, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…withheld run (kirodotdev#12980)

* fix(aws-control): take the sessions upload lock only when Layer B rides

The sessions upload enters `_upload_lock()` unconditionally, including on the
path where the lock can order nothing. The recheck it exists for reads
`layer_b and not sessions_layer_b_enabled(account)`, so when `layer_b` is False
the condition short-circuits on its first operand and the refusal it guards
cannot fire. On that path the state lock is held across the PUT while guarding
nothing, and withholding is the default, so that is the common path.

The lock file is `_state_path()`'s sidecar, `backup.json` in the app data
directory -- one path for every account, not one per account. Every state
writer reaches it, so one account's upload puts every other account's state
write behind it for up to `_STATE_LOCK_TIMEOUT_SECS`, including the toggles an
operator uses to turn the nightly off or change retention.

Taking no lock on the withheld path also serves `_authorize_upload`'s own
invariant more directly than holding one does: with no blocking acquisition in
the block, the authorization and the PUT are adjacent. The permitted path is
unchanged and still holds the lock across the authorization, the recheck and the
PUT, so the revocation ordering against `set_sessions_layer_b` is untouched.

Concurrency does not rest on this lock: `_stamp()` carries entropy because a
manual run can race the nightly loop into the same second, and `_record_run`
compare-and-sets on `(process, sequence)` for same-kind runs that reach the file
lock out of order.

Tests: `test_a_withheld_upload_does_not_hold_the_setter_lock` probes the sidecar
file lock from another thread inside `put_file` on a withheld run and asserts it
is free. It fails with the unconditional `with`. Its permitted-path sibling,
`test_the_upload_holds_the_setter_lock_across_the_permission_read`, still asserts
the lock is held, so the two pin the scope exactly.

Closes kirodotdev#12548

* fix(aws-control): keep the upload lock for scheduled withheld sessions runs

The lock skip is keyed on the attended owner rather than on layer_b. The crew
display half rides on every sessions run, and for a scheduled caller that half is
authorized by the unattended grant, which _authorize_upload re-reads inside the
gate and set_nightly_sessions writes under the same sidecar lock. Skipping the
lock whenever layer_b was False left that read unordered against the PUT, so a
revocation landing mid-block could ship the transcript after withdrawal, with no
recovery available.

Only an owner-initiated withheld run has no such read: both scheduled-only
re-reads are gated on CALLER_SCHEDULED, the Layer B recheck short-circuits, and
what remains lives outside this module's state file. That run keeps the
cross-account stall fix; every other shape holds the lock.

Refs kirodotdev#12548
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws-control: the sessions upload holds a global state lock on the withheld path, where it guards nothing

2 participants