Skip to content

fix(aws-control): gate Layer B in the sessions archive on the operator's permission - #12178

Merged
bolichen97 merged 1 commit into
mainfrom
fix/archive-layer-b-opt-in-11559
Sep 22, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/archive-layer-b-opt-in-11559

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

run_sessions_backup tarred the kiro-cli session directory wholesale, so Layer B -- the byte-exact, unredacted model context window -- rode every sessions backup. It rode because the raw files were in the directory being tarred, not because any permission was consulted.

The file-export path requires an operator opt-in before the same payload leaves in a bundle. Of its conditions, the sessions archive already met one and met neither of the others:

condition file export sessions archive (before)
dashboard operator, not an app yes yes, already
standing permission, off by default yes absent
per-invocation opt-in yes no channel for one

The operator-identity half was already covered: both routes that can start a sessions backup are owner-gated (apps/job_routes.py requires is_owner_dashboard_request, and the app's own start route requires the drive). The per-invocation half has nowhere to live here -- JobSDK.start_async(kind, dedupe_key) carries no parameters, and the nightly path has no request at all. The standing permission was the missing condition.

Why it matters

Layer B is strictly more sensitive than the transcript it accompanies. The transcript is what was displayed, with display-time redaction applied; Layer B is what the model actually held, unredacted. It also cannot be redacted on the way out -- the thinking blocks inside it carry a provider signature over their own content, so rewriting one invalidates the conversation. That leaves exactly two options, byte-exact or absent, which is why a permission and not a filter is the right instrument.

Before this change an operator pressing the backup button could not tell, from any surface, that the archive carried unredacted model context. Nothing recorded it either, so a restore could not tell what it had.

What changed

Where the permission lives, and why not config.json. config.json is writable by any auto-approved agent shell, so a permission honoured from there is one a prompt-injected agent can grant itself: edit the key, wait for the owner to run an ordinary sessions backup, and the unredacted context uploads with no consent. An object already in a bucket cannot be recalled, so there is no recovery path. An authorization whose subject can write it is not an authorization, and this repo already states the rule -- CredentialPolicy.exempt_exact_hosts says such a value is "NEVER sourced from config.json".

So the answer lives in the app's own state document, backup.json, inside the apps/aws-control/data directory that is already registered in security._CREW_SECRET_LEAVES. That is the read+write keystone floor: the agent's file tools refuse every path in there and the shell forms are refused too. It is the same placement, and the same reasoning, as the nightly bit stored beside it, which authorizes unattended paid uploads -- and as computer_use.json, aws_service_consent.json and file_delivery_consent.json elsewhere in the tree.

The sole writer is the new owner-gated POST /api/apps/aws-control/backup/{account}/layer-b. Every route in this app goes through _guarded, which requires is_owner_dashboard_request; there is no agent path to it. The handler opens the state file directly rather than through the agent file gate, which is what lets the operator's toggle work while the agent's writes stay refused.

Per account, like nightly and unlike the export key, because the risk this permission prices is the destination: the archive lands in one account's bucket, so granting it for that bucket must not grant it for another the operator adds later.

The read. backup.sessions_layer_b_enabled(account) defaults to False. A non-boolean stored value reads as False (bool("false") is True in Python, so the value is validated, never coerced), and an unreadable or unparseable state document reads as False too -- the same fail-closed posture nightly_enabled documents for itself. The permission is read once, before the archive is opened: reading it twice would let a write landing mid-build produce a tar that carries Layer B alongside a record saying it does not, and the record is what a restore trusts.

The record. The run record carries layer_b, so which layers an archive holds is stated rather than inferred from an absent key, mirroring the export path's layer_b_skipped. The value is taken from the number of kiro-cli files actually added, never from the permission: a granted run whose kiro-cli directory is absent or empty -- an ordinary state on a fresh or CLI-idle install -- adds none, the crew half alone carries the run past the empty-archive guard, and a record is written once with nothing afterwards to correct it, so reading the permission there would state a fidelity the object does not hold. The field is omitted for the snapshot kind, where the question does not arise. It is a record for a reader, not an input: restore_download does not consult it, and earlier revisions of this description and of the code comments claimed it did. Two archives with the same stamped name are otherwise identical, so without the field nobody can tell which one can resume a session at full fidelity.

A withdrawal landing mid-build refuses the upload. Reading the permission once leaves a window: enabled at the read, withdrawn while the tar is written, and the bytes would upload under a permission the operator has withdrawn. The recheck raises instead, and nothing is uploaded and no run record is written -- so the read-once invariant is untouched, because there is no record left to disagree with anything. Rebuilding without Layer B would instead be the torn state that invariant exists to prevent. Only the withdrawn direction refuses; a grant arriving mid-build leaves an archive without Layer B, which the next run picks up.

The recheck is the last statement before storage.put_file, after _authorize_upload. That call goes to the network, so a check placed in front of it leaves the whole round trip inside the window it exists to close. A test pins the ORDER rather than the behaviour, because both orders pass every other case. The refusal goes through _refuse_upload, the module's audited refusal helper, so a withdrawn permission lands in the SEL beside every other refused upload rather than being the one refusal path that leaves no record.

Both directions of the decision are audited. The allow/withhold decision emits aws_control.backup_layer_b_decision at the point it is made, carrying the account and which way it went. Only refusals reached the SEL before, which left the ALLOW direction -- the one that ships the bytes -- with no event at all. caller is threaded through so an unattended nightly run is not recorded against the dashboard owner, and the audit is best-effort: a failed audit must never be what stops a backup.

No console control in this PR, stated rather than implied. There is no DrivePage row for this permission, so an existing operator's sessions archive drops Layer B with no in-product way to see that or opt back in -- the owner-gated endpoint is the only way to grant it today. That is a deliberate scope line: a toggle row is four new i18n keys across thirteen catalogs plus the generated pseudolocale, the translator-context sidecar and the dead-key baseline, and translations this author cannot verify do not belong on the same head as a security placement fix. A follow-up PR adds the toggle and state indicator beside the nightly row on DrivePage, and the run record already states which way each archive went in the meantime. The status response deliberately does NOT carry sessionsIncludeLayerB yet: a field with no consumer is not the smallest honest version, so it ships with the row that reads it.

The request field is validated, never coerced, for the same reason the nightly toggle is -- and the cost of getting it wrong is higher here, because the wrong direction is unrecallable rather than merely billable.

Why the default is withhold, and the counter-argument. There is a real case for the opposite default, and a reviewer should see it. security_posture.py states a doctrine for exactly this payload on the tunnel path: "What bounds the exposure is the destination rather than the payload", which is why a tunnel send carries Layer B byte-exact with no opt-in. rfc-s3-backup.md likewise asks for a private, ownership-enforced, encrypted bucket of transcripts rather than a payload gate, and storage.py provides that. Under that reading the previous behaviour was consistent and this permission should default on.

The default is withhold anyway, for one reason: the destination is not provably a single operator's. This app supports several installs writing one drive and says so -- ORIGIN_UNVERIFIED exists because "anyone who can write to the bucket can write to a name" -- so a co-writer can reach an archive here. Being wrong in the withhold direction costs a restore its full-fidelity resume until the operator flips one toggle, and the record states that it happened. Being wrong in the carry direction puts unredacted context somewhere it cannot be recalled from. Only one of those is recoverable.

A maintainer who reads the destination as bounded can flip the default in one place.

The default is a recorded decision, D4, and this change implements it. docs/request-for-change/rfc-s3-backup.md:155 states it: "D4 -- session fidelity is a tier. Default sessions = crew transcripts + uploads + artifacts, which buys readable history. Opt-in (O2) adds <kiro home>/sessions/cli/<sid>.* for reachable sids only ... which buys real resume instead of a lossy prefix. Measured: 1.17 GB selected versus 30 GB unselected." That is this change's shape: the default carries the crew half, the kiro-cli tree rides on an opt-in, and the tiers come with a measurement. The RFC's own numbering separates its decisions from its open questions -- D-items are decided, O-items are open -- and the only O-item touching this area is O1, whose line says it "Blocks neither milestone". So the document being status: draft describes its review state, not the status of each item inside it. What remains genuinely open is narrower and is a maintainer's: whether an acceptance of D4 must be recorded before merge. First Principles Review blocks this head on the wider claim that the decision exists nowhere, and its two named clearances -- an RFC merged non-draft on base, or an override -- are both outside the author's reach. The keystone placement is boundary-derived and is not in question; only the default is.

One lock order, and every path obeys it. The recheck, the live authorization and the PUT run inside one hold of the state file's sidecar lock, taken through _upload_lock, which holds the FILE half only and deliberately not _run_lock -- _run_lock also serializes last_runs, which the dashboard's backup-status read goes through, so holding it across a PUT allowed _PUSH_TIMEOUT_SECS stalled every account's status surface for one account's upload. Dropping it from the gate was necessary and not sufficient: the stall arrived through the contending WRITER. _state_lock took _run_lock before parking on the file lock, so a writer meeting an in-flight upload -- a mid-upload revocation, or any second account's _record_run finishing -- held _run_lock for that upload's whole duration and last_runs queued behind the writer. So this states one order, in a comment above _state_lock with _run_lock's own definition pointing at it:

_RETENTION_GATE -> state sidecar FILE lock -> _run_lock -> leaf locks

Nothing may hold _run_lock while waiting for the file lock. _state_lock takes the file lock first, and _record_run and _record_skip no longer wrap it in _run_lock at all. Swapping the order inside _state_lock alone would have been worse than the stall: both of those reach _state_lock through _record_run_locked -> _locked_state_update, so one thread would hold _run_lock waiting for the file lock while another held the file lock waiting for _run_lock -- a cycle. Of the file's 24 textual mentions of _run_lock only 5 are acquisitions; uploaded_objects and last_runs take it alone and never reach the file lock. Removing the outer hold is also not a pure deletion: it was the only thing serialising the _run_sequence bump, and (process, sequence) is the identity the compare-and-set inside mutate reads, so _record_run_locked now takes _run_lock for that bump alone -- which cannot park -- and keeps it outside mutate so the state-read and state-write failure paths still hand a uniquely numbered record to _remember_unpersisted.

The spec. docs/system-specs/modules/aws-control.md describes the per-account backup.json key, its owner-gated writer, and why not config.json. The first commit's paragraph documented the config.json placement the second commit removes as unsafe; leaving it would have instructed the next contributor to re-implement the self-grantable shape.

Backwards compatibility

This changes what an existing sessions archive contains: until the operator grants the permission, new archives carry the transcript half only, and a restore from one replays a transcript rather than resuming through session/load. Archives already in a bucket are untouched. The loss is stated on the run record rather than silent.

No signature changes reach existing callers: layer_b is keyword-only with a default on _record_run, and the new state key is additive -- the writer is a read-modify-write of the whole document under the existing sidecar lock, so the nightly bit beside it survives (pinned by a test).

Tests

test/test_aws_control_backup.py::TestSessionsArchiveLayerBGate (12 cases) and three new cases in test/test_aws_control_routes.py::TestBackupEndpoints. 496 pass across test_aws_control_backup.py, test_aws_control_backup_job.py, test_aws_control_routes.py and test_aws_control_app.py.

Block path: the default withholds with no state file at all (the real reader, not a stub, so the default itself is what is pinned); a non-boolean "true" withholds; an unparseable state document withholds and the backup still runs; an enabled field that is not a boolean is refused by the route with invalid_enabled and never reaches the writer. Each asserts the kiro-cli half is absent from the tar and that the record says layer_b is false.

Allow path: with the permission recorded both halves ride, and the record says so. Both Layer B filenames are fixtured (<sid>.json and <sid>.jsonl) so a narrower filter cannot pass by matching one.

Also pinned: an agent-writable config.json cannot grant the permission under any of the three spellings an earlier draft read, so moving the read back there reddens; the export key does not grant here; a grant for one account does not carry to another; STATE_DIR_LEAF is in security._CREW_SECRET_LEAVES and the app's data dir is that leaf, so moving the store out of the fenced directory reddens; the writer round-trips both directions and keeps nightly beside it; the permission is read exactly once; a granted run whose kiro-cli directory holds nothing records layer_b false while the permission still reads true; a snapshot record carries no layer_b key; a write that fails returns state_persist_failed without reporting the setting as applied and without echoing the state file's absolute path.

Red on base. The original class run against origin/main (f31f25a) with the fix absent: 6 of 7 failed, three of them on assert ['cli/abc.json', 'cli/abc.jsonl', 'crew/t.jsonl'] == ['crew/t.jsonl'], which is also the measurement of the exposure -- no operator choice recorded, Layer B in the archive.

Mutation-verified, each mutation applied by hand and both files restored byte-identical afterwards:

mutation reddens
honour config.json -> dashboard.backup_include_layer_b again the agent-writable-config case only
record the permission instead of the archived file count the granted-run-with-no-cli-files case only
pin the recorded value to false the three allow cases only
isinstance(raw, bool) else False -> bool(raw) the non-boolean case only
honour any account's grant instead of this account's the per-account case only
default the stored value to True 6 cases: default, unparseable, agent-writable config, export key, per-account, writer round-trip; the allow cases correctly survive
route: isinstance(raw, bool) -> raw is not None + bool(raw) the route's non-boolean case only
_run_lock back before the file lock in _state_lock the writer-parked status-read case only
outer _run_lock back in _record_run the writer-parked status-read case only
_run_sequence bump moved outside _run_lock the sequence-identity case only

The last three are the lock-order half. Each reds only its own test, with its own assertion text, so the two lock tests are independent and each half of the fix is load-bearing rather than one change covering for the other. The earlier status-read test could not observe any of this: it had no concurrent writer, so it passed with the old order restored. It is kept for the solo-upload case and the new one carries the contention.

Manual verification

None beyond the tests: the upload path is mocked in every case, so no bucket is reached.

Related Issues

Closes #11559

Pattern harvest

Rule candidate: review-prompt -- when a change introduces a permission, a consent record, or any stored value that AUTHORIZES an irreversible action, ask where that value lives before asking whether the check is correct. A check that reads an agent-writable file is not a gate, however carefully it validates what it finds there, because the party the gate constrains can write the answer. The concrete test is whether the store sits behind the agent file floor and whether exactly one owner-gated writer opens it directly.

Rule candidate: review-prompt -- when a change adds or moves a bulk filesystem copy (a tar, zip, or rsync of a whole directory), ask whether any path inside that directory is governed elsewhere by an explicit permission, and require the bulk path to consult the same permission. The original defect here was not a wrong decision; it was a decision that no code made, because taring a directory reaches whatever is in it.

A payload that cannot be redacted has only two states, so the instrument is a permission rather than a filter -- and when a permission decides what an artifact contains, read it once and record the answer on the artifact, or the artifact and its record can disagree about the same bytes.

When two paths handle one payload differently, the question is which condition each path's destination justifies, not which path is wrong. Here the operator-identity condition already held on both, the per-invocation condition has no channel on an unattended job, and only the standing permission was genuinely missing.

@chenmingwei23
chenmingwei23 requested review from a team and hoang-phan98 September 20, 2026 04:26
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of cb039fcd65a4b63135b2b1d18e7a674492f6947b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cb039fc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cb039fcd65a4b63135b2b1d18e7a674492f6947b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

CI triage for 446f89c

All four AI review lanes hit the same external cause in one window (04:26-04:34 UTC): the GitHub App installation's API rate limit. Each lane's log carries API rate limit exceeded for installation.

What that produced, per lane:

  • GPT 5.6 Review -- review incomplete; no verdict for this head.
  • Opus 4.8 Review -- review step outcome=skipped, no [OPUS-REVIEWED] marker; fails closed with no verdict.
  • UX Review -- VERDICT: SKIPPED; no verdict produced.
  • First Principles Review -- a verdict WAS reached, and its comment was then withheld: "this PR's head was unreadable after 3 attempts, so 446f89c cannot be confirmed current; the comment lookup also failed, so nothing was posted". So the concern text exists nowhere readable. It is being re-run rather than overridden -- an override against a verdict nobody can read would launder a missing review into a green one.

The remaining two reds are one root cause, and it is not a review finding:

  • Backend Tests (3.12, 8) -- test/test_source_corpus.py::TestNoGateEnumeratesTheCheckoutByWalking::test_no_gate_scans_the_checkout_from_the_filesystem, reported as worker 'gw3' crashed, not as a failed assertion; 1 failed of 12594. That file is byte-identical to this branch's base (git diff origin/main...HEAD -- test/test_source_corpus.py is empty), this diff touches three files and none is that test or any gate script, and the test passes locally on this branch. Infrastructure, re-run only.
  • Coverage Gate -- downstream: its log says backend-test=failure -- failing closed. It clears when the shard above does.

PR Hygiene was the one red that was genuinely mine: the ## Pattern harvest section was present but carried no Rule candidate: line, which is what the gate matches. Fixed in the description.

@chenmingwei23
chenmingwei23 force-pushed the fix/archive-layer-b-opt-in-11559 branch from 446f89c to 93d42b8 Compare September 20, 2026 05:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition of the GPT 5.6 blocking finding on 446f89c2 (F1, UPHOLD-FENCED): accepted and fixed, not overridden.

The finding was right. The permission read config.json -> dashboard.backup_include_layer_b, and config.json is writable by any auto-approved agent shell, so the value that authorizes an irreversible upload of unredacted model context was writable by the party it constrains. The adjudication's condition chain is ordinary operation, and there is no recovery path once an object is in a bucket.

Fixed at the placement rather than by hardening the read, which is what the finding's own Fix line asks for. Head 93d42b8a:

  • the answer now lives per account in the app's backup.json, inside the apps/aws-control/data directory already registered in security._CREW_SECRET_LEAVES -- the read+write keystone floor, beside the nightly bit that authorizes unattended paid uploads;
  • the sole writer is the new owner-gated POST /backup/{account}/layer-b, which opens the file directly rather than through the agent file gate, so the operator's toggle works while the agent's writes are refused;
  • the read still fails closed on anything unparseable and is still taken once, before the archive opens.

Pinned so the regression cannot return quietly: one test offers all three config.json spellings an earlier draft read and asserts Layer B is still withheld; another asserts the leaf is in _CREW_SECRET_LEAVES and that the app's data dir is that leaf, so moving the store out of the fenced directory reddens. Mutating the reader back to honour the config key reddens exactly the first of those.

Also in this head: a rebase onto current origin/main (7e8f2c45), which push_guard required and which picks up the test_source_corpus.py shard fix. The PR body is rewritten to describe the keystone design rather than the config key it replaced.

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of cb039fcd65a4b63135b2b1d18e7a674492f6947b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Existing installs' backups silently degrade: Layer B drops with no console control until a follow-up PR, while retention ages full-fidelity archives out of the bucket.

Watch

  • Interim regression window: "There is no DrivePage row for this permission" + default-off means every post-merge nightly archive is crew-only; after retentionKeep runs, no full-fidelity archive remains anywhere, and the only signals are a run-record field "nothing reads programmatically" and a raw owner-gated POST. Ship the toggle follow-up before a release, or call the loss out in release notes.
    Clears when: the DrivePage toggle PR lands on the same release train, or the release notes name the loss and the endpoint.
  • _upload_lock is held across the PUT unconditionally — even for the default crew-only archive, where no Layer B withdrawal can exist to order against. One account's hour-long upload now serializes every account's sessions upload and parks every state writer (_locked_state_update: nightly/retention toggles, all _record_runs) for up to _STATE_LOCK_TIMEOUT_SECS (3660s); dashboard toggle requests from unrelated accounts hang behind it. Taking the lock only when layer_b is true restores the old concurrency for the default path at zero cost to the guarantee.
    Clears when: the gate is scoped to layer_b runs, or a stated rationale covers why crew-only uploads must also hold it.

Suggestions

  • The grant ships more than the decision it cites: D4's opt-in is "<sid>.* for reachable sids only … explicitly not the whole directory", but _add_tree(tar, cli_sessions, "cli") tars the whole tree, co-tenant sids included. Note the divergence in the spec paragraph so the next reader doesn't take D4 as the bound the toggle enforces.

[DESIGN-REVIEWED] cb039fc

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 756fb50ba56c16be693ec0e6151aef74b8bd187b — this comment is updated in place on each push.

Review details

Based on my analysis:

Candidate 1 — verified real. On the skip path (backup.py:3869), _record_skip calls _record_run_locked with the default layer_b=None, so the skip record built at line 1746 omits the layer_b key. The compare-and-set in mutate passes (baseline still in slot) and runs[KIND_SESSIONS] = record replaces a baseline that carried layer_b: true with a record that carries no layer_b at all. The PR's stated guarantee that the record states which layers the archive holds is broken after any unchanged-tree skip. Purely informational (restore_download doesn't read it), so FINDING not BLOCKING. In-diff minimal fix.

Candidate 2 — dropped. The refusal when layer_b_files == 0 requires a narrow race (withdrawal landing exactly in the build window) combined with an empty kiro-cli dir; the outcome is a retryable failure with no data exposure and a message telling the operator to retry. It is arguably the intended conservative posture ("only the withdrawn direction refuses"), the candidate itself rates it low, and it does not clear the 80 bar as an observable wrong outcome.

No new groundable findings surfaced in Step 2.

One advisory finding: the unchanged-tree skip path silently drops the archive's layer_b record.

FINDING — src/kiro_crew/apps/builtins/aws_control/backend/backup.py:3869 — _record_skip(account, KIND_SESSIONS, baseline, tree) threads no layer_b, so the skip record omits the key and replaces a baseline that carried layer_b: true, breaking the PR's own "which layers an archive holds is readable from the record" guarantee after any skip → Fix: pass layer_b=baseline.get("layer_b") through _record_skip to _record_run_locked.

[OPUS-REVIEWED] 756fb50

Verdict parsed from the review's SHA-scoped output markers for commit 756fb50ba56c16be693ec0e6151aef74b8bd187b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 756fb50ba56c16be693ec0e6151aef74b8bd187b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ human override accepted

@chenmingwei23 overrode this lane for 756fb50ba56c16be693ec0e6151aef74b8bd187b via /ai-review override (this commit only).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/archive-layer-b-opt-in-11559 branch from 1f71ff5 to 15a811f Compare September 20, 2026 06:35
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition of the three lanes that failed on 1f71ff54. Head is now 15a811f4, squashed to one commit.

GPT 5.6 F1 (security-fenced, adjudicator FLAGGED as likely edge case): accepted and fixed, not overridden. The finding named a real window -- enabled at the read, revoked while the tar is written, bytes upload under a permission that no longer exists. The adjudicator was right that the remedy it proposed, rebuild-without-Layer-B, would reintroduce the torn state the read-once rule exists to prevent. Refusing does not: run_sessions_backup re-reads immediately before the PUT and raises, so nothing is uploaded and no run record is written, and there is no record left to disagree with anything. Only the revoked direction refuses; a grant arriving mid-build leaves an archive without Layer B, which the next run picks up. Both directions are pinned, and both mutations kill exactly their own test: disabling the refusal reddens the revocation case, widening it to any disagreement reddens the grant case.

First Principles item 3 (zero consumers): taken as the subtraction the reviewer named. sessionsIncludeLayerB is removed from the backup status response. A field with no reader is not the smallest honest version; it ships with the DrivePage row that reads it.

PR Hygiene (commit count): squashed. Three commits against a ceiling of two. Now one.

First Principles blocker (the default flip) is not the author's to clear, and is left for a maintainer. The reviewer is right that the shape is proposed rather than decided: docs/request-for-change/rfc-s3-backup.md is status: draft on base even though its D4 proposes this exact tier, and the standing doctrine in security_posture.py reads the other way -- which this PR's description has conceded from the first revision. Neither clearance path is available here: an RFC cannot be merged to base from this branch, and an override is a maintainer's decision. The description now states the open question in those terms. The keystone placement is boundary-derived and is not what is in question; only the default is.

Evidence on this head: 498 tests pass across the four aws-control files; black, flake8, isort, mypy, comment-history, brand and docs-lint all exit 0; zero non-ASCII added lines; one commit.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/archive-layer-b-opt-in-11559 branch from 15a811f to 4220bee Compare September 20, 2026 06:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the lock scope.

_state_lock() is not only the cross-process file lock — its body is with _run_lock, open_lock_file(...) (src/kiro_crew/backup/backup.py:299), so it also takes the process-wide _run_lock. This PR newly wraps the put_file upload inside that block; on main that call site holds no lock at all.

last_runs() (backup.py:3225) takes the same _run_lock, and the backup status polling in src/kiro_crew/dashboard/routes.py:2304 goes through it. So while any one account is uploading, the whole backup status surface blocks — across accounts — for as long as the upload takes, which for a large snapshot can exceed an hour. The module already warns against exactly this at backup.py:1452.

The consent re-read this PR adds is correct and worth keeping; it's the duration that's the problem. Please narrow the critical section so the network PUT happens outside _state_lock() (re-read the grant under the lock, release, then upload — or hold only the file lock, not _run_lock, for the upload window).

Nothing else blocking: the per-kind grant / redaction gate ordering checks out, and CI is green on the current head.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Accepting the lock-scope finding. It is correct, and the module had already made this
decision one level over, so this is not a judgement call.

Verified against the current head before replying:

  • _state_lock() line 299 is with _run_lock, open_lock_file(lock_path) as fd:, so it
    does take the process-wide lock as well as the sidecar file lock.
  • last_runs() line 3225 takes _run_lock, and the backup status read reaches it.
  • On the base, put_file sat at a call site holding no lock; this branch removed it from
    there and re-added it indented inside with _state_lock():.

One correction to the citations, offered so the finding is not dismissed on it: the paths
name src/kiro_crew/backup/backup.py, which does not exist in the tree. The file is
src/kiro_crew/apps/builtins/aws_control/backend/backup.py. Every line number quoted is
correct for that file.

The decisive argument is this module's own: the comment above _RETENTION_GATE states that
it is deliberately not _run_lock, "that one also serializes last_runs, so holding it
across a purge would stall a status read for the length of the deletion." The Layer B
upload gate does exactly what that comment rejects, and for a longer window.

Taking the second of the two suggested remedies, not the first. Re-reading the grant under
the lock, releasing, then uploading would reopen the race an earlier review round blocked
this branch on: a revocation landing between the re-check and the PUT, which is the
specific hole the gate exists to close. Narrowing the critical section so the network PUT
does not hold _run_lock keeps that hole closed.

One detail is load-bearing and is being settled before the change lands rather than
assumed. _run_lock is a threading.RLock and its own comment scopes it to "this
process"; the sidecar file lock is what orders separate processes. Dropping _run_lock
from the upload window therefore does not by itself preserve exclusion against
_locked_state_update, which still takes it, because two different in-process locks do
not exclude one another. Whether the file lock alone covers that gap depends on whether
file_lock uses flock semantics, which are per open file description and do exclude two
threads, or fcntl record locks, which are per process and do not.
_delete_under_the_retention_gate is the model: it composes a dedicated lock with the
sidecar lock and says "BOTH locks, because either alone leaves a real hole." The shape here
will follow it, with the exclusion argument stated rather than implied.

The fix will arrive as a single push carrying the narrowed scope and its tests, rather than
several. The recorded override on this lane is pinned to the current SHA and a new head
voids it, and the RFC that would otherwise clear that lane is still draft on main, so each
additional head costs a re-record. Tests will pin both directions: a revocation arriving
inside the narrowed window is still refused, and an ordinary upload still succeeds.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Thanks -- this was a real regression and the narrower of your two suggested
remedies is what the new head does. New head is 86dfb6af66.

What changed

The upload window now holds ONLY the state file's sidecar file lock, not
_run_lock. _upload_lock() (backup.py:307) is a new context manager that takes
open_lock_file + file_lock and deliberately omits _run_lock. The gate is at
:2640 and put_file at :2662, both inside _upload_lock() and outside
_run_lock. _state_lock() is unchanged and still takes both, because the
setter path wants the process-wide lock.

Measured across the three heads, by walking the AST for every storage.put_file
call and its enclosing with items:

head encloses the sessions put_file holds _run_lock?
main TemporaryDirectory only no lock at all
cf0e7cc7ca (what you reviewed) _state_lock() yes, via its body
86dfb6af66 (new) _upload_lock() no

last_runs still takes exactly [_run_lock], unchanged.

Why dropping _run_lock costs the guarantee nothing

The invariant the gate owns is that a consent withdrawal cannot interleave
between the recheck and the PUT. That ordering is against the SETTER, not against
the reader. The setter is set_sessions_layer_b -> _locked_state_update ->
_state_lock, and it takes this same sidecar file lock EXCLUSIVELY -- so an
exclusive hold across the upload already orders a revocation wholly before the
block or wholly after it, across processes as well as threads. _run_lock was
never the thing carrying that ordering; it was only also serializing last_runs,
which is the stall you found.

This is the shape _delete_under_the_retention_gate already composes in the same
file, for the same reason: the sidecar file lock plus a dedicated gate rather than
_run_lock, so a purge does not stall the status read either. The comment you
cited is the module stating that rule about itself.

Tests, mutation-verified in both directions

Both pin the lock scope by probing from another thread without blocking, so a
blocking acquisition cannot deadlock against the lock under assertion.

  • test_a_status_read_is_not_blocked_by_an_in_flight_upload -- from inside
    put_file, a real last_runs call completes. Reintroducing _run_lock in
    _upload_lock reddens it with "the status read blocked behind the in-flight
    upload".
  • test_the_upload_holds_the_setter_lock_across_the_permission_read -- the
    setter's file lock is NOT available to another thread during the PUT, so a
    revocation still cannot overtake the upload.

Together they show the change is conditional rather than a blanket loosening:
the reader is freed, the writer is not.

I also checked the hazard the new lock could introduce and you did not raise:
a lock-order inversion. _state_lock takes _run_lock then the file lock, so
anything taking the file lock first and then _run_lock could deadlock. Nothing
reachable inside the block does -- _authorize_upload, sessions_layer_b_enabled
and _refuse_upload take no lock at all, and _record_run (which does take
_run_lock) runs after the block has released.

470 tests pass across the three files this change owns.

One note on the citations

Your line numbers were all exact. Two of the file paths were off by a directory
prefix, so anyone following them lands nowhere -- flagging it only so the next
reader is not sent hunting. The real files are under
src/kiro_crew/apps/builtins/aws_control/backend/; neither
src/kiro_crew/backup/ nor src/kiro_crew/dashboard/routes.py exists in the
tree. Against cf0e7cc7ca, every line you named resolved:

  • backup.py:299 -- with _run_lock, open_lock_file(lock_path) as fd: in
    _state_lock, which is the finding
  • backup.py:3225 -- def last_runs
  • backup.py:1452 -- the _RETENTION_GATE comment warning against _run_lock
    for precisely this reason
  • routes.py:2304 -- "runs": await asyncio.to_thread(backup_mod.last_runs, account)

So the review read the right lines in the right files; only the paths were
mis-stated, and the finding itself held.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 7703ba0: Default accepted as the product decision: the sessions archive withholds Layer B unless an operator opts in. rfc-s3-backup.md D4 already records session fidelity as a tier, so its draft status is a documentation lag, not an unmade decision. The opt-in path is reachable and the description states the changed default. Accepting the recorded-RFC gap.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 7703ba07cf28cd0c9f1225cd4571b2a2b235598c.

Default accepted as the product decision: the sessions archive withholds Layer B unless an operator opts in. rfc-s3-backup.md D4 already records session fidelity as a tier, so its draft status is a documentation lag, not an unmade decision. The opt-in path is reachable and the description states the changed default. Accepting the recorded-RFC gap.

This decision applies only to this commit. A new push requires a new judgment.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Branch frozen: the first-principles override recorded above is pinned to 7703ba0. Any further push voids it and the block returns. Do not push this branch.

bolichen97
bolichen97 previously approved these changes Sep 21, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 7703ba07cf.

The lock-scope regression @buluoray requested changes on is fixed, and I verified it rather than taking the reply. Reading the diff at this head: _upload_lock() takes open_lock_file + file_lock and deliberately omits _run_lock; the withdrawal recheck and storage.put_file are both inside it; _state_lock() is unchanged and still takes both for the setter path. That is the narrower of the two remedies offered, and it is the correct one — re-reading the grant under the lock and releasing before the PUT would have reopened the revocation race an earlier round blocked this branch on. The ordering the gate owns is against the setter, and the setter goes through the same sidecar file lock exclusively, so dropping _run_lock costs the guarantee nothing while freeing the status read. Same shape _delete_under_the_retention_gate already composes in this file, for the same stated reason.

The permission placement holds. The store is backup.json under the app's data dir, which is in security._CREW_SECRET_LEAVES, so an agent shell cannot write the answer to its own gate — the point the description makes and the reason config.json was rejected. Sole writer is the owner-gated route through _guarded. Read once, before the archive opens, defaulting False on absent/non-boolean/unparseable.

The record describes the archive, not the permission. layer_b=layer_b_files > 0 is taken from the count actually added, so a granted run on a CLI-idle install records false rather than claiming a fidelity the object does not hold. That is the right direction and it is what a restore reads.

Two things I am flagging rather than blocking on:

  • No console control ships here. An existing operator's sessions archive drops Layer B and the owner-gated endpoint is the only way to opt back in. The default moves in the safe direction and the run record states it, so this is recoverable — but the follow-up toggle should not sit long.
  • Merge order. This, #12381 and #12190 all rewrite backup.py, and both this branch and #12381 carry SHA-pinned first-principles overrides that a rebase voids. I am landing the two frozen branches first for that reason.

@buluoray your CHANGES_REQUESTED is still recorded against cf0e7cc7ca; please dismiss it if you agree the narrower remedy answers the finding.

@bolichen97

Copy link
Copy Markdown
Contributor

Approved above at 7703ba07cf, but the branch went dirty before it could land: #12381 (4e8af7b6bf) and #12190 (619c10fc33) both merged into main minutes ago and both rewrote the same region of backup.py. Flagging so my approval is not read as "ready" against the current base.

git merge-tree origin/main gives 7 conflicts across 3 files; docs/system-specs/modules/aws-control.md, test_aws_control_backup.py and test_aws_control_routes.py all auto-merge. Every conflict is keep-both with one parameter to thread through, so recording the map rather than leaving it to be re-derived:

backend/backup.py (4)

  1. _record_run signature — main now carries tree: str = "", uploaded: bool = True and the assert recorded is not None / return split from feat(aws-control): stop the nightly re-uploading a tree that has not changed #12381; this branch carries *, layer_b: bool | None = None. Keep all three kwargs and forward all three to _record_run_locked.
  2. _record_run_locked signature — same three-way, same resolution.
  3. The upload block in run_sessions_backup — the one that needs reading rather than merging. Main is now _authorize_upload(account, profile, region, caller=caller, payload_kind=KIND_SESSIONS), because feat(aws-control): carry transcripts in the nightly under their own consent bit #12190 added payload_kind for the per-kind consent re-read. This branch wraps the block in with _upload_lock(): and calls _authorize_upload without it. Take this branch's _upload_lock() block and add payload_kind=KIND_SESSIONS to the call inside it — dropping it would silently lose feat(aws-control): carry transcripts in the nightly under their own consent bit #12190's scheduled-consent re-read, which is the same class of gap as the lock-scope one.
  4. The _record_run(...) call site — keep layer_b=layer_b_files > 0 beside whatever feat(aws-control): stop the nightly re-uploading a tree that has not changed #12381 passes.

backend/routes.py (2) — an import line and one route-table region; both additive.

test/test_aws_control_app.py (1) — the pinned route inventory; keep both new routes.

Two things worth saying plainly before the push:

  • The rebase voids the override. The first-principles override recorded at 7703ba07cf is SHA-pinned, so the BLOCK returns on the new head and needs a fresh maintainer judgment on the withhold default. That is the one part of this PR I am not deciding on someone else's behalf — it is a security default with an unrecallable wrong direction.
  • @buluoray's CHANGES_REQUESTED is still recorded against cf0e7cc7ca. The lock-scope remedy is genuinely in (_upload_lock() omits _run_lock; the recheck and put_file sit inside it), so it wants a dismissal rather than more work.

Happy to do the rebase and run the aws-control suites if you would rather not touch it — say so and I will, but I will not re-issue the override.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 078f740: Verified on this head that the only occurrence of record[layer_b] in the aws-control backend is the write at backup.py:1392, so no reader exists; the skip record still carries the baseline key, bytes, fingerprint, version and tree (backup.py:2742-2751) so restore fidelity is untouched; and the next real upload restates it at backup.py:3288. Recoverable, inspection-only gap.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 078f740: Owner decision to ship the sessions-archive Layer B consent gate now and leave rfc-s3-backup.md draft. Item 1 is the RFC state rather than a defect; items 6 to 8 are scope objections to the mid-upload revocation gate that the human review required, and that gate takes only the sidecar state lock, never the run lock, so status reads are not serialized.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 078f740b127fa86e07a0f4e21313d0f316574c73.

Verified on this head that the only occurrence of record[layer_b] in the aws-control backend is the write at backup.py:1392, so no reader exists; the skip record still carries the baseline key, bytes, fingerprint, version and tree (backup.py:2742-2751) so restore fidelity is untouched; and the next real upload restates it at backup.py:3288. Recoverable, inspection-only gap.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 078f740b127fa86e07a0f4e21313d0f316574c73.

Owner decision to ship the sessions-archive Layer B consent gate now and leave rfc-s3-backup.md draft. Item 1 is the RFC state rather than a defect; items 6 to 8 are scope objections to the mid-upload revocation gate that the human review required, and that gate takes only the sidecar state lock, never the run lock, so status reads are not serialized.

This decision applies only to this commit. A new push requires a new judgment.

@buluoray

Copy link
Copy Markdown
Contributor

Re-verified at 078f740b. _upload_lock() is the right move and four of the five things it claims hold exactly as written. The fifth does not, so the availability regression is narrowed rather than closed and my earlier request stands.

What checks out:

  • _upload_lock() really takes only the file half — body is open_lock_file + file_lock(exclusive=True), no _run_lock (backup.py:324-362).
  • The revocation ordering is intact. set_sessions_layer_b -> _locked_state_update -> _state_lock takes the same sidecar lock file exclusively, and file_lock is descriptor-based, so a withdrawal lands wholly before or wholly after the block, across processes as well as threads.
  • The in-lock recheck is still the last statement before the PUT (:3262 then put_file at :3271, nothing between), and it is fail-closed: sessions_layer_b_enabled returns raw if isinstance(raw, bool) else False, _refuse_upload is NoReturn.
  • #12190's payload_kind=KIND_SESSIONS survived the rebase (:3252).

What does not hold — the _upload_lock docstring's closing claim that omitting _run_lock "leaves the status reader free":

_run_lock is an RLock (:1133), and _state_lock() takes it before parking on the file lock with a _STATE_LOCK_TIMEOUT_SECS ceiling (:318). So any state writer that contends with an in-flight upload sits on _run_lock for the upload's duration:

set_sessions_layer_b -> _locked_state_update -> _state_lock
    with _run_lock, file_lock(... timeout=_STATE_LOCK_TIMEOUT_SECS)   # :318
                    ^ parked here, still holding _run_lock

last_runs then stalls behind it (with _run_lock: at :3966), which is the cross-account status stall the change set out to remove. Two reachable triggers, and the first is the feature's own headline path:

  1. A mid-upload revocation. It must contend on the file lock for the ordering guarantee above to mean anything — so the guarantee and the stall are the same event.
  2. Any concurrent second-account or second-kind run finishing: _record_run (:1316) takes _run_lock, then _record_run_locked -> _locked_state_update -> _state_lock re-enters it and parks on the file lock while still holding it.

On main the PUT takes no lock at all, so both cases are introduced here. This is a real improvement over cf0e7cc7c — a solo upload with no contending writer no longer stalls status — but the docstring at :344-346 and the override both assert an invariant the code does not satisfy under contention, which is the part I would not want to land as written.

test_a_status_read_is_not_blocked_by_an_in_flight_upload passes vacuously: it has no concurrent writer, so it cannot observe the stall. A test that starts a second-account _record_run (or a set_sessions_layer_b) while the upload holds _upload_lock, then times last_runs, would redden.

The smallest fix I can see inside lines this PR already touches is to take the file lock before _run_lock in _state_lock (:318), so a parked writer holds no _run_lock. No other site takes _run_lock then the file lock — :991 and :2741 take _run_lock alone, _upload_lock takes the file lock alone, _delete_under_the_retention_gate composes the file lock with _RETENTION_GATE — so the reorder gives one consistent acquisition order and no cycle. Worth confirming that yourself rather than taking my word for it. If you would rather not touch _state_lock, the alternative is to correct the docstring and the override to say what is actually guaranteed (a solo upload does not stall status; a contending writer still does) and carry the rest as a known limitation.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed at 020deeefe9. Your finding is right and I verified it independently rather than taking it on trust, as you asked. One correction to the remedy, below, which is why the change is larger than the one line you suggested.

The finding reproduces

_run_lock is an RLock, _state_lock took it before parking on the file lock with a _STATE_LOCK_TIMEOUT_SECS ceiling, and last_runs queues on it. Both triggers you name are real, and the second is the one that matters most because it needs no revocation at all -- any second account's run finishing during an upload was enough. test_a_status_read_is_not_blocked_by_an_in_flight_upload did pass vacuously: it has no concurrent writer, and it still passes with the old order restored.

Where your remedy's premise does not hold

Your fix paragraph says "No other site takes _run_lock then the file lock -- :991 and :2741 take _run_lock alone". :2741 is not one of those. _record_skip at :2741 calls _record_run_locked, which reaches _locked_state_update, which enters _state_lock -- so it takes _run_lock and then the file lock. _record_run at :1316 has the identical shape, which your own trigger 2 describes correctly two paragraphs earlier.

So swapping the order inside _state_lock alone would not have been a fix. One thread would hold _run_lock waiting for the file lock while another held the file lock waiting for _run_lock: a cycle, which is worse than the stall it replaces.

For the record, the file has 24 textual mentions of _run_lock but only 5 acquisitions: _state_lock :318, uploaded_objects :991, _record_run :1316, _record_skip :2741, last_runs :3966. Three of the five couple to the file lock; uploaded_objects and last_runs never reach it.

What the change does instead

One acquisition order for the module, written in a comment above _state_lock, with _run_lock's own definition pointing at it:

_RETENTION_GATE -> state sidecar FILE lock -> _run_lock -> leaf locks

Nothing may hold _run_lock while waiting for the file lock. _state_lock takes the file lock first, and _record_run and _record_skip no longer wrap it in _run_lock at all. _upload_lock (file lock alone) and _delete_under_the_retention_gate (_RETENTION_GATE then the file lock) were already consistent with it. _unpersisted_lock and _fallback_lock are leaves -- install_identity takes _fallback_lock either inside its mutate callback, so under the file lock, or in its except OSError handler after everything has released, and it acquires nothing under it.

Removing the outer _run_lock is not a pure deletion, which is the part that is easy to miss. It was the only thing serialising the _run_sequence bump, and (process, sequence) is the identity the compare-and-set inside mutate reads and the pair _run_is_newer orders by -- so a shared sequence would let a stale baseline pass a check it must fail. _record_run_locked now takes _run_lock for that bump alone, which cannot park, and keeps it outside mutate so the state-read and state-write failure paths still hand a uniquely numbered record to _remember_unpersisted.

The false claim, everywhere it was stated

You flagged the _upload_lock docstring. It was stated in three more places, all corrected in the same commit: the closing sentence of the _upload_lock docstring, the run_sessions_backup comment above the gate ("_record_run and every status read do not wait"), the commit message, and the spec doc's "Dropping _run_lock therefore keeps the guarantee and frees the reader". The PR body never carried it. Each now says that omitting _run_lock from the gate was necessary and not sufficient, and why.

Tests

test_a_status_read_is_not_blocked_by_a_writer_parked_behind_an_upload is the test you described. It starts a real second-account _record_run while the upload holds the file lock, then times last_runs from a third thread. The park point is reached deterministically rather than slept on: file_lock is spied on and signals when the writer's own thread reaches it, and under the old order _run_lock was necessarily already held by then because _state_lock took it before open_lock_file. The writer is joined in a finally with a bound, so a failing assertion cannot leave a thread parked on an hour-long ceiling.

test_the_run_sequence_is_bumped_under_the_run_lock holds _run_lock and asserts _run_sequence cannot move while another thread holds it.

Mutation matrix, each mutation applied alone:

mutation status-read test sequence test
none pass pass
_run_lock back before the file lock in _state_lock fail pass
outer _run_lock back in _record_run fail pass
sequence bump moved outside _run_lock pass fail

Each reds only its own test, with its own assertion text, so the two are independent and each half of the fix is load-bearing rather than one change covering for the other.

Scoped runs, -n0: test_aws_control_backup.py 140 passed, plus test_aws_control_nightly_sessions.py and test_aws_control_backup_unchanged.py 132 passed, plus test_aws_control_backup_job.py, test_aws_control_backup_retention.py, test_aws_control_app.py and test_aws_control_routes.py 515 passed. black at the repo's 100-column width, isort, flake8 and mypy clean on both changed files.

One stale rationale your change surfaces indirectly: test_a_long_holder_does_not_refuse_a_waiting_contender justified taking the file lock directly by saying _run_lock would serialize the two threads first. That is no longer true under the new order, so the sentence is rewritten to the reason that does hold.

The head moved, so the two SHA-pinned overrides recorded on 078f740b12 no longer apply to it.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head cb039fcd65. Three of the five reds on 020deeefe9 had one root cause; the GPT finding was real and is fixed; First Principles items 6/7 are rebutted with the interleaving that review's own clearance line asks for. Item 1 remains a maintainer's.

The three cascading gates

The Comment History Gate was the root cause, and it was entirely mine: my new comments narrated what the code changed instead of stating what it does, so the gate flagged 7 markers ("used to", "no longer") on 6 added lines. All 6 rewritten to present tense. The counterfactual is now stated as a conditional -- "a _state_lock that took _run_lock before parking on the file lock would leave that writer holding it" -- which is design rationale rather than history. The history belongs in this commit message and the PR body, and stays there.

Await Fast Gate and Coverage Gate were downstream of it, not separate problems. Comment History is a member of the Fast Gate run, Await Fast Gate reported Fast Gate concluded 'failure' ... the heavy jobs are skipped on purpose, and Coverage Gate then failed closed on frontend-test=skipped. Verified locally with the repo's own gate plus its self-test (21 flagged probes, 15 clean, so the gate is not vacuously green), against both the PR base and origin/main.

GPT 5.6: real, self-introduced, fixed

The finding is correct and I introduced it. Before this change the outer _run_lock in _record_run spanned both the _run_sequence bump and the state write, so write order equalled sequence order. Removing that hold -- which is what frees the status read -- decoupled them: _run_lock is released before the file lock is taken, so two same-kind runs can reach the file lock in an order that differs from their sequence order, and the unconditional runs[kind] = record let the lower-sequenced writer persist last. The adjudication's reachability argument is right too: the nightly loop calls work directly and so does not take the Job SDK's (kind, account) dedupe.

I had reasoned past this on the strength of the comment at that line ("this new locked write supersedes prior state, regardless of its clock or process"). That comment was true only because the outer hold made the two operations one critical section, which my change ended. The comment is rewritten.

mutate now skips the assignment when this process has already persisted a higher-sequenced run for the same kind. Scoped to THIS process deliberately: that is the same-process half of _run_is_newer and not a call to it, because its other half compares wall time, and applying that here would let a peer install with a lagging clock refuse a locked write that really is newer. A foreign record still loses to this write exactly as an unguarded assignment would have it. The uploads and versions merge stays unconditional -- the object is in the bucket whichever run persists, and uploaded_versions is what retention reads before it erases object versions.

test_a_lower_sequence_run_does_not_clobber_a_newer_one produces the inversion rather than simulating it: the low-sequence writer bumps, parks inside file_lock, and is released only after the high-sequence writer has committed. Removing the guard reddens it on a lower-sequence run overwrote a newer one, and it reddens nothing else.

First Principles items 6 and 7: the premise does not hold

Item 6 says holding the sidecar lock across the PUT "changes nothing the operator can observe -- a revocation landing on either side of the recheck still finds the bytes shipped". There is a third position, and it is the one the lock exists for: a revocation landing AFTER the recheck and BEFORE put_file.

Without the lock, sessions_layer_b_enabled takes no lock of its own, so this ordering is available:

  1. the upload rechecks the permission and reads True
  2. the operator's set_sessions_layer_b(account, False) commits the withdrawal to the state file
  3. the upload calls put_file and ships Layer B under a permission that was already withdrawn when the PUT began

With the exclusive sidecar hold spanning the recheck and the PUT, step 2 cannot land between 1 and 3. The setter reaches the same lock file through _locked_state_update -> _state_lock, so the withdrawal lands wholly before the block -- in which case the recheck inside the lock reads False and _refuse_upload fires, and no bytes are shipped -- or wholly after it. That is the difference the bare recheck cannot make, and bytes in a bucket have no recovery.

This is not hypothetical. test_the_upload_holds_the_setter_lock_across_the_permission_read probes, without blocking, whether the setter's own file lock is free from another thread at the moment put_file runs, and asserts it is not. It is mutation-verified in the removal direction: with no lock held, it reddens.

So the proposed subtraction -- delete _upload_lock, _STATE_LOCK_TIMEOUT_SECS and the lock-order work -- would reopen the exact check-to-use window this PR closed, and would also reverse the narrowing a human maintainer requested in review (comment 5762456674), which explicitly called the consent re-read "correct and worth keeping" and asked only for the duration to be narrowed. Item 7 rides along with item 6 and falls with it.

The lock-order regime item 7 objects to is not an optional addition either. Once the hold exists, _state_lock coupling _run_lock to the file-lock wait is what put last_runs behind an in-flight upload, and that is the defect the maintainer's review is about. Naming one order and making every path obey it is smaller than the alternative, which is a per-call-site judgement every future author has to re-derive.

Item 1 is unchanged and is not mine to clear

Item 1 -- the default flip without a non-draft RFC -- is the same premise blocker this PR has carried, and its two named clearances are an RFC merged non-draft on the base or a maintainer override. Neither is reachable from here. The keystone placement is not in question; only the default is. Nothing in this head changes that item either way.

Verification on this head

Scoped runs, -n0: test_aws_control_backup.py with test_aws_control_nightly_sessions.py and test_aws_control_backup_unchanged.py 273 passed; test_aws_control_backup_job.py, test_aws_control_backup_retention.py, test_aws_control_app.py and test_aws_control_routes.py 515 passed. black at the repo's 100 columns, isort, flake8 and mypy clean on both changed files. Comment-history gate passes with its self-test. Still one commit.

…permission

run_sessions_backup tarred the kiro-cli session directory wholesale, so Layer B
-- the byte-exact, unredacted model context window -- rode every sessions
backup. It rode because the raw files were in the directory being tarred, not
because any permission was consulted, while the file-export path required an
operator opt-in for the same payload.

The permission is stored per account as sessionsIncludeLayerB in the app's
backup.json, inside the apps/aws-control/data directory already registered in
security._CREW_SECRET_LEAVES -- the read+write keystone floor, beside the nightly
bit that authorizes unattended paid uploads. Deliberately NOT a config.json key:
config.json is writable by any auto-approved agent shell, so a permission
honoured from there is one a prompt-injected agent can grant itself, and an
unredacted archive already in a bucket cannot be recalled. An authorization whose
subject can write it is not an authorization.

The sole writer is the owner-gated POST /backup/{account}/layer-b, which opens
the state file directly rather than through the agent file gate. The read
defaults to withhold and fails closed on anything unparseable. It is taken once,
before the archive opens, so the tar and its run record cannot disagree; a
revocation landing in that window refuses the upload rather than rebuilding,
because nothing uploaded and nothing recorded leaves no record to disagree with.
sessions_layer_b_enabled takes no lock of its own, so rechecking alone left a
check-to-use window in which a revocation committing between the recheck and
put_file still shipped the bytes. The recheck, the live authorization and the PUT
therefore run inside one hold of the state file's sidecar lock, taken before
_authorize_upload, which makes a revocation land wholly before the upload or
wholly after it.

That hold takes ONLY the sidecar file lock, deliberately NOT _run_lock -- the
same shape _delete_under_the_retention_gate composes for the retention sweep, and
for the same reason. An earlier revision held _run_lock as well (via
_state_lock). But _run_lock also serializes last_runs, and the dashboard's
backup-status read goes through it, so holding it across a PUT allowed
_PUSH_TIMEOUT_SECS stalled every account's status surface for one account's
upload -- for a large snapshot, over an hour. The revocation guarantee does not
need _run_lock: the setter (set_sessions_layer_b -> _locked_state_update ->
_state_lock) takes this same sidecar file lock exclusively, so an exclusive hold
across the upload already orders a revocation wholly before or wholly after it,
across processes as well as threads.

Dropping _run_lock from the upload gate was necessary and NOT sufficient, because
the stall arrived through the contending WRITER rather than through the upload.
_state_lock took _run_lock BEFORE parking on the sidecar file lock, so a writer
meeting an in-flight upload held _run_lock for that upload's whole duration and
last_runs queued behind the writer, bounded only by _STATE_LOCK_TIMEOUT_SECS.
Both of the feature's own paths reach that: a mid-upload revocation, which must
contend on the file lock for the ordering above to mean anything, and any second
account's _record_run finishing. So this states ONE acquisition order for the
module and makes every path obey it:

    _RETENTION_GATE -> state sidecar FILE lock -> _run_lock -> leaf locks

Nothing may hold _run_lock while waiting for the file lock. _state_lock takes the
file lock first, and _record_run and _record_skip no longer wrap it in _run_lock
at all. Swapping the order inside _state_lock alone would NOT have been enough
and would have been worse: _record_run and _record_skip both reach _state_lock
through _record_run_locked -> _locked_state_update, so one thread would hold
_run_lock waiting for the file lock while another held the file lock waiting for
_run_lock -- a cycle, not a stall.

Removing their outer hold is not a pure deletion either. It was the only thing
serializing the _run_sequence bump, and (process, sequence) is the identity the
compare-and-set inside mutate reads and the pair _run_is_newer orders by, so a
shared sequence would let a stale baseline pass a check it must fail.
_record_run_locked therefore takes _run_lock for that bump alone, which cannot
park, and keeps it outside mutate so the state-read and state-write failure paths
still hand a uniquely numbered record to _remember_unpersisted. The order is
written in a comment above _state_lock and _run_lock's own definition points at
it. The remaining cost is unchanged: the setter and the nightly loop wait for the
upload, bounded by _PUSH_TIMEOUT_SECS, and no status read waits.

Decoupling the bump from the write has a second consequence, and mutate now
guards it. Since _run_lock is released before the file lock is taken, two
same-kind runs in this process can reach the file lock in an order that differs
from their sequence order, and the unconditional `runs[kind] = record` would then
let the lower-sequenced writer persist last -- leaving the slot holding the older
key, tree and layer_b, so the status surface and the next baseline comparison
both read a superseded run. A manual run overlapping a nightly wake for one
account reaches it, because the nightly loop calls work directly and so does not
take the Job SDK's (kind, account) dedupe. The write is therefore skipped when
this process has already persisted a higher-sequenced run for the same kind.

That guard is the same-process half of _run_is_newer and deliberately not a call
to it: its other half compares wall time, and applying that here would let a peer
install with a lagging clock refuse a locked write that really is newer. A
foreign record still loses to this write exactly as an unguarded assignment would
have it, so clock and process comparisons stay confined to best-effort recovery.
The uploads and versions merge stays unconditional, because the object is in the
bucket whichever run persists and uploaded_versions is what retention reads
before it erases object versions.

The run record carries layer_b, taken from the number of kiro-cli files actually
added rather than from the permission: a granted run whose kiro-cli directory is
absent or empty adds none, the crew half alone carries the run past the
empty-archive guard, and a record is written once with nothing afterwards to
correct it.

Tests pin both directions, that no config.json key can grant it, that a grant
does not cross accounts, that the store stays inside the fenced directory, that a
revocation refuses while a grant does not, and that a granted run with no
kiro-cli files records layer_b false. The lock tests are mutation-verified in
BOTH directions, probed without blocking from another thread: the upload holds
the setter's sidecar file lock across the recheck and PUT, so reintroducing no
lock reddens the revocation-guarantee tests. The status-read test drives a real
second-account _record_run while the upload holds the file lock and times
last_runs from a third thread, reaching the park point deterministically by
spying on file_lock rather than sleeping -- restoring the old order plus the
outer _run_lock reddens it, which the earlier single-threaded version could not
observe and which is why it passed vacuously. A separate test holds _run_lock and
asserts _run_sequence cannot move while another thread holds it, so leaving the
bump outside that hold reddens too. A third produces the write-order inversion
rather than simulating it -- the low-sequence writer parks in file_lock until the
high-sequence writer has committed -- and asserts the newer run survives while
both uploads still reach the versions map, so removing the newer-sibling guard
reddens it. Each mutation reddens the test it is aimed at. The two that reintroduce the
coupling also deadlock the inversion test's harness, which parks its
low-sequence writer and waits for the high-sequence writer to commit: that
writer cannot proceed once a parked writer holds _run_lock again, so the second
red is a further symptom of the same cause rather than a separate defect. The
bump and guard mutations redden one test each. Together they
show the fix is conditional: the reader is freed, the writer is not.

That hold spans a PUT allowed an hour, so the sidecar lock is taken with a
ceiling derived from the work it covers rather than file_lock's 300s default,
which is sized for a sub-second read plus a rename and must be overridden by any
caller that can hold the lock longer. A short ceiling would refuse a contender
that is only waiting, and that refusal arrives as the OSError _record_run absorbs
by keeping a completed upload's record in memory alone, so a short-lived process
that exits before the next successful write loses it and leaves the nightly loop
due and re-uploading. frontend._STAGING_LOCK_TIMEOUT derives its ceiling the same
way for a lock spanning a frontend build. A test asserts the value reaches
file_lock, so neither dropping the argument nor sizing it below the push bound
survives. A second test drives the consequence rather than the value: with the
module default patched down, a holder keeps the sidecar lock past it and a
contender must still acquire once the holder releases. Without the explicit
ceiling that contender is refused with "a holder is stuck" while the holder is
only working, which is the failure being fixed.

Closes #11559
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 756fb50ba56c16be693ec0e6151aef74b8bd187b and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- test/test_aws_control_backup.py:1256 -- >= backup._PUSH_TIMEOUT_SECS still passes if the authorization allowance is removed -> Fix: include _AUTHORIZE_TIMEOUT_SECS in the asserted lower bound.
[GPT-REVIEWED] 756fb50

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 756fb50ba56c16be693ec0e6151aef74b8bd187b: <one-sentence reason>

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 756fb50: Items 7-9 ask to delete the lock-ordering fix that buluoray's CHANGES_REQUESTED asks to keep, and the lane's own clearance test is unsatisfiable: no revocation interleaving separates the two lock orders -- one landing after the recheck is admitted under both, one during the in-lock STS call is rejected by the unlocked recheck and admitted by the locked one. Item 1 is the draft RFC, tracked outside this PR. Overriding the lane only; the human review stands.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 756fb50ba56c16be693ec0e6151aef74b8bd187b.

Items 7-9 ask to delete the lock-ordering fix that buluoray's CHANGES_REQUESTED asks to keep, and the lane's own clearance test is unsatisfiable: no revocation interleaving separates the two lock orders -- one landing after the recheck is admitted under both, one during the in-lock STS call is rejected by the unlocked recheck and admitted by the locked one. Item 1 is the draft RFC, tracked outside this PR. Overriding the lane only; the human review stands.

This decision applies only to this commit. A new push requires a new judgment.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

@buluoray your CHANGES_REQUESTED is recorded against cf0e7cc7ca and is now the only thing holding this board. I am not dismissing it -- asking for a re-look, because the code it was filed against is gone.

Your finding was right, and this head is what it produced. _state_lock no longer waits for the sidecar file lock while holding _run_lock: it takes the FILE lock first and _run_lock second, and the module now states one total order as an invariant -- _RETENTION_GATE -> state sidecar FILE lock -> _run_lock -> leaf locks, nothing may hold _run_lock while waiting for the file lock. The doc sentence you flagged as false under contention is gone.

The vacuous test is covered by name. test_a_status_read_is_not_blocked_by_a_writer_parked_behind_an_upload is the contending-writer case the old test could not see, beside test_a_status_read_is_not_blocked_by_an_in_flight_upload.

One correction to the minimal fix you sketched: :2741 does not take _run_lock alone. It reaches the file lock through _record_run_locked -> _locked_state_update -> _state_lock, and :1316 does the same, so reversing :318 by itself would have deadlocked both against the upload. The outer _run_lock came off both instead. The _run_sequence bump stays under _run_lock alone because that section cannot park and (pid, sequence) is the identity the compare-and-set in #12381 reads; test_the_run_sequence_is_bumped_under_the_run_lock pins that.

Head 756fb50ba5: 79 green, 11 skipped, 0 red, PR Readiness passed. First Principles carries a recorded human override on this SHA, for asking to delete the very fix you asked to keep. That override does not touch your review.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source-verified at 756fb50ba56c16be693ec0e6151aef74b8bd187b (read through the refs/pull/12178/merge tree). The blocker I raised at cf0e7cc7 is closed. Dismissing my earlier request for changes and approving.

What I asked for was that the network PUT stop happening under _run_lock, while the consent read stays inside a lock that orders against the setter. Both hold at this head:

  • _state_lock() still takes _run_lock, and now takes it AFTER the file lock (src/kiro_crew/apps/builtins/aws_control/backend/backup.py:355-362). That reversal matters on its own: the old order parked a contending writer on the file lock while holding _run_lock, which is what queued last_runs() behind an upload even though the upload gate held no _run_lock itself.
  • The Layer B upload gate takes the new _upload_lock() (:3938), whose body is open_lock_file + file_lock and nothing else (:411-415) — no _run_lock in the function at all; the other _run_lock mentions in that range are docstring prose. So storage.put_file (:3960) runs without _run_lock, and the _run_lock-only status read (last_runs) is no longer behind it.
  • No TOCTOU traded in: _authorize_upload (:3942) and the withdrawal refusal if layer_b and not sessions_layer_b_enabled(account) (:3952) are both inside the same _upload_lock() block as the PUT, with the authorize call moved ahead of the recheck so the whole decide-then-upload span is one critical section.
  • Both locks use _state_path().with_suffix(".lock") (:354, :411), the same sidecar, so exclusion against the setter (set_sessions_layer_b → _locked_state_update → _state_lock) is real rather than two locks over one document.
  • Relative to main this diff ADDS a lock where there was none: on main the sessions Layer B upload ran _authorize_upload then put_file with no state lock at all. The - lines are the acquisition-order refactor, whose body relocates intact.

The effect is pinned, not just the call: test_a_status_read_is_not_blocked_by_an_in_flight_upload (test/test_aws_control_backup.py:782) runs a real last_runs() from inside put_file; test_the_upload_holds_the_setter_lock_across_the_permission_read (:711) and test_a_status_read_is_not_blocked_by_a_writer_parked_behind_an_upload (:829) cover the other two halves. test_aws_control_backup.py + test_aws_control_nightly_sessions.py are 274 passed at this head.

The lock-order comment block above _state_lock enumerating every site that takes both locks is the right shape for this — it puts the one acquisition order where a future holder will look.

Not verified: I did not mutation-test the lock guards; the three concurrency tests reproduce the old-order hang directly and pass here. The Design lane's 🟡 is pinned to an older sha and is advisory either way.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 756fb50 with Windows compatibility as the primary lens. Both new lock helpers (_state_lock, _upload_lock) reuse platform_compat.open_lock_file + file_lock, which are Windows-correct (byte-0 msvcrt.locking, no truncation, no pre-primed lock byte); the route handler validates bool rather than coercing. run_sessions_backup refuses on Windows via _CAN_PIN_TRAVERSAL (POSIX-only openat), so the Layer B upload path itself never executes there - the gate, withdrawal recheck and lock plumbing that do run on Windows are the setter route and run records. buluoray's lock-scope finding is resolved: the upload holds the sidecar file lock only, deliberately without _run_lock, matching _delete_under_the_retention_gate; buluoray has since approved at this head. The grant lives in backup.json under the fenced state leaf, not agent-writable config.json. Local run on native Windows (worktree src): test_aws_control_backup/routes/app.py 520 passed, 30 skipped, 0 failed; the 19 Layer-B skips (incl. the four lock-scope ordering tests) are gated on _CAN_PIN_TRAVERSAL and are covered by the Linux/macOS shards - noting for the record that Windows CI cannot catch a regression in that fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sessions archive carries Layer B with no opt-in gate (asymmetry with file-export triple gate)

3 participants