Skip to content

fix(publish): upload the personal-drive body from an opened descriptor - #24733

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/personal-drive-body-fd
Oct 28, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/personal-drive-body-fd

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The personal cloud drive uploads the exact bytes it staged and checked, never a file name the AWS CLI re-opens afterwards.

PersonalDriveProvider._put passed the staged artifact to s3api put-object as --body <path>. Its callers (publish, push_version) read the payload and take its content digest from the bytes on disk, but then hand the CLI a name — so the CLI re-opens that name after this process measured it. Between those two moments a process running as the same user can re-point the name, and the bytes that leave are not the bytes that were staged.

Why it matters

The upload is off-host and unattended, with no recall. A substituted body means an object in the user's own bucket, under the user's own key, carrying content the product never produced. Nothing downstream can tell: the digest the provider stored as object metadata was taken from the staged bytes, not from what the CLI re-opened.

Not a goal

  • Does not touch cloud/source.py. That site is already safe on origin/main: it stages the tarball in a launcher-owned private directory and pins the upload with --checksum-sha256 (digest taken at write time) + --expected-bucket-owner, which closes the same window downstream of the CLI's open (landed in fix(cloud): stage the launch source tarball in a launcher-owned directory #13640). Converting it to the descriptor seam would be a redundant refactor of already-safe code.
  • Does not add a new descriptor seam or refactor the shared engine. It reuses the stdin_fd parameter already on deploy.engine.run_aws / _checked.

What changed (motivation -> approach -> change)

  • Symptom: _put hands --body <path>, so the CLI re-resolves the name after the payload was read and digested.
  • Root cause: the body is identified to the CLI by name, which is re-resolvable, rather than by a descriptor this process already opened and holds.
  • Change: _put now opens the path once (O_NOFOLLOW, proven a regular file with S_ISREG), hands that descriptor to engine._checked(..., stdin_fd=fd), and spells --body as /dev/stdin, so the CLI resolves no path at all. The descriptor is closed in a finally. Windows has no /dev/stdin, so there _put keeps the by-name body — the same platform split the aws-control upload seam already draws, and for the same reason.

The seam this reuses was built and merged for the backup archive in #13272 (storage.put_file body_fd, run_aws stdin_fd, --body /dev/stdin); this change routes one more product site through it.

Why this change

#13370 asked to fix the remaining by-name s3api put-object sites. Read against current origin/main, two of the three originally-named sites are already handled: put_file and cloud/source.py both gained protection (the archive descriptor seam, and #13640's private-staging + checksum pin respectively). personal_drive._put is the one product site still passing a re-resolvable name with no pin, so it is the whole of what this PR changes. The fix is opt-in-free and changes no default, UI, or output for any user — the uploaded bytes and stored metadata are identical; only how the CLI is told where to read the body changes.

Backwards compatibility

Compatible: _put's signature and the object it writes (key, content-type, cache-control, sha256 metadata) are unchanged; only the body-delivery mechanism changes. Removes nothing: no path, record, key, validator, or public function is removed or tightened.

Tests

test/test_publish_personal_drive_body_fd.py (POSIX-gated):

  • test_publish_hands_the_body_over_as_a_descriptor_not_a_name — the recorded put-object uses --body /dev/stdin over a real descriptor.
  • test_a_swapped_name_cannot_change_the_uploaded_bytes — the stub plays the CLI, re-points the artifact name to different bytes at open time, and reads what actually reaches it through the handed-over descriptor; the original bytes are what leave.

Both fail on origin/main (which passes --body <path> with no descriptor) and pass on this branch. The shared test helper and four local _checked stubs are widened to accept the stdin_fd keyword the merged seam now passes.

Proof: PROVEN (both new assertions fail with the production hunk reverted) (head 46f9424)

Manual verification

N/A — unit coverage drives the real _put path with the engine seam stubbed; no AWS is reached.

Screenshots / video

N/A — no user-visible frontend surface is touched.

Related Issues

Fixes #13370

Sibling sweep

Swept: git grep -nE '"--body"' -- src/kiro_crew (every product s3api put-object body site)

  • Sibling: src/kiro_crew/apps/builtins/aws_control/backend/storage.py (put_file) -- not affected -- archive callers already hand over a descriptor via body_fd; bare-name bodies there are the label sidecar / library push, out of this issue's scope.
  • Sibling: src/kiro_crew/cloud/source.py (upload_source) -- not affected -- already pinned by --checksum-sha256 + private staging on main (fix(cloud): stage the launch source tarball in a launcher-owned directory #13640).
  • Sibling: website/electron/scripts/sign-windows.js -- not affected -- release-signing host, guarded by a sha256 compare either side; out of product scope.

Pattern harvest

Not generalizable: the remaining product body sites were enumerated in the sibling sweep and are each already handled or out of scope; this is the last unconverted one, not a recurring class.

Checklist

  • At most two commits (one here), Conventional Commits title
  • Existing tests pass and new tests added
  • Self-review completed; follows project style (black/isort/flake8/mypy clean)
  • Documentation updated (N/A — no doc-visible behaviour change)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The design holds up: uploading the in-memory payload means no name is resolved a second time, and the Windows checksum pin copies the pattern cloud/source.py already uses. The problem is the description, which describes a different implementation.

Notes

  • The description doesn't match the diff. It says _put "opens the path once (O_NOFOLLOW, proven a regular file with S_ISREG)" and that on Windows "_put keeps the by-name body". The diff does neither. On POSIX it writes payload into a tempfile.TemporaryFile and passes that file's descriptor as stdin. On Windows it writes a private named temp file and adds --checksum-sha256. Rewrite "What changed" and "Tests" to describe the code that ships. The listed test names are also wrong: the tests are test_a_file_swapped_after_the_read_cannot_change_the_uploaded_bytes and test_windows_pins_the_named_body_with_a_checksum.
  • The Windows branch now sends --checksum-sha256 on every upload. If S3 rejects a mismatch, that upload fails where it used to succeed. This change isn't mentioned anywhere in the description, so state it, and say that the flag works without --checksum-algorithm just as in cloud/source.py.
  • The description says "Manual verification: N/A", but the Windows branch has never run against a real AWS CLI. Windows CI only runs it against the stubbed engine. One real upload through the Windows branch would cover that gap.

[DESIGN-REVIEWED] fde5fac

Description read

[DESCRIPTION-READ] fb33469dcb9c494a935fa59be8640537e094f11e9b4938416fd42a62f57b72a2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The Windows branch ships a temp copy plus a checksum pin, but the description says Windows "keeps the by-name body".

Notes

  • Item 2 — undeclared, oversized: the description says "Windows has no /dev/stdin, so there _put keeps the by-name body". The diff instead adds tempfile.NamedTemporaryFile(prefix="kirocrew-drive-body-", delete=False) plus --checksum-sha256. The checksum alone makes S3 reject swapped bytes. The temp copy is still a name the CLI re-opens, so it removes no harm the pin does not already remove. Subtraction: delete the Windows NamedTemporaryFile/os.unlink block and send the caller's file_path with the --checksum-sha256 pin.
  • Item 1 — undeclared mechanism: "What changed" says _put "opens the path once (O_NOFOLLOW, proven a regular file with S_ISREG)". The diff opens no path; it streams the in-memory payload through tempfile.TemporaryFile. The test names given in the description also differ from the ones the diff ships.

What this change ships

Inventory (4 items) — 2 justified

Intent: make the personal drive upload exactly the bytes it read and digested, not a file name the CLI could find swapped by the time it opens it — FIX (#13370).

  1. On POSIX the drive upload now sends the already-read bytes through stdin (--body /dev/stdin), never the artifact's name — undeclared mechanism (described as an O_NOFOLLOW open of the path)
  2. On Windows the upload now goes through a private temp copy and is pinned with --checksum-sha256 — undeclared, oversized
  3. The put-object argument list moved into a _put_args helper — justified
  4. Test stubs now accept the stdin_fd keyword, and new tests cover the swapped-file race and the Windows checksum pin — justified

[FIRST-PRINCIPLES-REVIEWED] fde5fac

Description read

[DESCRIPTION-READ] b165295fefaa763d3f473a7bc08d0ec8c9f8b719fe8f3462c0d8c1e4157d08ed

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fde5fac

Verdict parsed from the review's SHA-scoped output markers for commit fde5fac748d3b7850db65f02dcc7f0aafc51a4e1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fde5fac

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fde5fac748d3b7850db65f02dcc7f0aafc51a4e1 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the fix/personal-drive-body-fd branch from 46f9424 to f7ce12a Compare October 27, 2026 05:51
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Thanks — the blocking finding was right, and it is fixed in f7ce12a7e1.

The window was moved, not closed (First Principles, Opus 5.5, GPT 6.1). The previous head re-opened the artifact by name inside _put, after _read_payload had already read and digested it and after several AWS round trips (_live_account, _ensure_drive/_require_drive, _head). A same-user replace in that window was opened by _put, so substituted bytes shipped under the original digest.

Fix: _put no longer takes a path and never re-reads the name. Its callers already hold payload (the bytes they read, digested and size-checked), so _put now takes payload: bytes and uploads exactly those — on POSIX via an anonymous in-memory file (memfd_create) streamed as --body /dev/stdin, on Windows via a freshly minted private temp this method owns and removes. There is no second resolution of any name, so there is no window to swap.

FIFO / non-regular open (GPT 6.1). Also resolved by the same change: _put no longer calls os.open on a caller-controlled path, so there is no open to block on a substituted FIFO, and no S_ISREG gate to reason about. The bytes come from memory.

Duplicate constants (First Principles, item 4). Removed. _DESCRIPTOR_BODY and _CAN_PASS_BODY_DESCRIPTOR are gone from personal_drive.py; the POSIX branch uses the /dev/stdin literal and platform_compat.IS_POSIX directly.

Tests updated: test_a_file_swapped_after_the_read_cannot_change_the_uploaded_bytes now replaces the file immediately after _read_payload returns — the exact window named above — and asserts the original bytes are what reach the upload. Both tests fail on origin/main and pass on the branch.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/personal-drive-body-fd branch from f7ce12a to ed8a71e Compare October 27, 2026 06:01
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in ed8a71e64a.

macOS crash (First Principles, Opus 5.5, GPT 6.1). Right — os.memfd_create is Linux-only, and IS_POSIX is also true on macOS, so every publish/push there would have raised AttributeError before uploading. The POSIX branch now uses tempfile.TemporaryFile(), which is anonymous and portable (Linux and macOS): it is unlinked the moment it is created, so the bytes live only behind the descriptor and there is still no name on the filesystem to swap. The descriptor goes to the engine seam as stdin_fd with --body /dev/stdin, unchanged.

Windows arm description (First Principles, item 3). The docstring now matches the diff: on Windows (no /dev/stdin) the bytes are written to a private temp this method mints and owns, uploaded by that name, and removed. That temp is created here from payload and never shared, so it is not a name a same-user process can re-point — the same no-second-resolution guarantee, by a different mechanism.

The earlier core finding (upload pinned to the already-read payload rather than a re-read name) stays fixed. Tests unchanged and still prove it; 154 related tests + lint/type green.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/personal-drive-body-fd branch from ed8a71e to 7298f00 Compare October 27, 2026 06:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in 7298f008cd.

Windows body substitution (GPT 6.1). Correct — the Windows arm writes the payload to a named temp (kirocrew-drive-body-*) and hands that name to the CLI, so a same-user process can swap the file between the close and the CLI's open, and nothing tied the uploaded bytes to the checked payload. The Windows upload now passes --checksum-sha256 set to the base64 SHA256 of the in-memory payload, so S3 compares it against what it actually received and rejects a swapped body — the check lands downstream of the CLI's open, where a pre-flight inspection cannot reach. It is deliberately not paired with --checksum-algorithm, which would have the CLI hash whatever it re-opened and certify a substitution instead of catching it. The POSIX descriptor path is unchanged (no name, so nothing to swap).

New test test_windows_pins_the_named_body_with_a_checksum drives the Windows arm from any OS (forcing IS_POSIX false) and asserts the named body carries --checksum-sha256 equal to the payload's digest. It fails on origin/main.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/personal-drive-body-fd branch from 7298f00 to 088ca43 Compare October 27, 2026 07:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
The personal cloud drive's _put passed the staged artifact to the AWS CLI
as --body <path>, so the CLI re-opened that name after this process had read
and digested the bytes. A same-user process could re-point the name between
the read and the CLI's open, uploading bytes the product never produced --
off-host, under the user's own key, with no recall.

Route the body through the merged engine seam instead: open the file once
here (O_NOFOLLOW, proven a regular file), hand the descriptor to run_aws as
stdin_fd, and spell --body as /dev/stdin so the CLI resolves no path at all.
Windows (no /dev/stdin) keeps the by-name body, matching the aws-control
upload seam's own split.

Fixes #13370
@chenmingwei23
chenmingwei23 force-pushed the fix/personal-drive-body-fd branch from 088ca43 to fde5fac Compare October 27, 2026 12:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@chenmingwei23
chenmingwei23 marked this pull request as ready for review October 28, 2026 00:17
@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 28, 2026 00:17
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 28, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 28, 2026 02:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: uploads the personal-drive body from the bytes already read and digested instead of re-opening the file by name; adds tests.

@KiroCrewCaptain
KiroCrewCaptain merged commit cb21331 into main Oct 28, 2026
200 of 213 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/personal-drive-body-fd branch October 28, 2026 02:28
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two s3api put-object sites outside put_file still pass --body by name, so the CLI re-resolves it

2 participants