Skip to content

feat(aws-control): curate a crew into a portable bundle - #9213

Merged
iamwhatever merged 2 commits into
mainfrom
feat/crew-bundle-builder
Sep 11, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/crew-bundle-builder

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

What this is

The packaging half of deploying a crew somewhere other than its owner's machine:
turn a local crew into a four-entry bundle (agent.json, mcp.json,
manifest.json, skills/) whose manifest carries a digest over every file it
describes.

Pure Python. No cloud dependency, no Docker, nothing to configure. This is the
first of three pieces; the container runtime and the Fargate backend follow
separately.

There is no in-repo design document for the bundle format yet, and no in-repo
caller of python -m packaging.build. The RFC on the Fargate backend covers the
compute side and does not mention bundling, so it is not the provenance for this
change and is not cited as such. What the format has to satisfy is written down in
this module's own docstrings and pinned by its tests; a design document and the
driver that consumes the bundle come with the piece that needs them.

Why it is 39 files

34 of them are tests: 31 new files under the packaging tree and three existing
repo-level suites this change touches. 363 test functions, run against a real
temporary filesystem rather than a dry-run stand-in. Most of them cover refusals,
because this module's job includes three recursive deletes and every one of them is
a way to destroy something the operator owns.

Guard What it stops
Three ownership rules on --out and <out>.previous, shared by one function A directory whose files use bundle names but were written by something else
A marker this build writes, for <out>.staging Staging is incomplete by construction so it has no digest to verify. Ownership is proved by the marker instead
Promotion moves the previous bundle aside rmtree then rename is two operations, and a failure between them lost the old bundle with nothing to restore
The prompt fence runs before resolution Resolving a Windows UNC path IS the outbound SMB probe, so a fence after it is too late
A shape this build never writes is refused is_file() is False for an empty directory, a FIFO, a socket and a link to a directory

No caller lands here

build.py exposes main() and nothing in this diff invokes it. That is
deliberate: the driver that calls it belongs with the Fargate backend, and landing
the builder on its own keeps that review about the cloud path instead of about
bundle semantics. Reviewing this one needs no AWS account and no image build.

Filesystem access layer

build.py walks, reads, writes, copies, renames and deletes against operator- and
author-supplied paths on POSIX and Windows. Rather than harden each operation as a
reviewer reaches it, every operation routes through one access layer with four
properties.

No-follow reads on both platforms. _walk_no_reparse (os.scandir, never
descends a reparse point) for walks, _read_text_nofollow / _read_text_openat for
reads. On POSIX O_NOFOLLOW refuses a reparse final component at the open; where
O_NOFOLLOW is 0, the reader lstat-refuses a reparse point before opening, so a
junction to a UNC share is never followed into an SMB or NTLM probe.

Mutation through a descriptor, in a run-private location. The <out>.previous
recursive delete goes through _purge_via_private_aside: rename the tree into a
directory this build alone created, verify ownership on the moved entry, which is the
exact inode the rename captured and cannot now be swapped, and delete only if it
passes. A tree
swapped in before the rename is renamed back untouched. The inode verified is the
inode deleted, so the check-to-rename window is closed rather than narrowed.

Nothing is created or destroyed by name alone. Every file this module creates is
claimed with O_CREAT | O_EXCL, so the outcome distinguishes "this call created it"
from "it was already there" without an is_file() check that a racer can slip
between. write_plan returns which of the two happened, because re-running plan on
an already-planned crew is a normal thing to do while truncating a concurrent
writer's plan is not. Reads of anything this module did not just create go through
hooks.safe_read_file_bytes_nolink, the one authority that owns the sensitive-path
verdict, the descriptor fstat and the hard-link refusal, across 21 call sites,
composed rather than reimplemented.

Explicit inclusion policy, failing closed. A file selected for a bundle that
cannot be read as scannable UTF-8 refuses the build and names the file. A directory
that exists but cannot be enumerated refuses rather than reading as empty. An
existing report or signed plan that cannot be read refuses rather than being treated
as absent, which would delete the report on rollback or overwrite the plan with stale
bytes. Absent, unreadable and unscannable never read as "not selected". The
standalone sensitive-path floor also catches a credential file by name (.env,
*.pem) and not only a credential directory, so --allow on a credential leaf fails
closed when the shared validator is unavailable.

Checks

59 check lanes green at 8cd7599ee, nothing failing and nothing running. That
includes all nine test lanes (four Linux shards, four Windows shards and the
namespace sandbox), plus both AI review lanes, the coverage gates and the
cross-platform portability, inclusive-language and first-principles gates.

packages = find: discovers the new package, so nothing in setup.cfg or
MANIFEST.in needs to change: the tree is Python only, and the non-Python payload
that does need a MANIFEST.in entry (templates, shell, Dockerfiles) arrives with the
pieces that own those files.

Scope note

Two tests that pinned this module's O_NOFOLLOW opener against the container
sidecar's deliberately duplicated copy are held back for the change that brings that
copy. The property is "the two agree", which cannot be asserted while only one
exists, and asserting it here with a skip would be a test that passes for the wrong
reason.

The builder is POSIX-only for now: _refuse_without_nofollow_primitive() refuses at
three entry points rather than running without the primitive its guarantees rest on.
That hold is tracked in #9496.

@chenmingwei23
chenmingwei23 requested review from a team and buluoray September 7, 2026 09:09
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 641cd153377a4c841765c8ca7f61ec1c5ffdb72d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All verification is done. Emitting the review.

First-Principles-Verdict: CONCERNS

A zero-caller builder citing an out-of-repo contract, plus two riders — a repo-wide Windows gate change and vendor-token constants — whose consumers this PR itself disables or contradicts.

Not justified as shipped

  • 1 — zero consumers: no in-repo caller of python -m packaging.build (grepped packaging\.build in src: only the new tree; other hits are the unrelated packaging/build-desktop.sh), and its cited authority (PACKAGING-CONTRACT.md, share-my-crew/) exists nowhere in this repo (glob/grep: 0 files).
  • 6 — undeclared rider: the description never mentions hooks.py; the "would slip the fence" defect only exists after this PR's own is_unc_shape exception opens the gate — on base \\?\C:\... was refused outright — and the motivating Windows walk is disabled (_refuse_without_nofollow_primitive, Windows no-follow read retains a residual check-then-open window in the crew packaging builder (shared primitive shipped; builder has not adopted it) #9496).
  • 5 — one consumer, generalized: VENDOR_TOKEN_PATTERNS has one real consumer (build.py:202), which restates the whole tuple as a literal fallback (build.py:204) — the exact per-site copy credential_patterns.py's own comment forbids — while deploy/scan.py + security.get_credential_patterns() remain unconsumed on the premise "NOT importable in this venv", contradicted by build.py's mandatory kiro_crew.hooks/security imports (scan.py imports only kiro_crew.security).
  • 10 — rides along: test_agent_home_isolation.py:732-735 comment starts mid-sentence and describes an _ALLOWED entry that is not in the set.

What this change ships

Inventory (10 items) — 6 justified

Intent: package a local crew into a digest-verified bundle so it can be deployed off its owner's machine — ADDITION.

  1. New python -m packaging.build command builds a four-entry bundle and prints SMC_BUNDLE_JSON= — zero consumers
  2. plan verb writes a signed deny-by-default review template with per-item sha256 pins — justified
  3. Refusal-only guards around the three recursive deletes (ownership, staging marker, aside-promotion) — justified
  4. Builder refuses to run on Windows until Windows no-follow read retains a residual check-then-open window in the crew packaging builder (shared primitive shipped; builder has not adopted it) #9496 — justified
  5. Seven vendor-token regexes + VENDOR_TOKEN_PATTERNS in credential_patterns.py — one consumer, generalized; duplicate of src/kiro_crew/deploy/scan.py
  6. Windows \\?\C:\... paths, previously refused everywhere hooks gates them, now fold and read — undeclared
  7. file:// prompt references are inlined into the bundle (feat(aws-control): read an external prompt reference safely #9327) — justified
  8. Three benign-spawn allowlist entries for the new CLI tests — rides along
  9. compileall syntax test now drops bytecode in tmp, not the checkout — justified
  10. Orphaned comment fragment in test_agent_home_isolation.py — rides along, zero effect

Watch

  • Bundle/plan formats and the SMC_BUNDLE_JSON stdout contract land pinned only by their own tests; the driver that depends on them is deferred. Clears when: the deploy-driver PR or a tracking issue for it is linked.
  • The hooks.py loosening is justified as "the lexical twin of the readlink-target fold" — symmetry — and its real consumer is the Windows builder this PR disables. Clears when: Windows no-follow read retains a residual check-then-open window in the crew packaging builder (shared primitive shipped; builder has not adopted it) #9496 lands, or the fold ships with it.
  • "That module is NOT importable in this venv" (build.py, re kiro_crew.deploy.scan) vs "the build FAILS CLOSED... when either is unimportable" (same docstring, re kiro_crew.hooks/security): both cannot hold. Clears when: the author names a venv where hooks imports and deploy.scan does not, or the builder consumes get_credential_patterns().

Subtractions

[FIRST-PRINCIPLES-REVIEWED] 641cd15

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 641cd153377a4c841765c8ca7f61ec1c5ffdb72d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All the evidence I need is gathered. Emitting the review.

Design-Verdict: CONCERNS

Sound, staged builder — but a core security-gate change rides along undocumented, and the format's normative contract lives outside this repo.

Watch

  • hooks.py changes is_unc_shape/validate_file_path semantics for every caller — \\?\C:\... paths previously refused fail-closed at the UNC gate are now folded and admitted as local. The reasoning in-code is careful, but the PR description never mentions it, so a platform-wide loosening of a keystone gate is being reviewed under "adds a builder with no caller." Cause: the builder hit extended-length readlink forms on Windows → fix landed in the shared gate → consequence: gate semantics shift for the dashboard/tool read path with no reviewer told to look.
    Clears when: the fold is named and justified in the PR description (or split into its own fix PR with the +97 hooks tests).
  • _default_kiro_home/_default_config_dir re-derive the data home by hand while the module requires kiro_crew installed (fails closed without hooks/security), so config.paths is always importable — the standalone rationale contradicts the fail-closed rule. The in-code comment records this duplicate already shipping wrong once (~/.kirocrew), silently omitting all skills; when config/paths.py conventions move next, this copy stays behind and the bundle curates the wrong home.
    Clears when: defaults import from kiro_crew.config.paths, or a test pins the two derivations equal.
  • The module's authority — PACKAGING-CONTRACT.md T1, share-my-crew/..., the SMC_BUNDLE_JSON protocol — exists nowhere in this repo, so neither contributors nor the two follow-up PRs can check conformance of a versioned, digest-pinned persisted format against anything public. Acknowledged in the description, but the deferral means the container-runtime PR reviews against an invisible contract.
    Clears when: the bundle-format spec lands under docs/system-specs no later than the first consumer PR.

Suggestions

  • Make the invocation python -m kiro_crew.apps.builtins.aws_control.crew.packaging.build: since kiro_crew must be installed anyway, this removes both the collision with PyPA's ubiquitous packaging package and the cwd-at-sys.path[0] shadowing that the spawn-audit comment currently defends as load-bearing.

[DESIGN-REVIEWED] 641cd15

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 641cd153377a4c841765c8ca7f61ec1c5ffdb72d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 641cd15

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 641cd153377a4c841765c8ca7f61ec1c5ffdb72d: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 7, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-bundle-builder branch from 38dacc8 to f0e65a6 Compare September 7, 2026 09:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-bundle-builder branch from f0e65a6 to 5b71879 Compare September 7, 2026 09:26
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 641cd153377a4c841765c8ca7f61ec1c5ffdb72d — this comment is updated in place on each push.

Review details

No findings.

The load-bearing candidate (the digest-order mismatch, candidate 1) depends entirely on PurePath.__lt__ comparing component tuples (_cparts), which is the pre-3.12 behaviour. This project pins requires-python = ">=3.12" (both pyproject.toml and setup.cfg), and on 3.12+ PurePath.__lt__ compares the full normalized string, so bundle_digest's Path-object sort and _inspect_captured_tree_fd's row[0] string sort produce identical orders — the candidate's own evidence confirms they coincide on 3.12. The fd-inspection path is further POSIX-only (_dir_fd_supported() is False on Windows, so it refuses before the compare), removing the Windows separator/case divergence too. The precondition (Python ≤3.11) does not occur in a supported runtime, so (a) fails.

The remaining candidates were self-flagged low confidence and do not clear the bar: candidate 2 is a docstring gap with no reachable non-regular leaf established; candidate 3 is operator-self-inflicted OOM on --allow (no boundary crossed); candidate 4 is the discouraged "another spelling reaches the fence" shape whose only outcome is reading the operator's own file (not shipped); candidate 5 is a benign O_TRUNC-identical double write the author itself assessed as non-defective. None re-derive (a)/(b)/(c) with an observable wrong outcome at 80+.

[OPUS-REVIEWED] 641cd15

Verdict parsed from the review's SHA-scoped output markers for commit 641cd153377a4c841765c8ca7f61ec1c5ffdb72d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 641cd153377a4c841765c8ca7f61ec1c5ffdb72d: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 5b71879: Provider REFUSED (not a finding); gate says re-runs do not clear this class. Diff is one pure-Python package + 17 tests, no AWS/Docker/network, main() uncalled by design, security guards each pinned by a mutation test, 137 pass, Opus did not block.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-bundle-builder branch from 94c7062 to e2c2515 Compare September 7, 2026 21:25
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 7, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-bundle-builder branch from e2c2515 to 6fba123 Compare September 7, 2026 21:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Rebuttal: concurrent staging claim, build.py staging.mkdir(parents=True)

OUT OF SCOPE, with the evidence already in this branch rather than an argument.

A translation of FileExistsError was written here and then removed, because a mutation showed it was unreachable: deleting the try/except left all 242 packaging tests passing.

Every way the staging path can already exist is refused above the claim, each with a message that names the cause better than the mkdir could:

  • a link or junction at the path
  • a path that exists and is not a directory
  • a tree without this build's own marker (the staging path ... already exists and this build did not create it)
  • a tree holding files this build does not own

The case those checks were thought to miss is an empty directory, on the reasoning that exists() and not is_dir() is False for one and an empty tree holds no unowned files. It is refused too, by the marker check. That is pinned by tests/test_sensitive_source_and_report_identity.py::test_an_empty_staging_directory_is_refused_by_the_marker_check, which asserts the refusal names this build did not create and fails if the marker check ever moves below the claim.

What remains is a race between the marker check and the mkdir in a second process. No test on one process can enter it, so a guard there is an error path nothing can redden. The repository's own rule is that such a guard is removed and the ordering pinned instead, which is what this branch does; the reasoning is recorded at the call site.

exist_ok stays False deliberately: creating the directory is how the build CLAIMS the path, and exist_ok=True would let two builds share one staging tree. That is pinned separately by test_the_claim_is_still_the_mkdir.

@bolichen97

Copy link
Copy Markdown
Contributor

@chenmingwei23 From the open-PR relationship audit (audited at 0636f23). This PR overlaps #9223 in exactly two files, and the two are complementary rather than competing, so both should land.

Where they overlap. Both PRs add src/kiro_crew/apps/builtins/aws_control/crew/__init__.py as a new file with byte-identical content (same blob index b1e16bed3), so it is an add/add collision for whichever lands second. Both also edit test/test_spawn_audit.py, but in disjoint regions: this PR appends three packaging CLI entries to BENIGN_SPAWNS, while #9223 adds _is_container_image_asset, two walk exclusions, and test_container_image_assets_are_not_imported.

What differs. This PR is the bundle producer (crew/packaging/build.py). #9223 is the image and runtime that consumes the bundle: its runtime declares BUNDLE_ENTRIES as manifest.json, agent.json, mcp.json and skills, its Dockerfile copies those three files, and it re-derives the digest skipping manifest.json, which is exactly this PR's bundle_digest contract. #9223 has no files under crew/packaging/, and nothing under crew/runtime/ imports crew.packaging, so there is no behavioural conflict.

Suggestion. Pick one owner for crew/__init__.py and have the other PR drop it instead of re-adding it. That docstring documents scripts/, templates/ and packaging/, and only packaging/ arrives with this PR, so if #9223 lands first the docstring points at nothing. Keeping the file here is the cleaner split.

Separately, #8470 duplicates all 17 files of this PR's packaging slice, 10 of them byte-identical. It should rebase and take this PR's build.py rather than land its own older copy.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 83eee305652c1af1b7a101fa92b3757159a6dc70; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 83eee305652c1af1b7a101fa92b3757159a6dc70: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 641cd153377a4c841765c8ca7f61ec1c5ffdb72d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

Turns a local crew into a portable, reviewed bundle: agent.json, mcp.json, skills/
and a manifest carrying the content digest. Deny-by-default -- nothing ships that a
signed curation plan did not select, and every string is scanned before it becomes
bytes.

Addresses the three blocking items from tech-lead review.

**The builder aborted every Windows build.** ``Path.write_text(text,
encoding="utf-8")`` leaves ``newline`` at ``None``, which translates "\n" to
os.linesep on write. The content pin compares ``_tree_hash`` (SOURCE bytes) against
``_staged_tree_hash`` (SHIPPED bytes), so an ordinary LF-authored skill hashed
differently once staged and the build refused with "changed while the bundle was
being written". ``bundle_digest`` runs over those same staged bytes, so the digest
was platform-dependent too.

Every ``write_text`` in the module now pins ``newline=""``, including the two whose
bytes are not hashed -- a rule with exceptions is one nobody can apply from the call
site. The suite could not catch this because it wrote its fixtures through the same
call, so both sides of the comparison moved together; the new tests write source
bytes with ``write_bytes``, and an AST tripwire holds the rule itself so the next
such call is covered rather than only today's four.

**skill_count undercounted nested ids.** A skill id is
``relative_to(skills_root).as_posix()`` and may nest, so ``aws/ec2`` and ``aws/s3``
are two skills under one top-level ``aws`` directory. Counting directories reported
1 for that pair, in the printed summary and in SMC_BUNDLE_JSON alike. It now counts
the ids the plan selected, which is the population ``_copy_skill`` was driven from.

**The prompt injection is deferred, not documented.** ``_inject_fingerprint_challenge``
prepended a ``[deployment verification]`` block to every deployed prompt, and its
only consumer is the deploy gate, which is not in this change. The whole fingerprint
path goes with that gate: the challenge, the injection, and the reported value, whose
purpose the content digest already serves here. What ships now is the prompt the
operator wrote, so a reviewer reading agent.json sees what will run. A source-text
test keeps the block from returning uncalled, and ``test_fingerprint.py`` travels with
the gate.

Verified end to end on a real crew: three skills including two nested, one MCP server
carrying a live-looking token. The token appears in no file in the bundle, the prompt
is byte-identical to the source, skill_count is 3, the skill bytes are unchanged with
no CRLF, and the manifest digest recomputes from the artifact.
The bundle builder inlines an agent's persona when the spec names it as
file://<path>, so a curated crew ships one self-contained agent.json.
Four shipped pptx_maker specs need this.

The read is routed to hooks.safe_read_file_bytes_nolink, which owns the
sensitive-path verdict, the fstat on the opened descriptor, the st_nlink
refusal and containment against within_root. Redirects in the chain are
judged before the path is handed over, because resolve() collapses links.

Refuses a NUL in the reference: the target comes from the crew's spec, and
a NUL-bearing string reaches a syscall as a bare ValueError. Checked on the
string, since Path accepts it and defers the error past every point that
could still name the reference.

Removes a local opener stack that duplicated hooks and had no production
caller, with the tests that pinned it. Scopes the byte ceiling to the
prompt read, so an oversized agent spec or plan is not refused by a limit
named for prompts. Restores nine POSIX-only markers dropped in a merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants