Skip to content

feat(scripts): publish the crew base image so a digest-pinned ref can exist - #11530

Merged
bolichen97 merged 1 commit into
mainfrom
feat/publish-crew-base-image-10901
Sep 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/publish-crew-base-image-10901

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

What is the problem?

cloud/fargate/taskdef.py refuses a movable tag and requires
<repository>@sha256:<64 hex> -- a revision is keyed on the image, so a tag would
let the content behind that key change after registration. A digest exists only
after a push.

The build lane already pushes. .github/workflows/crew-image-build.yml:172 runs
docker push, and its target is a throwaway registry on 127.0.0.1 started in
the same job. That is deliberate, and the lane says why in its own header:

WHY A LOOPBACK REGISTRY. The crew layer's base is referenced BY DIGEST, and a locally-built image has no repository digest -- only a push produces one. A throwaway registry on 127.0.0.1 produces a real one and needs no credentials, which is what keeps this lane runnable on a fork PR with no secrets at all.

So the machinery that produces a digest is complete -- the producers, the
recipe discovery in scripts/crew_image_build_plan.py, the base-then-layer
ordering, the read-back from the daemon -- and the digest it produces dies with the
job. Nothing can register a task definition against the crew image.

Why this issue matters to the user

A remote crew cannot be launched at all, and the missing piece is invisible from
the outside: the image builds, the build lane is green, and the digest it prints
looks exactly like a usable one. scripts/build_crew_base_image.sh states the
gap plainly in its header -- "It builds and verifies; it does not publish" -- but
nothing turns that sentence into a step a user can run.

How our fix solves it

scripts/publish_crew_base_image.sh is the durable half. It:

  • invokes the documented producer rather than docker build, so a published
    image cannot skip the wheel staging and the architecture cross-check a locally
    built one gets;
  • pushes, then reads the repository digest back from the daemon rather than
    parsing push output -- RepoDigests is empty until a push succeeds, so that read
    doubles as the check that the push landed;
  • refuses a reference that is not digest-pinned in the form taskdef.py
    accepts
    , at the push. Checked later, that refusal arrives far from the push
    that caused it;
  • --dry-run stops before the push and prints no digest, because a digest does
    not exist before one.

Three decisions worth stating, since each is a place the obvious shortcut is
wrong:

The repository has no default. The account and namespace belong to whoever
runs this. A default would either name someone's account in the tree or quietly
publish to the wrong one. Absent is refused, not guessed.

A repository carrying a tag or a digest is refused, not trimmed. repo:1.2 is
a tag or a port on a registry host, and the two readings are not distinguishable
here -- dropping the wrong half publishes somewhere the caller did not name.

It publishes the BASE image, not the per-crew layer. taskdef.py says "One
image serves every crew by design", so what a registry needs to hold is the base.
scripts/build_crew_image.sh bakes one crew's bundle into a layer, which is the
per-crew model that decision (#11408) replaces; publishing that would put one
image per crew in the registry.

Publishing stays a caller of the producer and is never wired into the fork-
runnable build lane.
Folding a push into the producer would put a
credential-needing step in the one place a fork PR runs. A test holds that lane
to naming no secret and to not invoking this script, because a lane that breaks on
a fork -- or skips -- reads exactly like one that passes.

What tests we did

test/test_crew_image_publish_contract.py -- 27 tests, 31 cases, static and
offline. Twelve of them execute the script or a POSIX tool beside it; every one is
bounded (REFUSAL_TIMEOUT_SECONDS = 15, IMPORT_TIMEOUT_SECONDS = 30), so a guard
a mutation removes cannot fall through into a real 1.88GB build -- it runs out of
time and reds. That is measured, not asserted: with the fully-qualified guard
widened to accept anything, the short-name case fails with
subprocess.TimeoutExpired ... timed out after 15 seconds rather than building.

Those twelve reach bash through the repository's own posix_test_shell
(test/conftest.py), which already resolves Git Bash on Windows and deliberately
avoids C:\Windows\System32\bash.exe -- the WSL launcher, which sits ahead of Git
Bash on PATH and, with no distro installed, prints its banner in UTF-16 and runs
nothing. A review round found that fixture after an earlier revision of this PR had
skipped the Windows shard instead; the skip and its marker are gone, so the
Windows shard now executes these tests rather than reporting them green without
running them.

posix_test_shell answers sh, and this script needs bash (env bash shebang,
${BASH_SOURCE[0]}, pipefail), so _shell_for_the_script takes a bash-named path
as it comes -- the Windows case -- and resolves bash by name otherwise. It asserts
rather than skips when a host has sh but no bash, so no new skip enters this file:
test_this_file_adds_no_skip_of_its_own walks this module's AST for
skip/skipif/xfail and holds that the only skip in the path is the shared
fixture's own. Because _shell_for_the_script is pure, the Windows branch is
exercised on Linux too
(test_the_shell_the_shared_fixture_resolves_is_taken_as_it_comes), and
test_the_shell_the_tests_run_the_script_with_actually_runs executes the resolved
shell inside a guarded test body -- nothing is spawned at import or collection
time, which test_importing_this_file_starts_no_process and
test_an_inherited_bash_env_hook_does_not_fire_when_this_file_is_imported pin from
both sides.

The operative path was run end to end against a loopback registry, with no AWS
identity.
A throwaway registry:2 on 127.0.0.1 needs no credential, which is
the same mechanism the build lane relies on. Pushing to
127.0.0.1:5000/example/kirocrew-crew-base passes the script's own tag guard --
the port colon is not in the final path segment -- and exercises the producer
invocation, the push, the RepoDigests selection by repository and the shape
check. Measured against the script as it ships here. This is the COMBINED
stdout+stderr capture
, which is how a human sees the run; every ==> line and
the digest: line come from stderr, and stdout is only the bare reference on its
own line, which is what a caller captures:

==> Pushing to 127.0.0.1:5000/example/kirocrew-crew-base:publish-20260917T161349Z-1684061-75b4f908
publish-20260917T161349Z-1684061-75b4f908: digest: sha256:7e8fdd1e58f4d0be172719d13291378e7ba25d93020bcefee142ca6b1caf68df size: 3259
==> Resolving the repository digest
==> Published
127.0.0.1:5000/example/kirocrew-crew-base@sha256:7e8fdd1e58f4d0be172719d13291378e7ba25d93020bcefee142ca6b1caf68df
publish exit=0

==> Dry run: built 127.0.0.1:5000/example/kirocrew-crew-base:publish-20260917T161519Z-1694523-78f58c31, stopping before the push
    no digest is printed, because a digest exists only after a push
dry-run exit=0

Four independent confirmations of that run:

  • the printed reference satisfies taskdef._DIGEST_REF_RE -- checked by importing
    the actual pattern, not by eye;
  • the registry lists exactly one tag,
    ["publish-20260917T161349Z-1684061-75b4f908"]. No latest, and no other
    shared name: the script writes none, which is the property
    test_no_shared_tag_is_written_at_all pins;
  • the registry's own Docker-Content-Digest for that tag equals the printed
    digest byte for byte;
  • the dry run added no tag to the registry -- the listing above is unchanged after
    it, and it printed no digest.

The registry container and the images it held were removed afterwards. No image
was pushed to any real registry and no AWS resource was created.

Every assertion was mutation-verified against the tree as it ships: the thing each
test exists to catch was broken, and the test required to go red.

# mutation caught by red
1 the repository gains a default required / no-default yes
2 the digest shape accepts 63 hex reference-shape yes
3 the resolved shell is second-guessed on Windows shell-taken-as-it-comes yes
4 the publish tag loses its random nonce unique-per-invocation yes
5 the digest is taken by position, not by repository selection-by-repository yes
6 the fully-qualified guard accepts a short name short-name refusal yes
7 a movable latest tag is reintroduced no-shared-tag yes
8 the dry run no longer exits before the push dry-run ordering yes
9 the digest guard is deleted tag-or-digest refusal yes
10 a 12-digit account id is hardcoded nothing-hardcoded yes
11 a concrete ECR namespace is hardcoded nothing-hardcoded yes
12 the digest shape accepts uppercase hex reference-shape yes
13 the push is removed entirely no-shared-tag / dry-run yes
14 the producer is replaced by a direct docker build caller-of-producer yes
15 the fork build lane names a secret fork-lane yes
16 the fork build lane invokes this script fork-lane yes
17 the committed exec bit is dropped committed-mode yes
18 the read-back inspects a shared tag unique-per-invocation yes
19 the base-image caveat is deleted names-the-base yes
20 the caveat is printed on stdout names-the-base yes

| 21 | a subprocess.run probe is put back at module scope | no-process-at-import + BASH_ENV sentinel | yes |
| 22 | a skip is added to this file | adds-no-skip-of-its-own | yes |
| 23 | step writes progress to stdout again | stdout-is-one-line | yes |
| 24 | an unreadable nonce is tolerated | nonce-stops-the-run | yes |
| 25 | an empty RepoDigests is tolerated | push-left-no-digest | yes |
| 26 | the Docker Hub refusal is deleted | docker-hub-refused | yes |
| 27 | the Docker Hub arm is placed after the *.* arm | accept-arm-ordering | yes |

27 mutations, 0 holes.

Row 9 was a real hole in the first draft, and it is the one worth reading. The
digest test asserted only that the refusal said must not carry. A digest
reference's last path segment also contains a colon, so with the digest guard
deleted the tag guard refuses that input -- with a different message the loose
assertion still accepted. The test passed while the guard it was written for was
gone. It now matches the specific message per case.

Rows 2 and 12 compare the script's POSIX ERE against taskdef._DIGEST_REF_RE by
verdict over a sample set, rather than comparing two patterns as strings: they
are written in different languages and can never be compared literally, but what
has to agree is which references they accept. The grep runs through the same shell
the script would get, so the comparison is not against whatever a host puts in
front of it.

Gates run locally on this tree: check_black_formatting.py, isort --check-only,
check_comment_history.py, check_brand_name.py, check_subprocess_encoding.py,
test/test_yaml_safe_loading.py, ruff and mypy on the changed file. Rebased onto
2627b0407c, so the Windows-run fixture fix (#11518) and the session-control fix
(#11593) are both in the base.

What the published digest is, said where it is handed over. A review round
established the chain: the base image carries no /app/crew-bundle, install_bundle
fails closed on a missing bundle directory before the backend starts
(bundle.py:546), and the directory cannot be redirected at runtime because
SMC_BUNDLE_DIR is in runtask.py's REFUSED_ENV. All three are true. So the
reference this prints is the INPUT to composing a crew, not a launchable artifact,
and the script now says exactly that as it hands the digest over -- on stderr, so
stdout stays one capturable line. Rows 19 and 20 pin both halves.

The composition step that closes the gap consumes this very digest
(build_crew_image.sh --base <repo@sha256:...>), and delivering a bundle to one
shared image rather than baking it into a per-crew layer is #11408's half, not
this one's: publishing a per-crew image is the model that decision replaces.

Any other suggestions on the work

Fixes #10901

@chenmingwei23
chenmingwei23 requested review from a team and buluoray September 17, 2026 12:13
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 892880796f8a977b7baac1db3bba33bd6e0e00f4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify one real ECR Public publish before anything depends on the digest — the end-to-end proof so far is loopback-only, and the script is inert until a maintainer runs it.

[DESIGN-REVIEWED] 8928807

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 892880796f8a977b7baac1db3bba33bd6e0e00f4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The checkout lock serializes only publishes, but the wheel-staging race it cites is the producer's — two plain builds, or a build beside a publish, still race.

Not justified as shipped

  • Item 4 — symptom-level: the lock's own comment names the cause ("the producer stages the wheel… into one FIXED path… build_crew_base_image.sh:29") yet guards it one caller up; the producer has 3 invocation paths (the CI plan, this publisher, a direct build_crew_base_image.sh run) and the shipped lock serializes exactly one pairing of them, leaving build-vs-build and build-vs-publish racing on the same vendor/ directory.
    Subtraction: move the mkdir lock into build_crew_base_image.sh beside the staging it protects and delete the publisher-side lock block (PUBLISH_LOCK acquisition, trap, and its three ordering tests shrink to one).
    Clears when: the lock is acquired inside the producer so every caller is covered, or a counted showing that no non-publish caller can run concurrently in a checkout.

What this change ships

Inventory (8 items) — 7 justified

Intent: let an operator publish the crew base image to a durable registry so a digest-pinned reference exists to register a task definition — an ADDITION.

  1. New scripts/publish_crew_base_image.sh publishes the base image and prints one digest-pinned reference — justified
  2. --dry-run builds and stops before any push, printing no digest — justified
  3. An absent, tagged, digested, short, or Docker-Hub repository is refused with a named reason — justified
  4. A second publish in the same checkout is refused via a lock directory — symptom-level: guards the producer's staging race at one caller of three
  5. stdout carries exactly the reference; all progress and the base-image caveat go to stderr — justified
  6. The fork build lane is now pinned to naming no secret and never invoking this script — justified
  7. No shared tag (latest or otherwise) is ever written; each run's tag is unique — justified
  8. New offline contract test file (27 tests) that the Windows shard executes rather than skips — justified

[FIRST-PRINCIPLES-REVIEWED] 8928807

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ human override accepted

Human judgment by @chenmingwei23 overrides the GPT 5.6 finding for 892880796f8a977b7baac1db3bba33bd6e0e00f4; the recorded reason is authoritative for this commit.

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 892880796f8a977b7baac1db3bba33bd6e0e00f4: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 892880796f8a977b7baac1db3bba33bd6e0e00f4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8928807

Verdict parsed from the review's SHA-scoped output markers for commit 892880796f8a977b7baac1db3bba33bd6e0e00f4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 892880796f8a977b7baac1db3bba33bd6e0e00f4: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from a86e2e8 to ca35b9e Compare September 17, 2026 12:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from ca35b9e to 0e5bc02 Compare September 17, 2026 12:44
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from 0e5bc02 to b1ab272 Compare September 17, 2026 13:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from b1ab272 to a5d7740 Compare September 17, 2026 13:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from 0da7cb4 to dc83932 Compare September 17, 2026 18:58
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 7 disposition: taken, and it was the real thing

Head dc83932dd9. The finding is correct, the adjudication's reading of it is
correct, and nothing about it was arguable. step() { echo "==> $*"; } wrote to
stdout, so a caller doing REF=$(scripts/publish_crew_base_image.sh --repository ...)
got this:

==> Building the base image through its own producer
==> Pushing to <repo>:publish-20260917T184657Z-2554138-68365b50
==> Resolving the repository digest
==> Published
<repo>@sha256:abab...

That is not a transcript I reasoned about. It is the assertion output from putting
step back on stdout after the fix. Every successful publish behaved that way.

Why the test suite said otherwise. test_the_published_digest_says_it_names_the_base_and_stdout_stays_one_line
sliced the script at step "Published" and scanned only the tail. Everything before
it was outside the window, so the script's own sentence -- "stdout stays exactly one
line, the reference" -- was a claim no test enforced. That is the shape this PR was
already warned about once, and it went unnoticed because the property was checked
where it was easiest to read rather than where it could break.

The fix. step, the producer invocation, docker push and the dry-run note all
write to stderr. stdout carries the reference and nothing else, and is empty on a
dry run. --help still prints to stdout, which is where explicitly requested help
belongs; it exits before anything is built or pushed.

Three tests, chosen so each closes a different way this could return

  1. The echo scan now covers the whole file and finds echo anywhere on a line, not
    just at its start -- which is what reaches the one inside the one-line step
    helper. It splits commands on semicolons OUTSIDE quotes, so a caveat containing
    prose punctuation is not cut in half. No shell needed, so this one holds on the
    Windows shards too.
  2. A scan of the commands that write output of their own: the producer and the push
    must both end in >&2. docker image inspect needs no redirect, because a
    command substitution is already the capture.
  3. An offline run of the whole success path. The script is copied so its
    SCRIPT_DIR lands beside a stub producer, docker is a stub on PATH, and stdout
    is compared byte-for-byte to the reference. The stub reports a different
    repository's digest first, so passing this also demonstrates the selection matches
    by repository rather than taking index zero. A dry-run variant asserts stdout is
    empty. Nothing builds, nothing pushes, and the same short cap a refusal gets is
    enough.

Test 3 is the one that would have caught this in the first place, and 1 and 2 exist
because a text property that holds on every platform is worth having beside it.

Matrix: 24 mutations, zero holes, zero missing anchors. Clean tree 28 passed, 0
skipped. Removing any one of the four redirects reds (2 to 4 tests each).

While measuring it I found two holes of my own and closed them rather than writing
them down: an unguarded empty nonce, and a tolerated empty RepoDigests. Both are
fail-closed paths no real registry produces on demand, so both now have offline
tests -- a stub od that prints nothing, and a stub registry that withholds
digests.

Round 6 is unchanged and still holds: importing the test file starts no process.
Nothing in the digest-resolution span was reopened. No /ai-review override
requested.

@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from dc83932 to d3d9e52 Compare September 17, 2026 19:47
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 8 disposition: taken, and it replaced my fix with a better one

Head d3d9e522ec. The finding is right, and the fix it names is better than what it
replaced, so the skip is gone rather than justified.

test/conftest.py:114-136 already ships posix_test_shell, which resolves Git Bash
on Windows by walking up from git on PATH and through the Program Files layout, and
which deliberately does not return C:\Windows\System32\bash.exe. That is the same
WSL launcher my os.name == "nt" -> None branch existed to dodge. The repository had
already solved this; I wrote a second, worse answer beside it.

What is gone

  • the os.name == "nt" -> None branch
  • the module-level BASH global
  • the executes_posix_shell skipif marker
  • test_a_posix_host_never_skips_the_tests_that_execute_the_script, whose whole job
    was watching a marker that no longer exists

Eleven tests now take publish_shell, which comes from posix_test_shell. The
Windows shard exercises the executable contract instead of skipping it, so the
ratchet tightens. Round 6 dissolves along with it: with nothing resolved at import,
there is no collection-time shell to worry about at all.

The one wrinkle, stated rather than hidden. posix_test_shell answers sh on a
POSIX host, and this script cannot run under sh -- its shebang is env bash, it
reads ${BASH_SOURCE[0]} (an array reference) and it sets pipefail. So
_shell_for_the_script takes a path that already names bash as it comes -- which
is the Windows case, and the case where a second lookup would find the launcher --
and resolves bash by name otherwise. A host with a POSIX shell but no bash at all
asserts, it does not skip. No new skip enters this file.

Three pins, because the shard that matters here is the one I cannot watch

  1. test_this_file_adds_no_skip_of_its_own walks this file's AST for any
    skip / skipif / xfail call and fails on one. The ratchet is now a test rather
    than a thing I remembered to do.
  2. test_the_shell_the_shared_fixture_resolves_is_taken_as_it_comes exercises the
    Windows branch from any platform, because the decision is a pure function of a
    path. It asserts C:/Program Files/Git/bin/bash.exe comes back unchanged, and that
    a POSIX sh is not accepted as a stand-in.
  3. Every path handed to bash is .as_posix() now, so Git Bash reads a forward-slash
    path instead of a backslash one it would treat as escapes.

Matrix: 25 mutations, zero holes, zero missing anchors. Clean tree 29 passed, 0
skipped. Re-looking-up bash instead of taking the resolved path reds; reintroducing a
skipif reds; a module-scope subprocess reds.

Not rebased, deliberately. origin/main is 5 commits ahead, but by content there
is no overlap: main touched neither of the two files here, and test/conftest.py
still defines posix_test_shell at the same lines. A rebase would only re-roll this
lane for nothing.

One thing I cannot measure from here and would rather name than discover quietly: the
Windows shard now RUNS the four offline stub tests, which write a stub docker onto
PATH and depend on Git Bash treating a shebang script as executable. If Windows reds,
that is the likely cause and it is a test-harness matter, not the script.

No /ai-review override requested for this one either.

@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from d3d9e52 to 58fedc1 Compare September 17, 2026 20:45
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The two Windows reds on d3d9e522ec, attributed and fixed

Head is now 58fedc19ce. Removing the skip in round 8 meant the Windows shard started
running these tests for the first time, and it found two things. Both are the test
harness meeting Git Bash, neither is a defect in the script.

Shard 7 is not this PR. test/test_session_control.py fails four tests there:
test_private_creator_cannot_change_memory[writer-canonical] and
[public-worker-canonical] with 'agent_store_mismatch' == 'memory_delegation_denied',
and test_create_refuses_unreadable_delegation_identity[OSError] / [ValueError] with
OSError: /private/identity/path and 400 == 403. PR #11534, an unrelated branch, reds
on the identical four with identical assertions. Compared by failing test name rather
than by lane, because a sharded lane does not name the same tests on two branches. This
diff is two files and mentions none of those symbols, so it is main's to fix and I am
waiting it out rather than folding anything in.

Shard 3 was mine, and what passed is what identified the cause. The three offline
stub runs went green on Windows. So a stub docker placed on PATH works there, and the
difference between those and the two failures is exactly the mechanism:

failure cause fix
the reference-shape probe called a valid reference invalid the ERE contains a colon, and Git Bash rewrites a colon-bearing argument as a path list, so grep received a pattern the script never wrote the ERE reaches grep through -f patternfile instead of argv
the empty-nonce refusal never fired Git Bash carries its own /usr/bin/od ahead of an inherited PATH entry, so the stub never ran and the real od answered the property is read from the script's text instead

Git Bash ships no docker, which is why shadowing that name works and shadowing a
coreutil does not. That is the whole difference between the tests that passed and the
one that did not, and it is why the nonce property is now textual while the other three
stay behavioural: a stub cannot reliably win a coreutil's name, and a test that only
works on one platform is the thing round 8 objected to.

The pattern file is checked for vacuity, because -f makes a silent always-match
possible. A trailing newline in that file is a blank line, and a blank pattern matches
everything, so the test asserts the file has none. Two deliberate breakages confirm the
comparison is still real: adding the trailing newline reds, and replacing the pattern
with . reds on registry.invalid/example/repo:latest.

Matrix: 25 mutations, zero holes. Clean tree 29 passed, 0 skipped, and every gate
green: comment-history, brand, black, subprocess-encoding, isort, flake8, ruff, mypy,
yaml.

No /ai-review override requested.

@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from 58fedc1 to 5969e77 Compare September 17, 2026 21:18
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 9 disposition: fixed, not overridden

Head 5969e775a2. The adjudication fenced this one and pre-drafted an override
rationale. I did not use it, because the fix is three words and the alternative leaves
the script's own header saying something untrue.

The header already argues this failure cannot happen:

A short name is rewritten by Docker into its own familiar form, so myuser/myrepo
comes back out of RepoDigests as something this script never received, the
exact-prefix match below then finds nothing, and the run dies AFTER a successful
push -- reporting failure for work that landed.

docker.io/acme/base is that same rewriting reached through a host that satisfies the
guard. It carries a dot, so *.* accepted it; Docker's reference parser reads
docker.io and index.docker.io as the official registry and drops the prefix, so
RepoDigests reports acme/base@sha256:.... The exact-prefix match finds nothing and
the run dies after the push landed. The finding is right, and the override rationale --
"only Docker Hub triggers it, and the failure is loud and recoverable" -- is also true.
It just argues for accepting a hole in a guard whose own paragraph says the hole is not
there. Rounds 7 and 8 were both a claim no test enforced; taking the override would
have made a third.

The fix. A refusal arm for docker.io|index.docker.io, placed BEFORE the arm that
accepts any dotted host, with a message that says what the read-back would have looked
for and names a registry that is not rewritten.

Pinned two ways, because arm ORDER is the part that can silently rot

  1. A behavioural refusal test for both spellings, asserting the run stops before
    anything is built. A push landing under a rewritten name is the whole point.
  2. An order assertion in the accept-arms test: arms.index("docker.io") < arms.index("*.*").
    The first matching case arm wins, so a refusal moved below *.* is unreachable and
    Hub is quietly accepted again. Deleting the refusal reds; moving it after the accept
    arm also reds, 3 tests each.

Matrix: 27 mutations, zero holes. Clean tree 31 passed, 0 skipped, every gate green.

Two notes on things not in this diff. The Backend Tests (Windows) (7) red is main's:
PR #11534, an unrelated branch, fails the identical four test_session_control.py tests
with identical assertions. And test/test_pod_e2e_video_guard.py carries its own
_usable_bash() beside posix_test_shell -- the same duplicate resolver round 8
objected to here, already on main. I left it alone rather than widening this PR.

… exist

taskdef.py refuses a movable tag and requires <repository>@sha256:<64 hex>,
because a revision is keyed on the image and a tag would let the content behind
that key change after registration. A digest exists only after a push.

The build lane already pushes -- to a throwaway registry on 127.0.0.1 that dies
with the job. That is deliberate: it needs no credential, which is what keeps the
lane runnable on a fork pull request with no secrets at all. So the machinery
that PRODUCES a digest is complete and the digest it produces is unusable, and no
legal task definition can be registered against the crew image.

This adds the durable half. scripts/publish_crew_base_image.sh takes the
repository to publish to, invokes the documented producer rather than docker
build -- so a published image cannot skip the wheel staging and architecture
cross-check a local build gets -- pushes, reads the repository digest back from
the daemon, and refuses a reference that is not digest-pinned in the form
taskdef.py accepts. --dry-run stops before the push and prints no digest, because
a digest does not exist before one.

The repository has no default. The account and namespace belong to whoever runs
this, and a default would either name someone's account in the tree or quietly
publish to the wrong one. A repository carrying a tag or a digest is refused
rather than trimmed: repo:1.2 is a tag or a port on a registry host, and the two
readings are not distinguishable here.

It publishes the BASE image, not the per-crew layer. One image serves every crew
by design, so what a registry needs to hold is the base; build_crew_image.sh
bakes one crew's bundle into a layer, which is the per-crew model that decision
replaces.

Publishing stays a CALLER of the producer and is never wired into the fork-
runnable build lane. A test holds that lane to naming no secret and to not
invoking this script, because a lane that breaks on a fork -- or skips -- reads
exactly like one that passes.

Fixes #10901
@chenmingwei23
chenmingwei23 force-pushed the feat/publish-crew-base-image-10901 branch from 9c54e85 to 8928807 Compare September 17, 2026 23:26
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

The inherited red cleared via #11593, and round 10 is fixed

Head is now 892880796f. Two separate things, both measured on this branch rather
than inherited from an older verdict.

1. The main-owned red is gone, named by test rather than by lane

On d3d9e522ec, Backend Tests (Windows) (7) failed on four tests in
test/test_session_control.py, none of which this branch touches:

test failure
test_private_creator_cannot_change_memory[writer-canonical] agent_store_mismatch where memory_delegation_denied was expected
test_private_creator_cannot_change_memory[public-worker-canonical] same
test_create_refuses_unreadable_delegation_identity[OSError] OSError /private/identity/path
test_create_refuses_unreadable_delegation_identity[ValueError] 400 where 403 was expected

That was attributed as main-owned with a control rather than asserted: PR #11534,
an unrelated branch on head 82458f235, failed on the identical four tests with
identical assertions. This diff is two files and mentions session_control,
memory_delegation and agent_store zero times.

#11593 (2627b0407c, "authorize a created child on the store it is bound to") is
the fix, and it is now in this branch's base. On the rebased head 9c54e850440,
Backend Tests (Windows) (7) concluded SUCCESS -- along with Windows shards 1,
2, 3, 4, 6 and 8. Shard 5 and the eight 3.12 shards were cancelled by the push
described below, so they are unmeasured on that head and are re-running on this one;
cancelled is not a pass and is not being read as one.

Windows shard 3 also concluded SUCCESS, which closes the other half: that shard
carried this PR's own two Windows failures after round 8 removed the skip, and the
stub-od and MSYS-argument fixes hold there now.

The rebase itself changed nothing in the diff. The patch hash and per-file numstat
were captured before and after and are identical: two files, +214 and +966, zero
deletions.

2. Round 10 disposition: taken, and it made a comment in this file true

The finding is a real race and it is in a span none of the earlier rounds touched,
so it is a fix rather than an escalation.

BLOCKING -- scripts/publish_crew_base_image.sh:135 -- Concurrent publishes race
on shared wheel staging. Anchor: residual/crash-data-loss-corruption.

Verified in the producer this script calls, which is main's file and is not in this
diff: build_crew_base_image.sh:29 fixes VENDOR_DIR at a single path inside the
checkout, empties it on entry (:71) and again in its own EXIT trap (:62-67),
copies its wheel in at :86, and the build reads that wheel through the BuildKit
bind mount at :95-98. Two publishes from one checkout therefore collide there, and
the loser's build dies on a missing wheel with nothing pointing at the run that
removed it.

The adjudication fenced it and pre-drafted an override rationale. I did not post it,
for the same reason as round 9: the per-invocation tag's own comment in this file
said the concurrent-publish class was handled, and that claim is exactly what the
race falsifies. Overriding would have left a comment describing a property the code
does not have -- the shape rounds 7, 8 and 9 each punished. The finding also came
back 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so an override could not have cleared it in any case.

The fix, in this PR's own file only. An atomic mkdir lock beside the staging
directory it protects, taken after validation and before the producer call, released
by an EXIT trap armed only once the lock is held -- a trap armed before the
acquisition would have a refused run delete the lock the holder is still using.
mkdir rather than flock, because macOS ships no flock.

It refuses rather than queues. Waiting would block on a multi-gigabyte image
build with nothing to say about how long, and refusing an input it cannot make safe
is what this script already does for a missing repository, a tag or digest suffix, a
short name and Docker Hub. A run killed outright leaves the lock behind, so the
refusal names the directory and the single rmdir that clears it.

Pinned three ways, and each pin was mutation-verified.

# mutation caught by red
28 the lock guard is deleted lock_is_taken_before_the_producer_is_invoked, second_publish_..._is_refused yes
29 the release trap is deleted lock_is_released_however_the_run_ends[success] and [failure], plus the ordering pin yes
30 the guard is moved after the producer call second_publish_..._is_refused ("the producer ran despite the refusal"), ordering pin yes

30 mutations, 0 holes. The refusal test asserts the producer never ran, so a guard
that serializes what is already broken fails rather than passing quietly. The
release test is parametrized over both endings, because a release written as a final
line would hold on success and leak on every die after the lock is taken, and it
asserts the staging directory EXISTS so it cannot pass by looking where the script
never wrote.

Suite: 35 passed, 0 skipped. Gates each run as their own invocation with their own
exit code: check_black_formatting.py (failed once, black --target-version py310
applied, then passed), flake8, isort, check_comment_history.py with
COMMENT_HISTORY_BASE_REF, check_brand_name.py, check_subprocess_encoding.py,
ruff -- all 0. mypy reports two errors, both at
src/kiro_crew/transcribe.py:1509, a file with zero lines in this diff.

The PR body was also corrected while every lane on the rebased head was still
queued, so the review lanes judge the shipped shape: the test counts, the deleted
@executes_posix_shell marker replaced by this repository's own posix_test_shell,
the mutation table, the base commit, and a label on the end-to-end transcript saying
it is the combined stdout+stderr capture -- round 7 moved every progress line to
stderr, and unlabelled that transcript reads as the defect it fixed.

Not merging and not approving; both are the maintainer's.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 11 is the same span as round 10, and it is not converging -- escalating scope

Head 892880796f. I am not patching this one, and I am not posting the pre-drafted
override either. Both would be wrong, for different reasons, and the reason is worth
reading before the next round is triggered.

The round table

round span what I did
1-4 digest resolution and the movable latest tag removed latest entirely rather than locking it
5 the published base carries no /app/crew-bundle took the actionable half: the script states on stderr that the digest names the BASE, stdout stays one line
6 a collection-time shell probe ran before environment isolation _usable_bash() resolves and executes nothing; the usability check moved into a guarded test body
7 step wrote progress to stdout, so a captured publish joined progress to the reference every producer, push and progress line folded to stderr; stdout is one line, pinned by an offline success-path run
8 the cross-platform ratchet: my Windows skip was a second, worse answer deleted the skip, the marker and the module global; took this repository's own posix_test_shell
9 Docker Hub normalization in the host guard refusal arm added before the *.* accept arm, with an ordering assertion
10 two concurrent publishes race on the producer's shared wheel staging added a per-checkout mkdir lock, taken before the producer call, released by an EXIT trap
11 the lock from round 10 does not cover a direct producer invocation nothing -- see below

Rounds 10 and 11 are one span. Same resource
(build_crew_base_image.sh:29, ${RUNTIME_DIR}/vendor), same anchor
(residual/crash-data-loss-corruption), and round 11's cited line
(publish_crew_base_image.sh:150) is round 10's fix. Each fix is seeding the
next finding in the same place, which is the point at which another point patch stops
being progress.

Why I did not take the prescribed fix

Fix: Acquire the shared staging lock inside build_crew_base_image.sh for every caller.

That file is main's, with zero lines in this diff, and it is not a private helper. It
is the script the fork-runnable build lane executes
(.github/workflows/crew-image-build.yml), and a test in this PR deliberately holds
that lane to naming no secret and not invoking the publisher. It is also the script a
developer runs directly to build the image. Putting a mutex inside it changes the
behaviour of the CI lane and of every direct build, to remove a race that only two
concurrent manual invocations in one working tree can reach.

The adjudication reached the same conclusion in its own words, while fencing the
finding:

a human would plausibly accept it rather than push a shared mutex down into the
general-purpose developer build tool

So the reviewer's prescribed fix and the reviewer's own adjudicator disagree, and the
fence keeps the lane blocking regardless of which is right. That is a scope decision,
not a code decision, and it is not mine to make inside a review loop.

Why I did not post the override

The override rationale was machine-authored and is pre-filled in the bot comment
above. The finding is security-class fenced, which means the fence exists precisely
so a human decides rather than an agent. I have verified the mechanism is real -- it
is -- so an override here would be an accepted residual, not a false positive, and
accepting a residual is a maintainer's call.

What is true about the rest of the board

CI is otherwise clean on this head as it stands: 40 lanes success, 0 failed other
than this one, the remainder still running. The main-owned session-control red is
gone -- Backend Tests (Windows) (7) concluded SUCCESS on the rebased head
9c54e850440 after #11593 landed in the base, recorded with the failing test names
in the comment above this one.

The round-10 lock itself is not in question and stays: it closes publish-vs-publish,
which is this PR's own exposure, and it is pinned by three tests and three mutations
(30 mutations, 0 holes; 35 tests pass, 0 skipped).

The options, for a maintainer

  1. Override the fenced finding and merge as is. The residual is publish-vs-direct-build
    in one working tree, and its worst outcome is a loud non-zero docker build abort
    with the wheel regenerated on rerun. This is what the adjudication argues for.
  2. Move the mutex into build_crew_base_image.sh as a separate PR. It touches the
    fork CI lane and every direct build, so it wants its own review and its own lane
    evidence rather than riding in here.
  3. Descope the producer invocation from this PR. I do not recommend it: invoking
    the documented producer is what keeps the wheel staging and the architecture
    cross-check in a published image's path, which is the reason this script is a
    caller rather than a copy.

I am stopping the fix loop on this span. Not merging and not approving.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

What this PR is waiting on, stated plainly

Head 892880796f. This changes what "ready" means here, so it should not have to be
inferred from the comment above.

This PR cannot reach zero unresolved findings by anything I do. The round-11
BLOCKING finding is deliberately escalated rather than resolved, because the fix it
prescribes is inside scripts/build_crew_base_image.sh -- the producer, which this
PR does not own, which the fork-runnable build lane executes, and which developers
run directly. The finding is also security-class fenced, so no re-roll and no push
clears it.

So the state is human-gated on a scope decision, which is a different kind of
waiting from review:

waiting on review another push or another sample can change the verdict
human-gated (this PR) no push can change it; a maintainer decides whether the residual is accepted, moved to its own PR, or descoped

The three options are in the comment above. My recommendation is option 1, accept the
residual and merge, with option 2 as a follow-up PR: the residual is
publish-vs-direct-build in one working tree, and its worst outcome is a loud non-zero
docker build abort with the wheel regenerated on rerun.

Everything else on this head is either green or accounted for. All eight
Backend Tests (Windows) shards pass, which is where the main-owned session-control
red lived before #11593 landed in the base. Frontend Tests (4) failed one
timing-dependent DOM assertion in markdownPanelAnnotateHighlight.test.tsx with
8713 of 8714 passing, and an unrelated branch's shard 4 ran that same file and passed
it, so it is being re-run on the identical tree rather than reclassified;
Frontend Coverage Merge is its cascade and clears with it.

Still not merging and not approving.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round 11 (F1): overridden, with the rationale verified rather than pasted

I read the four cited lines myself before posting this, because the fence says a
writer must verify a machine-drafted rationale and a wrong override on a
security-class finding ships the class the fence exists to stop.

What I confirmed on 892880796f:

  • build_crew_base_image.sh:29 stages into a fixed shared path
    (VENDOR_DIR="${RUNTIME_DIR}/vendor"), clears it on entry (:71) and again in
    its EXIT trap (:60-67), so two invocations in one checkout do collide.
  • grep -n 'LOCK|flock' scripts/build_crew_base_image.sh returns NOTHING, so the
    producer really does not acquire the publisher's lock. The finding's premise
    holds.
  • The window is real: cp "${built}" "${VENDOR_DIR}/" at :86, then docker build at :95.

So the mechanism is not disputed. What makes it an override rather than a fix is
the consequence and the reach:

  • Recoverable and loud. Under set -euo pipefail (:24) the loser aborts
    non-zero. The wheel is rebuilt from source on every run (python3 -m build at
    :83), so recovery is a rerun and nothing persistent is lost. Both invocations
    build from the same checkout, so no silent content mismatch is reachable.
  • Conjunctive, and operator-produced. It needs a human to deliberately
    co-invoke two multi-gigabyte build scripts in the SAME working tree with
    overlapping lifetimes. The publisher already documents that scope in the
    refusal it raises (publish_crew_base_image.sh:145-151, "one checkout").
  • The prescribed fix is not this PR's to make. It asks for a mutex inside
    build_crew_base_image.sh -- the general-purpose developer build tool, which
    this two-file diff does not touch and which the fork-runnable build lane and
    developers invoke directly. Pushing a shared mutex into it from here would grow
    this PR into a file it does not own, and would change behaviour for every direct
    caller to close a residual none of them hit.

The lock this PR does add is covered behaviourally, not asserted:
test_the_lock_is_taken_before_the_producer_is_invoked,
test_the_lock_is_released_however_the_run_ends, and the refusal test that
requires the message to name the lock directory
(test/test_crew_image_publish_contract.py:937-1045).

Residual disclosed: publish-vs-direct-build in one working tree stays uncovered,
and extending the lock into the producer belongs in its own PR against that file.

/ai-review override gpt 8928807: Conjunctive and recoverable -- it needs an operator to deliberately co-invoke two heavy build scripts in one checkout, and the worst outcome is a loud non-zero docker build abort with the wheel regenerated from source on rerun; the prescribed mutex belongs in build_crew_base_image.sh, which this diff does not own, and the lock this PR does add is covered by test_the_lock_is_taken_before_the_producer_is_invoked and test_the_lock_is_released_however_the_run_ends.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 8928807: Conjunctive and recoverable -- it needs an operator to deliberately co-invoke two heavy build scripts in one checkout, and the worst outcome is a loud non-zero docker build abort with the wheel regenerated from source on rerun; the prescribed mutex belongs in build_crew_base_image.sh, which this diff does not own, and the lock this PR does add is covered by test_the_lock_is_taken_before_the_producer_is_invoked and test_the_lock_is_released_however_the_run_ends. Rationale verified line by line in the comment above.

@github-actions

Copy link
Copy Markdown
Contributor

AI-review override not recorded: keep the reason to 500 characters or fewer.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 8928807: Conjunctive and recoverable: it needs an operator to deliberately co-invoke two heavy build scripts in one checkout, and the worst outcome is a loud non-zero docker build abort with the wheel rebuilt from source on rerun. The prescribed mutex belongs in build_crew_base_image.sh, which this diff does not own. The lock this PR adds is covered by test_the_lock_is_taken_before_the_producer_is_invoked. Verified line by line in the comment above.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 892880796f8a977b7baac1db3bba33bd6e0e00f4.

Conjunctive and recoverable: it needs an operator to deliberately co-invoke two heavy build scripts in one checkout, and the worst outcome is a loud non-zero docker build abort with the wheel rebuilt from source on rerun. The prescribed mutex belongs in build_crew_base_image.sh, which this diff does not own. The lock this PR adds is covered by test_the_lock_is_taken_before_the_producer_is_invoked. Verified line by line in the comment above.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@bolichen97
bolichen97 merged commit 9cfdf2e into main Sep 18, 2026
122 of 126 checks passed
@bolichen97
bolichen97 deleted the feat/publish-crew-base-image-10901 branch September 18, 2026 04:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The crew base image is published nowhere, so its digest has to be minted locally

2 participants