Skip to content

fix(browser): refuse a harness install that cannot build pillow - #32251

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/harness-install-old-glibc-build-deps
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/harness-install-old-glibc-build-deps

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On a Linux host too old for the locked pillow's wheels, such as Amazon Linux 2, installing Browser Harness says what is missing and how to install it, instead of failing with no cause.

The Harness install fails at installing_harness on Amazon Linux 2 hosts that lack a compiler or the image headers, and the error ends at Failed to build pillow. browser-harness==0.1.13 pins pillow==12.3.0 exactly, and 12.3.0 publishes Linux wheels for glibc 2.27+ only, so on glibc 2.26 pip compiles the sdist. The install error keeps the tail of pip's output, and the compiler's real message is cut off before it.

Why it matters

On such a host the browser stays "not set up on this host" and the only clue is a pip log that names no cause, so the person cannot tell that one yum install fixes it. Re-pinning pillow is not an option while browser-harness pins it exactly.

Not a goal

What changed (motivation → approach → change)

Before pip runs on a glibc older than 2.27, the install checks for what pillow's build needs and, if anything is missing, refuses at once with the list and the command that installs it:

Browser Harness needs pillow 12.3.0, which has no wheel for this host's glibc 2.26 (its wheels need glibc 2.27+), so pip has to build it here, and this host lacks jpeglib.h. Install them, then install Browser Harness again: sudo yum install -y libjpeg-turbo-devel
Piece Change
browser_harness/build_deps.py (new) Linux only, read from the filesystem: a C compiler ($CC, the interpreter's CC, or cc/gcc), the gateway interpreter's Python.h, and zlib.h / jpeglib.h wherever pillow 12.3.0's setup.py looks for them. With a custom pkg-config search path the headers are left to pip, since the check does not run pkg-config. Names the yum or apt-get packages for what is missing.
browser_harness/install.py Runs the check at preparing, before the marker is removed or the venv cleared, so a refused reinstall leaves a working install in place. A venv that already holds the locked pillow and is not being rebuilt is not checked: pip has nothing to build.
docs/system-specs/modules/browser.md Records the check in the driver section.

Backwards compatibility

Compatible: on a host the locked wheels cover nothing changes. On a glibc older than 2.27 the only new outcome is a refusal where pip would have failed anyway, since its build needs exactly what the check looks for.
Removes nothing: a host with the build tools installs as before; the check only names a failure that already happened.

Tests

  • test/test_browser_harness_build_deps.py (new) pins the check: Amazon Linux 2 with nothing installed lists the compiler, Python.h, zlib.h and jpeglib.h; gcc and zlib without libjpeg lacks only jpeglib.h; a full toolchain passes; glibc 2.27, 2.28, 2.34 and 3.0, macOS, Windows and musl are never checked, and off Linux the host is not even probed; headers count wherever pillow's setup.py reads them (C_INCLUDE_PATH, CPATH, INCLUDE, a glued -I in CFLAGS/CPPFLAGS/LDFLAGS, ZLIB_ROOT/JPEG_ROOT) and a spaced -I, which pillow ignores, does not; a custom pkg-config path leaves the headers to pip; $CC and the interpreter's CC count as the compiler; the refusal names the cause and only the packages for what is missing, uses apt-get where there is no yum, offers no package for Python.h; and the floor follows the pillow the lock pins, so a lock bump re-reads its wheel tags.
  • test/test_browser_harness_install.py adds three install-level tests: a refusal happens before any spawn, a refused rebuild leaves the existing install readable as installed, and a venv that already holds the locked pillow is not checked. An autouse fixture pins the check off so the file does not depend on the host's glibc.
  • Thirteen mutants of the guards (the glibc floor, the Linux checks, the compiler and $CC checks, the header check and each of its search sources, the pkg-config escape, ignoring an installed pillow, not raising) each turn these tests red. The two files ran 20/20 at -n0; the -n 4 and shuffled runs are left to CI.

Proof: not provable -- other: the fix adds build_deps.py, which both test files import, so reverting the production hunks fails at collection; the thirteen mutants above are the substitute

Manual verification

On a DevSpace from the devspaces-al2 image (Amazon Linux 2, glibc 2.26, normal user) that has gcc and zlib.h but not jpeglib.h, each tree installed and run through install_sync(home):

Tree Result
main fails after 5.3 s at installing_harness; the error ends at Failed to build pillow
this PR refused after 0.0 s at preparing with the message above
this PR plus #32210, libjpeg headers given through CPATH the check passes, pip builds pillow, the install finishes in 10 s; test() connects in 0.7 s and opens example.com

Related Issues

Closes #32241

Sibling sweep

Swept: git grep -n -E "require-hashes|requirements.lock" -- 'src/.py'; git ls-files 'src/**/requirements.lock' 'src/**/requirements*.txt'
Sibling: src/kiro_crew/apps/builtins/aws_control/crew/runtime/container/requirements.txt -- not affected -- installed while building that app's own container image, never into a host venv on the operator's glibc

Pattern harvest

Rule candidate: review-prompt
Pattern: an exact pin inherited from a dependency silently raises the minimum glibc, and a failed source build is reported by its log tail, which names no cause

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 7, 2026 09:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 53867a59b3158725e46ebe01630f3a93b48a9117 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 53867a59b3158725e46ebe01630f3a93b48a9117 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The check runs only on Linux with glibc older than 2.27, a refusal can be undone by a revert, and a full toolchain still installs as before.

Notes

  • A false refusal has no override. If the check misses a header or compiler that pip would find (an unusual multiarch include dir, a wheel already built in pip's cache), refusal_for_this_host() raises InstallError(STAGE_PREPARING, …) and the person cannot reach pip at all. The PR names no environment variable to skip the check. A one-line way to skip it would turn a heuristic miss into an inconvenience instead of a dead end.

[DESIGN-REVIEWED] 53867a5

Description read

[DESCRIPTION-READ] 2ec28fd4700b666d305104351d8f967a4ab9e5f554f6a523700cdae42d4af0cb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 53867a59b3158725e46ebe01630f3a93b48a9117 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The change targets a reported defect (#32241), shows evidence from a real AL2 host, and only refuses where pip would already fail. Nothing blocks.

Not checked: that pillow 12.3.0's setup.py searches exactly the include sources build_deps.py mirrors.

Notes

  • Item 4 — symptom-level: the description itself names the mechanism: "The install error keeps the tail of pip's output, and the compiler's real message is cut off before it" (InstallError.detail = detail[-ERROR_DETAIL_CAP:], install.py:120). The 171-line preflight is a second copy of pillow's own header search (rung c), and it must follow every pillow bump. Subtraction: none ships with this note; making the cap keep pip's own dependency line instead would be a smaller fix at the mechanism level.

What this change ships

Inventory (6 items) — 5 justified

Intent: on an old-glibc Linux host, a failed Browser Harness install names the missing build tools and the command that installs them. FIX.

  1. On glibc < 2.27 Linux without a compiler, Python.h, zlib.h or jpeglib.h, install refuses at preparing with a list of what is missing — justified
  2. The refusal message includes a yum/apt-get command for only the missing packages — justified
  3. A refused reinstall leaves the working install in place — justified
  4. The header search copies pillow's env sources (CPATH, glued -I, *_ROOT) — symptom-level, second copy of pillow setup.py's search
  5. With PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR set, the header check is skipped — justified
  6. A venv that already holds the locked pillow skips the check — justified

[FIRST-PRINCIPLES-REVIEWED] 53867a5

Description read

[DESCRIPTION-READ] 02a94ff7aa096a57f88d23164a789f1be53dd6626447a9a26fe30a77673c2376

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 53867a59b3158725e46ebe01630f3a93b48a9117 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 53867a5

Verdict parsed from the review's SHA-scoped output markers for commit 53867a59b3158725e46ebe01630f3a93b48a9117.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 53867a59b3158725e46ebe01630f3a93b48a9117 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on 53867a59b3158725e46ebe01630f3a93b48a9117; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/browser_harness/install.py:331 -- Preflight rejects reinstalls that can reuse a cached Pillow wheel

raise InstallError(STAGE_PREPARING, refused)
Trigger: On an old-glibc host, a successful install cached a compatible Pillow wheel; development headers are subsequently removed and the Harness venv deleted. Startup auto-install → _start_install_job → install_sync → _install_harness reaches this refusal before pip runs.
Outcome: Browser Harness cannot be reinstalled even though pip can reuse the cached wheel without compiling: its hash-checking path validates the cached wheel’s recorded source hash against the lock.
Anchor: defect
Fix: Allow pip’s compatible, source-hash-validated cache path, or attach the missing-dependency diagnosis after an actual source-build failure.

[BLOCK-MERGE-DOWNGRADED] 53867a5
[GPT-REVIEWED] 53867a5

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — LOW. The required conditions: a Linux host with glibc older than 2.27 (build_deps.py:94) that once built pillow successfully, then had its build tools or headers removed (build_deps.py:100-107), then lost its Harness venv so a rebuild runs (install.py:321-328). When that happens the install refuses at install.py:329-331. The refusal is visible and names the exact fix: the yum/apt-get command (build_deps.py:115-121). Running that command once clears it for good. Nothing is lost, and no corrupt or partial state is left behind: the refusal happens before the marker or venv is touched.
Real fix cost: the installer would have to model pip's private wheel cache (where it lives, wheel tags, recorded source hashes, which pip version is in use) as a mechanism serving only this rare path. The alternative is redesigning the refusal into a diagnosis after pip fails. Either one is a permanent burden for future readers that outweighs a loud, self-explaining refusal on a rare combination of conditions.

[ADJUDICATION] 53867a59b3158725e46ebe01630f3a93b48a9117 total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/browser_harness/install.py:331 reason=disproportionate-remedy
[GPT-ADJUDICATED] 53867a59b3158725e46ebe01630f3a93b48a9117

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 53867a59b3158725e46ebe01630f3a93b48a9117 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 10:14
On a Linux host with a glibc older than 2.27 (Amazon Linux 2: 2.26),
the locked pillow==12.3.0, which browser-harness pins exactly, has no
wheel, so pip builds its sdist. Without a C compiler, Python.h, zlib.h
or jpeglib.h that build fails, and the install error keeps only pip's
tail, "Failed to build pillow", which names no cause. Check for them
first, from the filesystem, and refuse at the preparing stage, before
the marker or the venv is touched, naming what is missing and the yum
or apt-get command that installs it. A venv that already holds the
locked pillow is not checked.

Closes #32241
@bolichen97
bolichen97 force-pushed the fix/harness-install-old-glibc-build-deps branch from d557fc7 to 53867a5 Compare November 7, 2026 15:00
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_security_gate_liveness.py budget and the platform ignore list since this head went red (e30915e -> d27a5fd); CI re-running.

Also in this head (53867a5):

  • Windows shard 7 and New-test stability: test_the_compiler_the_build_would_use_counts passed a Windows tmp_path in $CC. $CC is split the POSIX shell way (shlex.split), which drops backslashes, so C:\Users\...\gcc10-gcc became C:Users...gcc10-gcc and was not found. The check itself runs on Linux only; the test now passes the path as as_posix(). Not seen green on Windows yet; CI decides.
  • First Principles blocker: build_deps.py no longer has its own glibc reader. refusal_for_this_host calls constants._host_glibc_version(), the reader acp/kas_bundle.py already uses.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Second glibc reader

fixed in 53867a5: parse_glibc and host_glibc are deleted; refusal_for_this_host reads the version from constants._host_glibc_version(), the never-raising reader acp/kas_bundle.py already uses for its Amazon Linux 2 floor.
The tests now patch constants._host_glibc_version; the parse test for the deleted helper is gone.

@bolichen97 bolichen97 added the posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only label Nov 7, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 53867a5, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit d3aff60 into main Nov 7, 2026
99 of 115 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/harness-install-old-glibc-build-deps branch November 7, 2026 19:44
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
buluoray added a commit that referenced this pull request Nov 9, 2026
On a Linux host with a glibc older than 2.27 (Amazon Linux 2: 2.26),
the locked pillow==12.3.0, which browser-harness pins exactly, has no
wheel, so pip builds its sdist. Without a C compiler, Python.h, zlib.h
or jpeglib.h that build fails, and the install error keeps only pip's
tail, "Failed to build pillow", which names no cause. Check for them
first, from the filesystem, and refuse at the preparing stage, before
the marker or the venv is touched, naming what is missing and the yum
or apt-get command that installs it. A venv that already holds the
locked pillow is not checked.

Closes #32241

(cherry picked from commit d3aff60)
buluoray added a commit that referenced this pull request Nov 9, 2026
On a Linux host with a glibc older than 2.27 (Amazon Linux 2: 2.26),
the locked pillow==12.3.0, which browser-harness pins exactly, has no
wheel, so pip builds its sdist. Without a C compiler, Python.h, zlib.h
or jpeglib.h that build fails, and the install error keeps only pip's
tail, "Failed to build pillow", which names no cause. Check for them
first, from the filesystem, and refuse at the preparing stage, before
the marker or the venv is touched, naming what is missing and the yum
or apt-get command that installs it. A venv that already holds the
locked pillow is not checked.

Closes #32241

(cherry picked from commit d3aff60)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser Harness install on Amazon Linux 2 fails with no cause: pillow 12.3.0 has no glibc 2.26 wheel

2 participants