Skip to content

fix(browser): use the headless shell where libcups cannot load chrome - #32210

Merged
bolichen97 merged 1 commit into
mainfrom
fix/harness-old-libcups-headless-shell
Nov 7, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/harness-old-libcups-headless-shell

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Browser Harness starts and browses on Linux hosts whose libcups is too old for the full Chromium build, such as Amazon Linux 2.

On Amazon Linux 2 every Browser Harness start waits about 60 s and fails with the Harness daemon did not start in time, whatever browser.choice says. Kiro's Chromium (the full Chrome for Testing 154.0.8037.0 build) exits 127 at load with undefined symbol: ippValidateAttributes: it imports four libcups functions that cups-libs 1.6.3 does not define.

Why it matters

On such a host the agent has no browser: the browser tool, Settings' Test and the live view all fail, and the error names neither the browser nor the library. Settings offers no way out, because on a host without a display every choice launches the same binary. The earlier stacks carried a fallback for this (#2897, then #23585); #26950 retired them, and the fallback with them.

Not a goal

  • The 60 s wait when Kiro's Chromium dies at launch for any reason: A missing system library for the browser gives a 60 s generic timeout with no library name or install hint #30312.
  • A host with no libcups at all, or one missing another library.
  • A windowed Kiro's Chromium on such a host: the headless shell has none, so it is reported and run headless.
  • Deleting a full build an older release already downloaded there; it stays on disk, unused.
  • Installing the Harness venv on a host that cannot build pillow==12.3.0 from source (it has no glibc 2.26 wheel); seen while testing on Amazon Linux 2, separate from this crash.

What changed (motivation → approach → change)

A Linux host whose libcups lacks a function the full build imports now gets the same version's chrome-headless-shell, which links no libcups. Every other host, macOS and Windows included, keeps the full build.

Piece Change
browser_harness/libcups.py (new) Says whether the full build can load. It reads the first native 64-bit libcups.so.2 in the linker's system directories, never LD_LIBRARY_PATH, and parses its dynamic symbol table in Python: nothing is launched and no binutils is needed. No readable library keeps the full build.
browser_harness/chromium.py wanted_build() picks the build, and state, download and the executable the supervisor launches all follow it. The shell lives in chromium/154.0.8037.0-headless-shell/. A full build already on disk reads as not downloaded there, so the startup auto-install fetches the shell unasked. The status payload gains chromium.build.
browser_harness/supervisor.py _choose reports a kiro_chromium choice as headless when the build is the shell, so status, the live card and the startup warm never offer a window it cannot show.
docs/system-specs/modules/browser.md Records the fallback, when it applies, and how a version bump re-reads the symbol list.

Backwards compatibility

Compatible: on a host with a current libcups the download URL, directory, executable and flags are what they were, and chromium.build is an added key. On an old-libcups host with a display, a kiro_chromium pin now reports headless: true, which is what runs. On an old-libcups host the full build is no longer launched; it never started there.
Reader: src/kiro_crew/dashboard/handlers/browser_harness_status.py:_auto_components -- release -- reads chromium.present; false for an unusable full build, so it adds the chromium component and the install fetches the shell (test_a_full_build_already_on_disk_is_no_browser_for_an_old_libcups_host)
Reader: src/kiro_crew/dashboard/handlers/browser_harness_status.py:_warm_wanted -- release -- waits for chromium.present, which the shell's install then makes true
Reader: src/kiro_crew/browser_harness/supervisor.py:_launch_chromium -- chat -- launches state.chromium.executable, now the shell on such a host (test_the_browser_an_old_libcups_host_launches_is_the_headless_shell)
Reader: website/src/pages/settings/BrowserEngineSection.tsx:BrowserEngineSection -- crew page -- shows Install instead of Redownload until the shell is on disk; Redownload replaces the shell, the build that host runs

Tests

  • test/test_browser_harness_libcups.py (new) pins the probe against ELF files built in the test: Amazon Linux 2's shape is too old; one missing symbol is enough; a symbol the library only imports counts as missing; no file, an empty, non-ELF or truncated file is no evidence; a 32-bit or foreign-architecture copy is passed over for the next directory; the first native copy decides; only Linux is ever too old; LD_LIBRARY_PATH never decides.
  • test/test_browser_harness_chromium.py pins the build per platform, the shell's URL on both sources, its directory and executable, a full-build archive refused where the shell is wanted, the upgrade path from an existing full build, and that install_state(home).chromium.executable is the shell. An autouse fixture pins the probe so the file does not depend on the host's libcups.
  • test/test_browser_harness_supervisor.py pins that _choose turns a windowed kiro_chromium choice headless when the build is the shell, and leaves the full build and the person's Chrome windowed.
  • Nine mutants of the new guards (an undefined symbol counted as defined, the machine check, the Linux check, LD_LIBRARY_PATH consulted, the probe ignored, state or download ignoring the wanted build, either half of the _choose condition) each turn these tests red.
  • Determinism: the libcups and chromium files ran 20/20 at -n0, 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_browser_harness_install.py and test_browser_harness_discover.py, and their 40 ids in a shuffled order (seed 28709) green. With the supervisor test added, 20/20 at -n0; its -n 4 and shuffled runs are left to CI.

Proof: not provable -- other: the fix adds libcups.py, which both test files import, so prove.py's revert of the production hunks fails at collection (INCONCLUSIVE, and per hunk the dependent hunks fail to import); the nine mutants above are the substitute

Manual verification

In an amazonlinux:2 container (cups-libs 1.6.3, glibc 2.26), each tree installed and run through install_sync(home) then HarnessSupervisor(home).test(150), goto and page_info:

Tree Build downloaded test()
origin/main chrome-linux64/chrome after 60.6 s: error, the Harness daemon did not start in time
this PR chrome-headless-shell-linux64/chrome-headless-shell after 0.9 s: connected; goto example.com, page_info title Example Domain
this PR on main's home (full build already there) reads it as not downloaded, fetches the shell after 0.6 s: connected, same page

The container has no display, so the _choose change is covered by its unit test, not by this run. On an Amazon Linux 2023 host the probe keeps the full build, which connects as before. The headless shell over --remote-debugging-pipe with the supervisor's flags, --headless=new included, also passed navigation, Runtime.evaluate, a screenshot and a screencast frame on Amazon Linux 2.

Related Issues

Closes #32197

Sibling sweep

Swept: git grep -n -E "chromium.launch|chromium.executable_path|launch_persistent_context|chrome-linux64|ms-playwright|playwright install" -- src/
Sibling: src/kiro_crew/builtin_skills/browser-recording/scripts/driver.mjs -- not affected -- chromium.launch() with Playwright's headless default runs its own headless shell, not the full build
Sibling: src/kiro_crew/apps/builtins/dev_fleet/skills/feature-demo-recording/references/record_template.py -- not affected -- headless=True with no channel, Playwright's headless shell
Sibling: src/kiro_crew/apps/builtins/dev_fleet/skills/pod-e2e/scripts/pod-playwright.py -- not affected -- headless: True with no channel, Playwright's headless shell
Sibling: src/kiro_crew/apps/builtins/design_critique/skills/design-critique/scripts/render.mjs -- not affected -- launches the user's installed Chrome (channel: 'chrome'), then Playwright's default

Pattern harvest

Rule candidate: review-prompt
Pattern: a refactor that retires a module deletes a platform fallback the module carried, with no test outside the module pinning that the fallback still exists

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 7, 2026 08:20
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 6dda57d816b0719c69e0e07c7c47080df8120afc touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 6dda57d816b0719c69e0e07c7c47080df8120afc — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

One rider: the new chromium.build key in the status payload has no reader. It is small and easy to remove later, so it does not block.

Notes

  • Item 4 — zero consumers: "build": self.build, in ChromiumState.as_dict. A grep for chromium.build / "build" across website/src and src/kiro_crew/{browser_harness,dashboard} finds nothing that reads the dict key. supervisor._choose reads the dataclass field, not the payload. Subtraction: delete the "build" entry from as_dict and the {…build…} addition to the status line in browser.md. Keep the dataclass field.

What this change ships

Inventory (5 items) — 4 justified

Intent: let Browser Harness start and browse on Linux hosts whose libcups is too old for the full Chromium build (Amazon Linux 2). This is a FIX.

  1. On an old-libcups Linux host, Kiro's Chromium is now downloaded and launched as the same version's headless shell, not the full build — justified
  2. On such a host, a full build already on disk counts as not downloaded, so the startup install fetches the shell — justified
  3. On such a host, a kiro_chromium choice is reported as headless even when a display is present — justified
  4. The status payload gains a chromium.build field — zero consumers: no reader of the payload key
  5. A new filesystem-only probe reads the system libcups.so.2 symbol table to decide which build to use — justified

[FIRST-PRINCIPLES-REVIEWED] 6dda57d

Description read

[DESCRIPTION-READ] d71f94a406acc47db403c38e685384f213c6038be42c05c59d99e12e0eadb716

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 6dda57d816b0719c69e0e07c7c47080df8120afc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The fallback is decided by a probe that only reads files, and the state, download and launch paths all take their answer from wanted_build. Nothing is removed, so no reader breaks.

Notes

  • Directories other than /lib64 and /usr/lib64 are hard-coded in LIBRARY_DIRS. On a host that keeps libcups somewhere else (for example only in ld.so.conf.d paths), the probe finds no library and keeps the full build. That is the same behaviour as before this PR, so nothing gets worse; just know the fallback is not reached there.

[DESIGN-REVIEWED] 6dda57d

Description read

[DESCRIPTION-READ] 5bce83b2087ccd070ec728ca3a750080e568753aa3020e696bc428adc1f49564

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 6dda57d816b0719c69e0e07c7c47080df8120afc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6dda57d

Verdict parsed from the review's SHA-scoped output markers for commit 6dda57d816b0719c69e0e07c7c47080df8120afc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6dda57d816b0719c69e0e07c7c47080df8120afc false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 6dda57d816b0719c69e0e07c7c47080df8120afc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6dda57d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6dda57d816b0719c69e0e07c7c47080df8120afc false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 7, 2026 08:44
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Two reds on head 1db320b793, neither a defect in this diff:

Backend Tests shard 2 (Linux and Windows) / Coverage Gate: test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget fails with assert 34855 <= 34845. It counts kiro_crew.security, which this PR does not touch; the same count is red on origin/main at 0795ebc479. #32183 raises the budget to main's size. This PR waits for it and rebases once it merges.

PR Policy / Cross-Platform Portability: the four flagged lines are /usr/lib64, /usr/lib/x86_64-linux-gnu, /usr/lib/aarch64-linux-gnu and /usr/lib in browser_harness/libcups.LIBRARY_DIRS: the Linux dynamic linker's system directories. They are read only on Linux (libcups.too_old_for_chrome returns False on any other sys.platform before touching them; test_only_linux_is_ever_too_old), and a Windows or macOS host always gets the full build. Applying posix-only-approved for that reason.

@bolichen97 bolichen97 added the posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only label Nov 7, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

More evidence, on a real Amazon Linux 2 userland as a normal user, so with Chrome's sandbox on (the container run in the body was root, which adds --no-sandbox):

Host: a DevSpace from the devspaces-al2 image: Amazon Linux 2, cups-libs 1.6.3, glibc 2.26, non-root, user.max_user_namespaces=126043, no seccomp. This PR's tree together with #32251 (the Harness install preflight; its pillow build was given the libjpeg headers through CPATH):

  • install_sync(home): chromium.build = chrome-headless-shell, downloaded into chromium/154.0.8037.0-headless-shell/.
  • HarnessSupervisor(home).test(150): connected after 0.7 s, browser={'kind': 'kiro_chromium', 'headless': True}; goto example.com, page_info title Example Domain.
  • Separately, chrome-headless-shell 154 launched with the supervisor's argv minus --no-sandbox over --remote-debugging-pipe: navigation, Runtime.evaluate, a screenshot and a screencast frame all pass.

Not covered: that DevSpace runs Amazon Linux 2 userland on the Amazon Linux 2023 kernel, which allows unprivileged user namespaces. A host kernel with user.max_user_namespaces=0 would make Chrome's sandbox fail (No usable sandbox!); that is a host setting this PR does not change.

On a Linux host whose libcups lacks the symbols the full Chrome for
Testing build imports (Amazon Linux 2: cups-libs 1.6.3), Kiro's
Chromium exits 127 at load and Browser Harness reports a 60 s daemon
timeout instead. Pick the same version's chrome-headless-shell there,
which links no libcups. Download, state and launch all follow
chromium.wanted_build, so a full build already on disk reads as not
downloaded and the startup auto-install fetches the shell. The probe
reads libcups' ELF dynamic symbol table from the linker's system
directories, so the status path still launches nothing.

Closes #32197
@bolichen97
bolichen97 force-pushed the fix/harness-old-libcups-headless-shell branch from 1db320b to 6dda57d Compare November 7, 2026 15:08
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_security_gate_liveness.py line budget (#32252) and the Windows platform ignore list (#32209) since this head went red (0795ebc -> d27a5fd); CI re-running. Clean rebase, no code change. PR Policy was already green after posix-only-approved was applied for the Linux-only LIBRARY_DIRS.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: on a Linux host whose libcups lacks the symbols the full Chrome build needs, downloads and launches the headless shell build; with tests. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/browser.md.

@bolichen97
bolichen97 merged commit 23b4795 into main Nov 7, 2026
97 of 106 checks passed
@bolichen97
bolichen97 deleted the fix/harness-old-libcups-headless-shell branch November 7, 2026 16:09
buluoray pushed a commit that referenced this pull request Nov 8, 2026
…#32210)

On a Linux host whose libcups lacks the symbols the full Chrome for
Testing build imports (Amazon Linux 2: cups-libs 1.6.3), Kiro's
Chromium exits 127 at load and Browser Harness reports a 60 s daemon
timeout instead. Pick the same version's chrome-headless-shell there,
which links no libcups. Download, state and launch all follow
chromium.wanted_build, so a full build already on disk reads as not
downloaded and the startup auto-install fetches the shell. The probe
reads libcups' ELF dynamic symbol table from the linker's system
directories, so the status path still launches nothing.

Closes #32197

(cherry picked from commit 23b4795)
buluoray pushed a commit that referenced this pull request Nov 8, 2026
…#32210)

On a Linux host whose libcups lacks the symbols the full Chrome for
Testing build imports (Amazon Linux 2: cups-libs 1.6.3), Kiro's
Chromium exits 127 at load and Browser Harness reports a 60 s daemon
timeout instead. Pick the same version's chrome-headless-shell there,
which links no libcups. Download, state and launch all follow
chromium.wanted_build, so a full build already on disk reads as not
downloaded and the startup auto-install fetches the shell. The probe
reads libcups' ELF dynamic symbol table from the linker's system
directories, so the status path still launches nothing.

Closes #32197

(cherry picked from commit 23b4795)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser Harness never connects on Amazon Linux 2: Kiro's Chromium exits at load (libcups 1.6.3)

2 participants