Repository navigation
test: stop five e2e and integration tests failing on host timing - #17259
Conversation
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. [UX-REVIEWED] 3cd44ae Description read[DESCRIPTION-READ] 1de7dda47d5735eba0ac140ec48e49c18c5755ea27bdfdd849530d6154669106 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS One thing to check: the i18n render gate now always measures layout in the fallback font, so text that overflows only in the real Google webfont is no longer caught. [DESIGN-REVIEWED] 3cd44ae Description read[DESCRIPTION-READ] 78ac5d6b11b778ebfc02b38724f6615fa9f87390d8d32758776a7ce9738d3345 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The developer-home and credential isolation ( Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: stop five tests going red for host-timing or third-party reasons unrelated to the PR under test — FIX.
[FIRST-PRINCIPLES-REVIEWED] 3cd44ae Description read[DESCRIPTION-READ] a65428ffb82a35ef541feb524170ae39b71195595ca8aeb9fc0149d7f870b881 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
self-added: yes
|
|
self-added: yes
|
|
|
Merged-tree probe (no push): current Run from the merged tree's own |
af4b230 to
d680901
Compare
|
Automated rebase onto |
Five browser, end-to-end and in-process-gateway tests went red on runs whose diff had nothing to do with them. Each now waits on, or removes, the thing it raced: - i18n render gate: a context route answers the Google Fonts stylesheet with empty CSS and refuses every other off-origin request, so a slow third-party answer can no longer hold a capture past the quiet window, and both sweeps measure the same face. - integration boot smoke: the per-boot reset list also clears the path-resolver bookkeeping, the status-count cache, the inventory gauges and the thread-ledger baseline, and host clocks on their own backoff are listed with their reason; a new test proves a seeded cooldown and cached count do not cross a boot. - integration harness isolation: integration_home keeps the in-process gateway away from the developer's local credential stores and home-derived paths. HOME, USERPROFILE, APPDATA, LOCALAPPDATA and the XDG data, cache and state roots point at an empty tmp sibling, PATH leads with the test interpreter's directory, every declared home-override variable and every AWS and GitHub credential the environment carries is dropped, and the credit pill's credential reader is stubbed. A new test boots under a stand-in home and fails on any read beneath it, on a credential that reaches the boot, or if the real credential reader runs. - E2E boot matrix: agent.sandbox and agent.provider are read from GET /api/config/kirocrew instead of opening the live gateway's config.json, which Windows refuses while the gateway replaces it. - two-chat streaming: the fake ACP backend gains an opt-in [[SLOW_HOLD:<token>]] marker that holds a turn after one chunk until the test creates a release file, so "B streams while A is mid-turn" and "A streams while B is mid-turn" are constructed, not raced; unit tests pin its release, cancel and expiry paths. The fake also answers whoami with any flags, as the credit refresh calls it. - hooks.spec.ts: every test seeds its own uniquely named hook through the API, waits on page and response signals, asserts the outcome in the UI and the API, and deletes only the ids it created. The testing-conventions paragraph on ordering two chat turns now points at the hold marker instead of [[SLOW]]'s cadence.
d680901 to
3cd44ae
Compare
Problem / Motivation
Goal: Five browser, end-to-end and in-process-gateway tests stop failing for reasons that have nothing to do with the change under test: a slow third-party font server, a Windows file-sharing window, a loaded runner's clock, or another test's data.
Each of these was seen red in CI or reds under a forced condition on the parent commit, with nothing wrong in the diff it was judging:
/developer?tab=config never went quiet in 10000ms ... Still in flight: https://fonts.googleapis.com/css2?....test_a_second_boot_touches_only_known_module_globalsnamedsecurity.paths._path_resolve_thread_waits.PermissionError: [Errno 13]opening the live gateway'sconfig.json.test_streaming.py): its ordering held only while the second turn started within ~14.5 s of wall time.hooks.spec.ts: ten fixed sleeps, three tests that silently skip their own body, a test that deletes whatever hook sorts first, and anafterAllcleanup that never matches anything.Why it matters
A red that is not the PR's fault costs a rerun, hides real failures among the noise, and teaches people to rerun until green. The boot-smoke and i18n reds landed on unrelated PRs and on main; the hooks spec could delete a developer's own first hook when pointed at a live gateway.
Not a goal
testing-conventions.mdparagraph that described the replaced streaming pattern.website/scripts/lib/render-settle.mjs.What changed (motivation → approach → change)
index.htmlrequests the Google Fonts stylesheet on every navigation; routing turns the HTTP cache off, so each of ~124 captures waits on a live third-party answer, and one stall exceeds the 10 s quiet window. A truly offline host fails fast and passes.test/integration/conftest.py) resets home- or run-derived state per boot in_reset_home_bound_globals(resolver bookkeeping, status-count cache, inventory gauges, thread-ledger baseline). Host clocks are listed with their reason. A new test seeds a cooldown and a cached count and proves both reset points drop them.config.jsonwhile the gateway rewrites it by temp file plusos.replace; Windows refuses that open.agent.sandbox/agent.providerfromGET /api/config/kirocrew; the test never opens the live file.[[SLOW_HOLD:<token>]]marker insrc/kiro_crew/testing/fake_acp_backend.py: one chunk, then a hold until the test creates the token's release file. The test proves B streams while A is held and A streams while B is held.[[SLOW]]is unchanged.hooks.spec.tsisVisiblebranches, page-wide.first().docs/system-specs/common/testing-conventions.md's paragraph on ordering two chat turns now points at the hold marker instead of[[SLOW]]'s cadence.Harness isolation found on the way: the in-process integration gateway reads the developer's home wherever production does, so on a developer machine the onboarding import scan walked their foreign agents' homes and a credit refresh read kiro-cli's credential stores.
isolate_user_home(called byintegration_home) isolates the integration harness from the developer's local credential stores and home-derived paths:HOME,USERPROFILE,APPDATA/LOCALAPPDATAand the XDG data, cache and state roots point at an empty tmp sibling ofKIROCREW_HOME,PATHleads with the test interpreter's directory (a home and itsPATHmove together), every declared home-override variable is dropped, and the credit pill's credential reader is stubbed. Its store paths stay where import bound them, as the repository's security-anchor rule requires.isolate_credentialsdrops the channel credentials, every AWS credential pointeracp.client.CREDENTIAL_POINTER_ENV_VARSdeclares plus the profile selectors, inline keys andAWS_ROLE_ARN, and theGH_/GITHUB_namesgithub_runner.GH_ENV_PASSTHROUGHforwards togh; theawsCLI's two files point at paths that do not exist. The fake backend answerswhoamiwith any flags (the credit refresh passes--format json) instead of falling into its ACP loop. No product code changes.Backwards compatibility
Compatible: the fake backend's change is an added, opt-in marker; every existing marker and constant keeps its bytes. No production module changes.
Removes nothing: only test assertions, harness resets and one test-script route change; no product path, record or key is removed.
Tests
Every touched test went through the de-flake protocol on Linux, on a shared 32-CPU host, with
/proc/loadavgrecorded per run. "Parent" means the frozen baseaf1eb612d(the rebased base is3c1e383dd; the only owned-file changes upstream are #17118 removing two crew-log witness entries and #17054's conventions additions, both carried). Out-of-tree probes forced each condition; none is committed. Every run started underenv -iwith a scratchHOME,KIROCREW_HOMEandKIROCREW_WORKSPACE.check-i18n-render.mjs)never went quiet ... Still in flight: https://fonts.googleapis.com/css2?...(the CI line); DNS failure (truly offline) passesI18N_BASE_REF=6f70008e3→PASS [vs-base]test_a_second_boot_touches_only_known_module_globalsslow_thread5/5 red (status_counts._counts_cache_ts,metrics.provider._consent_checked_at); one forced 0.15 s resolution in boot 2: red with the CI's exact_path_resolve_thread_waitsslow_thread8/8 green; forced resolution green-n0; 10/10 at-n 4with governance and update-shutdown co-tenants; whole integration layer in CI shape (-n 2) 51/51 on 5 runs, then 52/52 on 5 runs with the new isolation test_path_resolve_thread_waits.clear()→ red by name; drop the cooldown or count reset → the new pin test redstest_streaming.pyB did not stream until A had finished-n0(slowest signal 0.12 s), 10/10 at-n 4B: a first chunk while A was held mid-turn ... did not arrive; hold never reached → red; hold dir unset → redtest_gateway_boot_matrix.pyconfig.jsonrefused the way Windows does: 3 failed withPermissionError: [Errno 13]at the CI frames-n0(module convention; real gateways)test_a_boot_reads_nothing_from_the_outer_home(new)integration_home left kiro-cli's real credential reader live; with the reader asserted last, the parent reads its~/.aws/sso/cachetoken files,CODEX_HOMEandOPENCLAW_CONFIG_PATH, the first commitLOCALAPPDATA/hermesandOPENCLAW_CONFIG_PATH-n0, 5/5 after the last edit;slow_thread1/1the refresh opened kiro-cli's own credential stores; dropAPPDATA/LOCALAPPDATA→ scan resolvesAppData/Local/hermes; drop the undeclared overrides →OPENCLAW_CONFIG_PATH/OPENCLAW_WORKSPACE_DIR; drop the import sources' variables →GEMINI_HOME,HERMES_HOME,OPENCLAW_STATE_DIR; drop the GitHub names →GH_TOKEN,GITHUB_TOKEN, ...; drop the AWS credential pointers →AWS_CONTAINER_*, ...hooks.spec.tsPlaywright_Test_Hookleaks;--workers=4 --fully-parallel --repeat-each=10: 1 failed; the old toggle "switch" resolved to the gateway-status button--workers=1 --repeat-each=10100/100 (four times);--workers=4 --fully-parallel --repeat-each=10with two co-tenant specs 210/210 (four times); all--retries=0The parent and first-commit reproductions run the new test module in those trees with one line back-ported: the
_UNDECLARED_HOME_OVERRIDESconstant the test reads, which is inert there; nothing else in those trees changes.Decoy check: decoy credential stores,
~/.awsfiles, a GitHub login and agent homes were planted in the scratchHOMEthe pytest process starts with, and an out-of-tree audit probe recorded every read of them. This branch's first commit opens both decoy credential stores during the route sweep; this PR opens none over five runs of the new test and the whole integration layer in CI shape (52 passed).The fake backend's hold has unit tests for release, cancel during the hold, expiry, a missing hold directory, a quoted marker and token validation (
test_fake_acp_backend.py; module coverage 83% → 93%).HOME pin: a probe recording every home the onboarding import scan resolves during the route sweep reads the operator's home on the parent and the empty tmp home with this PR (scan 16 s → 9 s for the test).
gil_hog(the card's named condition) also reds the parent (the witness names the same globals), but on this host it often pushes a boot past the harness's 60 s boot deadline in both parent and fix, soslow_threadis the reproducible proof. Unit owners of the fake and the shared helpers pass unmodified:test_fake_acp_backend.py,test_workflow_memory_backend_reset.py,test_e2e_recovery_wait.py(47 passed), plus 814 passed / 99 skipped across the targeted backend set, and 489 passed on the fake, identity-store, usage-API and host-isolation owners after the isolation fold (twice).local-gate.py --dry-runselects 1005 backend and 323 frontend test files; per the campaign's capacity rule the backend set ran narrowly as above and CI owns the broad set. Exact-SHA Linux, Windows and macOS lanes: in the PR checks.Follow-up outside this PR:
website/src/i18n/render-baseline.jsonreadsdeveloper-mcp-pool.layout 0 -> 2under the now-deterministic fallback face in the record-only path (noI18N_BASE_REF); CI decides with[vs-base], which compares like for like. A--build --updaterefresh belongs to the integration owner.Manual verification
N/A — every change is to a test or a test harness, and each is exercised by the forced-condition and repeat runs above.
Related Issues
Part of #16536
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)