Skip to content

test(e2e): wait on the fork response, not the transient "Fork of" title - #19063

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/e2e-fork-spec-flake
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/e2e-fork-spec-flake

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The offline E2E job stops failing on fork.spec.ts "fork button appears on assistant message and creates new tab", which passes only on a retry under load.

On #19040 (a diff with no fork code) the "E2E (stub ACP backend, offline)" job failed with 1 spec(s) passed only on a retry, ceiling is 0: fork.spec.ts > Fork Session E2E > fork button appears on assistant message and creates new tab. The first attempt timed out at await expect(page.getByText(/Fork of /).first()).toBeVisible({ timeout: 10000 }) with "element(s) not found".

Why it matters

The suite's flaky ceiling is 0, so every PR that draws this race goes red on a job unrelated to its diff and has to push again to clear it.

Not a goal

  • No change to the fork title lifecycle (chat_fork.fork_slot, refresh_forked_title). Renaming a fork straight after it opens is the shipped behaviour from fix(chat): refresh stale inherited titles after conversation fork #18231.
  • No retry, timeout or flaky-ceiling change.
  • The second test in the file (Forking from long timestamped sessions) is left alone: its imported session has no person-typed question, so refresh_forked_title returns without renaming and its Fork of assertion cannot race.

What changed (motivation → approach → change)

  • Symptom: Fork of text never appears after clicking Fork.
  • Root cause: the spec races the app. fork_slot births the child as ↳ Fork of <parent>, answers the POST, then starts refresh_forked_title. That pass renames the child to ↳ <generated title> and pushes slot_title. The stub ACP backend answers the title prompt at once (pong from the fake ACP backend), so on a loaded runner the rename lands before the client paints the new row and the birth title is never on screen. The failing page snapshot shows the fork row as ↳ pong from the fake ACP backend.
  • Change (spec only): wait on the POST /fork response. Assert it is OK, that its title is the ↳ Fork of birth title, that the URL switches to the returned key, and that the sidebar row for that key ([data-slot-key] [data-session-title]) starts with the ↳ marker, which survives the rename. Each step is synchronised on a specific response or element for the fork's own key, not a fixed wait or a page-wide text match.
  • Why not wait longer or make the stub's title pass slower: the birth title is meant to be replaced, so any spec that has to see it on screen is timing-dependent, and the response already carries it deterministically.

Backwards compatibility

Compatible: test-only change to one Playwright spec.

Removes nothing: no product code, route, key or record changes.

Tests

  • website/playwright/fork.spec.ts "creates new tab": now pins a fork key that differs from the parent's, the fork response's birth title, the switch to the fork's key, and the ↳ marker on that key's row. The URL and row assertions keep the 10 s budget of the line they replace.

Local runs against the harness gateway (spawn_feature_gateway, stub ACP backend), --retries=0, Chromium CPU throttling via CDP Emulation.setCPUThrottlingRate in a scratch copy of the spec:

  • origin/main spec, throttle 4x, --repeat-each=30 --workers=2: 29 failed, all at the Fork of line.
  • this spec, throttle 4x, same settings: 30/30 passed.
  • this spec, unthrottled, whole file, --repeat-each=30 --workers=2: 61 passed (60 plus setup).
  • after the review nits (parent-key check, 10 s budgets): whole file unthrottled --repeat-each=15 --workers=2, 31 passed (30 plus setup); throttle 4x --repeat-each=10, 10/10 passed.

Gates on the head: website npm run typecheck, eslint and a strict tsc --noEmit on the spec; scripts/check_*.py guards, including comment-history and changelog-history against origin/main, harness parity from the merge-base, and docs_lint.

Manual verification

N/A: the change is a Playwright spec, and it was exercised above under load.

Related Issues

Fixes #19049

Pattern harvest

Rule candidate: review-prompt
Pattern: an E2E assertion on a value the app deliberately replaces moments later (a birth title, a placeholder, a pending label) in place of the response or state that carries it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 11, 2026 07:19
@jeeshofone
jeeshofone requested a review from CrysisDeu October 11, 2026 07:19
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ✅ PASS

UX-level review of 0533ceb61a61972aa133f37e9827d6de31b93eb3 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Nothing to check.

[UX-REVIEWED] 0533ceb

Description read

[DESCRIPTION-READ] a94ac67088aac4d97c3b5bcdec7467597961945b94b03f32f2ce45e72d110349

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 0533ceb61a61972aa133f37e9827d6de31b93eb3 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 0533ceb61a61972aa133f37e9827d6de31b93eb3 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm that [data-slot-key] [data-session-title] are real sidebar attributes, so the ↳ row assertion can actually fail if the row breaks.

[DESIGN-REVIEWED] 0533ceb

Description read

[DESCRIPTION-READ] 2ef74d974eaeb9e96226f4473d90bbb5f360f7d53e9ea272f89e19a86d2d0529

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 0533ceb61a61972aa133f37e9827d6de31b93eb3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0533ceb

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 0533ceb61a61972aa133f37e9827d6de31b93eb3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0533ceb

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for head 1e99741.

Design (PASS) punchline, sidebar stub rows: checked. A windowed row renders as a stub only when it is not the active one. website/src/pages/chat-sidebar/rows.ts windowOf sets keepMounted: isActive(slot), and the spec asserts the URL switched to the fork's key (?sid=<forkKey>) before it reads the row, so by then the row is active and fully mounted with [data-session-title]. No change needed.

Backend Tests (Windows) (2): test/test_session_control.py::test_the_inter_stage_append_persists_before_returning_success. That is the open flake #14218 (Windows executor hop outruns the zero-sleeps). This PR changes one Playwright spec and no Python. I'll push one content-identical fresh head to retry it.

E2E (stub ACP backend, offline): passed on this head with no retry.

First-pass check: R1-R8 OK. The diff is test-only, Live decisions has no entry for this PR, and no Python changed, so the R8.2 Python lint and mypy floor does not apply. Website typecheck and eslint ran, and so did every scripts/check_*.py guard.

@jeeshofone
jeeshofone force-pushed the fix/e2e-fork-spec-flake branch from 1e99741 to 2d8d860 Compare October 11, 2026 08:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for head 2d8d860, Design (PASS) punchline on stub rows. This is the same point as on 1e99741 and the code has not changed. A row stubs only when it is not active. website/src/pages/chat-sidebar/rows.ts windowOf returns keepMounted: isActive(slot), and WindowedSessionRow renders the real row whenever keepMounted is set. The spec asserts ?sid=<forkKey> first, so the fork's row is the active one and carries [data-session-title] by the time it is read.

Backend Tests (Windows) (2) passed on this head, so the #14218 flake did not recur. E2E (stub ACP backend, offline) passed with no retry on both heads.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Delta for head 76ed01715 (amends the single commit on the same base 3dc503d92; spec only):

  • The "creates new tab" test now records the parent key in beforeEach and asserts expect(forkKey).not.toBe(parentKey) before the URL check, so "new tab" is checked against the parent rather than assumed.
  • The toHaveURL and [data-session-title] toHaveText assertions pass { timeout: 10000 }, the budget of the line they replaced, so the fix does not cut CI headroom.

Local runs on the harness gateway (stub ACP backend, Chromium, --retries=0): whole file unthrottled --repeat-each=15 --workers=2, 31 passed (30 plus setup); "creates new tab" under 4x CDP CPU throttle --repeat-each=10, 10/10 passed. Website typecheck, strict tsc --noEmit on the spec, comment-history, changelog-history and the description check (fork-check and pr-hygiene callers) pass.

Independent review dispositions (APPROVE on 2d8d860d6):

  • NIT 1 (10 s budgets reduced to the 5 s default): taken, both assertions.
  • NIT 2 (no parent-key check): taken.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 11, 2026
@jeeshofone
jeeshofone force-pushed the fix/e2e-fork-spec-flake branch from 2d8d860 to 76ed017 Compare October 11, 2026 10:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
The fork spec asserted that some "Fork of " text became visible after
clicking Fork. The child is born as "↳ Fork of <parent>", but its
background title pass renames it straight after the fork answers, and the
stub ACP backend answers that pass at once. Under load the rename lands
before the new row paints, so the text never appears and the spec only
passes on a retry.

Wait on the POST /fork response instead: assert its birth title, then the
switch to the returned key and the "↳ " marker on that key's sidebar row,
which survives the rename.

Fixes kirodotdev#19049
@jeeshofone
jeeshofone force-pushed the fix/e2e-fork-spec-flake branch from 76ed017 to 0533ceb Compare October 12, 2026 04:19
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Delta for head 0533ceb61 (rebase only; no code change).

The previous head 76ed01715 sat on base 3dc503d92. Two checks on it were red for reasons outside this diff:

git range-diff between the old and new head shows =: the single commit is unchanged. The only file this PR touches, fork.spec.ts, is not touched by main since the old base, so the rebase is textually and semantically clean. Local checks on the new head: npm run typecheck, eslint on the spec, comment-history and changelog-history against origin/main.

All review lanes on the previous head were PASS or no-findings (Opus, GPT, Design, UX; First Principles skipped because the diff ships no capability), so no per-span dispositions are owed.

First-pass check: R1-R8 OK (spec-only; no new state, no sensitive path, no new strings).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for head 0533ceb, Design (PASS) punchline on the sidebar selectors. Confirmed in the app source: website/src/pages/ChatSidebar.tsx renders data-session-title, and the slot row carries data-slot-key (stubbed rows repeat it in the same place, website/src/pages/chat/sessionRowWindow.tsx:158-159). The ↳ assertion fails if the row for the fork's key is missing or its title loses the marker. The spec is unchanged since 76ed01715 (this head is a rebase only), and the earlier dispositions on 1e9974117 and 2d8d860d6 cover the stub-row point.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This de-flakes the offline E2E fork spec at its cause. The test was asserting a birth title, "↳ Fork of ", that refresh_forked_title deliberately replaces moments later, and the stub backend answers that title pass at once. The spec now waits on the POST /fork response. It asserts the response is OK, that its key differs from the parent's, and that its title is the birth title. It then waits for the URL to switch to that key and for that key's sidebar row title to start with the ↳ marker, which survives the rename. I checked the contract the spec relies on. api_chat_slot_fork returns key and title (chat_fork.py:396-401). The sidebar row carries data-slot-key={s.key} with a nested data-session-title (ChatSidebar.tsx:1679, 2133). The only POST ending in /fork on this page is forkChatSlot. The author shows 29/30 failures with the old spec under 4x CPU throttling and 30/30 passes with the new one. Product code is untouched, checks are green, and both AI lanes pass. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever merged commit 9348afa into kirodotdev:main Oct 12, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

E2E flake: fork.spec 'creates new tab' asserts the transient "Fork of" birth title

2 participants