Skip to content

ci: warn on a stale find_ui index; fail only at release or when a PR causes it - #18997

Merged
buluoray merged 1 commit into
mainfrom
ci/ui-index-freshness-context
Oct 11, 2026
Merged

buluoray merged 1 commit into
mainfrom
ci/ui-index-freshness-context

Conversation

@buluoray

@buluoray buluoray commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A stale committed find_ui index stops breaking every build on main, nightly, PRs and local machines, while a release and a PR that causes the staleness still fail on it.

src/kiro_crew/docs/ui-index.generated.json and guidePlans.gen.ts are committed, and the index digest covers every input, the 12 catalogs included. Each PR is fresh at its own head, but branch protection does not require an up-to-date branch, so fresh PRs merge into a stale main. npm run build started with a strict gen-ui-index.mjs --check, so a stale main broke main's Build, the nightly, every PR's Build Wheel / Build Desktop / E2E (merge ref), and every local make build. Five regen-only PRs were opened on 2026-10-10 alone (#18753, #18803, #18958, #18799, #18800).

Why it matters

Every open PR and every contributor's local build goes red for a staleness none of them caused, until someone lands a regen commit. Then the next merge can stale it again.

Not a goal

  • Making the index non-committed or generating it at merge time.
  • Requiring up-to-date branches or a merge queue.
  • Changing what the index contains or how find_ui reads it.

What changed (motivation → approach → change)

Symptom: a stale main fails every build. Cause: the only freshness gate was a whole-tree check inside npm run build, so it fires on whoever builds next, not on whoever caused it. Change: the check now depends on where it runs.

Where Stale index How
npm run build (local, make build, PR build lanes, main's Build, nightly) warns, build goes on gen-ui-index.mjs --check --warn-if-stale
frontend-lint on a PR / merge group, base fresh fails: this PR changed index inputs .github/scripts/ui_index_freshness.py checks the merge ref, then base.sha
frontend-lint on a PR / merge group, base already stale warns: main is stale same script
frontend-lint on a push to main warns same script
release.yml build lanes fails before any lane publishes ui_index_strict: true -> KC_UI_INDEX_STRICT=1 overrides --warn-if-stale
  • website/scripts/gen-ui-index.mjs: stale now exits 3 (1 stays "generator problem"); new --warn-if-stale (only with --check); KC_UI_INDEX_STRICT=1 turns it off. Under warn the auto tier is still written so the build can go on. The modes live in a small pure module, scripts/lib/ui-index-freshness.mjs. A generator problem still fails every mode.
  • .github/scripts/ui_index_freshness.py: classifies the merge ref; when stale on pull_request/merge_group, checks out base.sha as a detached worktree sharing node_modules and classifies that. A base from before exit 3 existed is read as stale by its message. An unreadable base fails closed.
  • build-wheel.yml, build-desktop.yml, build-windows.yml: new boolean input ui_index_strict (default false, declared on every trigger), mapped to the workflow-level env KC_UI_INDEX_STRICT. release.yml passes true on all three; nightly.yml does not.
  • Why strict lives inside npm run build and not in a separate pre-check job in release.yml: the check then runs on the exact tree each lane packages, in the same step that produces the bundle, so no release lane can build an index-mismatched bundle whatever job graph it sits in (a pre-check job only guards lanes that remember to need it). It also lets anyone reproduce a release build locally with KC_UI_INDEX_STRICT=1 npm run build. The cost is one boolean input per reusable build workflow, declared on each trigger because the env expression reads it on both.
  • Under --warn-if-stale in Actions, the generator also prints a ::warning annotation, so a stale index shows on the run page of a green build and not only in its log.
  • Docs in the same commit: docs/ci/ci-and-reviews.md (new subsection with the table), docs/build/release.md runbook step 2 (run npm run gen:ui -- --check on the target commit; if stale, land a regen on release/X.Y before the RC tag), docs/architecture/mcp.md, the generator header and the ci.yml step comment. docs/decisions/ records nothing about this check.

Backwards compatibility

Compatible: every command that passed before still passes; --check without the new flag still fails on a stale index (exit 3 instead of 1, both non-zero).

This loosens a gate (npm run build no longer fails on a stale index). Readers of that gate:

Reader: .github/workflows/release.yml:build-wheel -- release -- passes ui_index_strict, so its npm run build still fails; test_ui_index_freshness.py::TestWorkflowWiring::test_release_builds_are_strict
Reader: .github/workflows/release.yml:build-desktop -- release -- same input, read by packaging/build-desktop.sh's npm run build; test_release_builds_are_strict[build-desktop]
Reader: .github/workflows/release.yml:build-windows -- release -- same input; soft-fail lane as before; test_release_builds_are_strict[build-windows]
Reader: .github/workflows/ci.yml:frontend-lint -- release -- still fails a PR that stales a fresh base; test_a_pr_that_makes_a_fresh_base_stale_fails
Reader: .github/workflows/nightly.yml:build-wheel -- release -- now warns by design, never strict; test_nightly_never_asks_for_strict
Reader: src/kiro_crew/ui_index.py:_attach_auto -- chat -- a build from a stale tree ships an auto tier built against another index; the loader refuses it and answers auto_tier: "unavailable" with the committed tiers intact; test_find_ui_auto.py::test_a_bad_auto_file_disables_only_the_auto_tier

Tests

  • test/test_ui_index_freshness.py (24): classify (exit 0/3, the old exit-1 stale message, problems stay broken); main for each table row (PR-caused fail, merge-group fail, inherited warn, push warn, unreadable base / no base sha / broken base fail closed, generator problem fails everywhere); checkout_base against a real two-commit git repo; workflow wiring (ci.yml step env, release passes strict to all three, nightly never does, each reusable build maps the input to KC_UI_INDEX_STRICT with default false on every trigger).
  • website/src/uiLocations/uiIndex.test.ts "gen:ui --check freshness modes" (8): verdict for each mode, env parsing, npm run build carries --warn-if-stale, --warn-if-stale alone is refused, and the real generator on this tree (warn-only exits 0; strict equals plain --check).

Mutations, each run alone and restored, each reddened the named test:

Mutation Red test
freshnessVerdict ignores strict fails a stale tree again when KC_UI_INDEX_STRICT is set
warn branch exits 3 only warns on a stale tree under --warn-if-stale
generator passes strict: false wires the flag and the env into the real generator (*)
build script drops --warn-if-stale runs npm run build with --warn-if-stale...
decide passes a fresh base test_a_pr_that_makes_a_fresh_base_stale_fails, test_a_merge_group_that_makes_a_fresh_base_stale_fails
no old-message fallback in classify test_a_base_from_before_exit_3_reads_stale_by_its_message, test_a_pr_on_an_already_stale_base_only_warns
release drops ui_index_strict on build-wheel test_release_builds_are_strict[build-wheel]
nightly passes ui_index_strict test_nightly_never_asks_for_strict

(*) That one reddens only while the tree is stale (it was, before #18975 landed). On a fresh tree the three modes agree by construction, so it cannot tell them apart.

  • test/test_prepare_pr_profiles.py: the new script is named in exempt_scripts with its reason (the prepare-pr floor already runs the plain gen:ui -- --check; the base comparison needs an Actions event).

Also run: 7858 workflow/release/find_ui/build tests pass; src/uiLocations/ vitest 401 pass; eslint, black, isort, flake8 clean on touched files; docs-lint passes. actionlint is not installed on this host, so it was not run.

Manual verification

  • On a stale tree (main before chore(ui-index): regenerate ui-index.generated.json after #18431 #18975): npm run build exit 0 with the WARNING lines; KC_UI_INDEX_STRICT=1 npm run build exit 3.
  • The CI script run locally as a PR against a stale main (old generator at the base): exit 0, ::warning::... base ... is already stale. That exercised the old-exit-1 path.
  • After chore(ui-index): regenerate ui-index.generated.json after #18431 #18975 (fresh main), with an English label used by the index edited in the worktree: as a PR, exit 1 with ::error::This change makes the find_ui index stale; as a push, exit 0 with a warning.
  • Not run: a real release.yml run, so the inputs -> workflow-level env mapping is checked by a test reading the YAML, not by Actions executing it.

Related Issues

Refs #18995

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 11, 2026 00:56
@buluoray
buluoray requested a review from dodgeblaster October 11, 2026 00:56
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 11, 2026
@buluoray
buluoray enabled auto-merge (squash) October 11, 2026 00:57
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 66deb29370bfdd03126f6c191280de0620940d2b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Nothing to check.

[UX-REVIEWED] 66deb29

Description read

[DESCRIPTION-READ] db225922100be59818242c2494af55176fe853afe1f453b82e43d98b02c93a56

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 66deb29370bfdd03126f6c191280de0620940d2b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Once no open PR's base predates exit 3, check whether classify's exit-1 stale-message fallback in ui_index_freshness.py can be deleted.

What this change ships

Inventory (8 items) — 8 justified

Intent: stop a stale committed find_ui index from breaking builds for people who didn't cause it, while still failing releases and the PRs that cause it. This is a FIX.

  1. npm run build (local, make build, PR build lanes, main's Build, nightly) now warns on a stale index and keeps building — justified
  2. frontend-lint fails a PR or merge group only when its base was fresh and the merge result is stale; it only warns on an already-stale base — justified
  3. frontend-lint on a push to main warns instead of failing — justified
  4. Release build lanes still fail on a stale index, via the new ui_index_strict input and KC_UI_INDEX_STRICT=1 — justified
  5. The generator now exits 3 for a stale index, while 1 stays "generator problem"; a base from before exit 3 is recognised as stale by its message — justified
  6. A green build with a stale index shows a ::warning annotation on the run page — justified
  7. --warn-if-stale without --check is refused — justified
  8. Release runbook step 2 now says to regenerate on release/X.Y before tagging the RC — justified

[FIRST-PRINCIPLES-REVIEWED] 66deb29

Description read

[DESCRIPTION-READ] 30dc1a56a879eaa96ae6d4d5cd436c08e2e168ea25025a97e0bf1293b29d5793

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 66deb29370bfdd03126f6c191280de0620940d2b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Once main is stale, the PR gate stops failing anything, so a PR that adds its own staleness passes until the next release tag forces a regen.

Watch

  • The "PR caused it" gate turns off whenever main is stale. The decide() row stale / stale → pass with a warning asks only whether the base is already stale. It never asks whether this PR changes the regenerated output. So once one merge stales main, every later PR that edits index inputs without regenerating also passes, with a warning. Nothing pushes anyone to regen except the ui_index_strict release lanes. The description's goal, "a PR that causes the staleness still fail on it", only holds while the base is fresh. Meanwhile drift builds up, nightlies ship find_ui answers pointing at paths that no longer exist, and the regen work lands on the release captain at RC time.
    Clears when: on a stale base, the script compares the base's regenerated bytes with the merge ref's and fails when they differ, so a PR that changes index inputs must regen (which also fixes main). Or: the goal line and docs/ci/ci-and-reviews.md state that the gate is off while main is stale.

[DESIGN-REVIEWED] 66deb29

Description read

[DESCRIPTION-READ] 5a76b207b1586fe88fde1a4593355c5e0393c8ba0fd2cc7595ceb025bba10865

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 66deb29370bfdd03126f6c191280de0620940d2b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 66deb29

Verdict parsed from the review's SHA-scoped output markers for commit 66deb29370bfdd03126f6c191280de0620940d2b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 66deb29370bfdd03126f6c191280de0620940d2b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 66deb29370bfdd03126f6c191280de0620940d2b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 66deb29

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 66deb29370bfdd03126f6c191280de0620940d2b: <one-sentence reason>

@buluoray
buluoray force-pushed the ci/ui-index-freshness-context branch from e84dbdb to 4fdf86f Compare October 11, 2026 01:12
@buluoray

Copy link
Copy Markdown
Contributor Author

Dispositions for the e84dbdb round, fixed or answered in 4fdf86f:

  • Backend Lint (subprocess-encoding gate): real, from this diff. Two text-mode subprocess.run calls lacked encoding="utf-8"; both pinned.
  • Frontend Tests (1): ChatPage.dirStaging.test.tsx failed one case at 1949ms. Not in this diff; the file passes 66/66 locally on this head. Treated as a shard flake.
  • First Principles CONCERNS (item 4, strict mode via input + env): kept on purpose. The strict check runs inside npm run build, on the exact tree each release lane packages, so no lane can build a mismatched bundle whatever job graph it sits in. A separate pre-check job only guards the lanes that need it. It also makes a release build reproducible locally with KC_UI_INDEX_STRICT=1 npm run build. Recorded under What changed. Item 6 (annotation) is now declared there too.
  • Design PASS note (who regenerates; main may stay stale until the RC): that lag is the accepted trade-off. The release runbook step 2 now runs gen:ui -- --check on the target commit and lands a regen on the release branch before the RC tag, so it is caught before soak, not at the release tag.
  • UX PASS note (auto tier unavailable on a stale build): the loader already degrades to the committed tiers with auto_tier: "unavailable"; pinned by test_a_bad_auto_file_disables_only_the_auto_tier.

@buluoray

Copy link
Copy Markdown
Contributor Author

Backend Lint on 4fdf86f is inherited from main: isort flags test/test_crew_log_types.py (unsorted lease import from #18984), a file this PR does not touch and which fails isort identically on main. #18999 fixes it. Once that merges this PR needs a rebase onto main to clear the lane; no change here.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@buluoray
buluoray force-pushed the ci/ui-index-freshness-context branch from 4fdf86f to b19439b Compare October 11, 2026 01:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

First Principles item 4 on b19439b (strict via input + env vs one pre-build job): kept. The maintainer chose this shape when the design was agreed: release's calls to the reusable build workflows pass an input that makes the build's own check fatal. A path one pre-build job cannot cover: any future caller of build-wheel/desktop/windows that publishes and forgets to need the gate job, plus the window between that job's checkout and each lane's own checkout. Here the strict check runs on the exact tree each lane packages. Advisory, so no change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
… for PR-caused staleness

npm run build now runs gen-ui-index --check --warn-if-stale, so a stale
committed index warns instead of failing local builds, PR build lanes,
main's Build and nightly. CI's freshness step fails a PR only when its
base is fresh and its merge ref is stale; an already-stale base warns.
release.yml passes ui_index_strict to its build lanes, which sets
KC_UI_INDEX_STRICT=1 and makes the build fail before any lane publishes.
The release runbook regenerates on the release branch before the RC.
@buluoray
buluoray force-pushed the ci/ui-index-freshness-context branch from b19439b to 66deb29 Compare October 11, 2026 03:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@buluoray
buluoray merged commit e6e42a2 into main Oct 11, 2026
103 checks passed
@buluoray
buluoray deleted the ci/ui-index-freshness-context branch October 11, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants