Skip to content

test(ci): add backend-test canary runs-on to resolver-consumer census - #11876

Merged
iamwhatever merged 1 commit into
mainfrom
fix/parity-census-backend-test-canary
Sep 18, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/parity-census-backend-test-canary

Conversation

@helenastafford

@helenastafford helenastafford commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Main's own CI is red. test/test_ci_fleet_routing_expression_parity.py fails on every run. The census test finds a new runs-on: that reads needs.changes.outputs.linux_runner_large in ci.yml's backend-test job (the shard-1 CodeBuild canary), but that job is not in _EXPECTED_RESOLVER_CONSUMER_JOBS.

Why it matters

Every open PR inherits the red through its merge ref. One backend shard fails for all of them, no matter what they change. Nobody can tell a real failure from this one.

What changed (motivation → approach → change)

The census test locks every resolver read to its exact expression text. The canary read is real and correct, so the fix is to list it, not to loosen the test.

One entry is added to _EXPECTED_RESOLVER_CONSUMER_JOBS in test/test_ci_fleet_routing_expression_parity.py: ("ci.yml", "backend-test"), bound to the canary's exact conditional expression as a named constant _CANARY_SHARD1_CONSUMER_EXPR. The byte-match branch of the test now pins the canary form too. A hardcoded fleet label or a dropped resolver read in that job still fails the test.

Backwards compatibility

Compatible: test-only change. No runtime code or workflow file is touched.

Tests

  • test/test_ci_fleet_routing_expression_parity.py — 3/3 pass with this change. Without it, test_every_ci_yml_resolver_consumer_reads_the_resolver_not_a_literal fails on main tip.

Manual verification

N/A — unit coverage sufficient; the change is one entry in a test's expected set.

Related Issues

Fixes #11883

Pattern harvest

Rule candidate: review-prompt — a PR that adds or changes a runs-on: reading the fleet resolver must update _EXPECTED_RESOLVER_CONSUMER_JOBS in the same commit.
Pattern: a guard test owned by one file and the behavior it guards owned by another, merged separately, each green alone and red together.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@helenastafford
helenastafford requested review from a team and chenmingwei23 September 18, 2026 19:43
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of fbc0cb0ecb4d3a724820576e01ab4f8e14196607 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fbc0cb0

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fbc0cb0ecb4d3a724820576e01ab4f8e14196607: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of fbc0cb0ecb4d3a724820576e01ab4f8e14196607 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm the canary runs-on at ci.yml:598 landed via its own reviewed PR — this entry ratifies it as correct rather than reverting drift.

[DESIGN-REVIEWED] fbc0cb0

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed fbc0cb0ecb4d3a724820576e01ab4f8e14196607 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fbc0cb0

Verdict parsed from the review's SHA-scoped output markers for commit fbc0cb0ecb4d3a724820576e01ab4f8e14196607.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fbc0cb0ecb4d3a724820576e01ab4f8e14196607: <one-sentence reason>

@pepmach

pepmach commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@helenastafford -- thanks for the quick allowlist heal. Main has been red on test_every_ci_yml_resolver_consumer_reads_the_resolver_not_a_literal (Linux(2) + Windows(2)) since the 19:02 push, and because pull-request CI runs the merge commit every open PR carries the pair right now, so this is the unblocking change for the whole board. On head 65fe8db80d the PR's own lanes still show Await Fast Gate, Comment History Gate and Coverage Gate red (the annotation on the Comment History Gate job says what it wants for the edited test); macOS(1) is the unrelated #11884. Could you clear those so it can be reviewed?

@bolichen97 -- flagging as #11699's owner in case you would rather land the ('ci.yml', 'backend-test') entry yourself; either way one heal is enough, nobody else should open a second.

…census

#11699 routed backend-test shard 1 to the CodeBuild fleet via a
canary-conditional runs-on that reads needs.changes.outputs.linux_runner_large,
but did not add the job to _EXPECTED_RESOLVER_CONSUMER_JOBS in
test_ci_fleet_routing_expression_parity.py. The census test's added-consumer
assertion now fails on main for every PR (and main itself).

Add the entry with its exact canary expression so the byte-match check also
pins the canary form against silent drift.
@iamwhatever
iamwhatever force-pushed the fix/parity-census-backend-test-canary branch from 65fe8db to fbc0cb0 Compare September 18, 2026 21:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@iamwhatever
iamwhatever merged commit 1ae8f6e into main Sep 18, 2026
90 of 92 checks passed
@iamwhatever
iamwhatever deleted the fix/parity-census-backend-test-canary branch September 18, 2026 22:25
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…census (kirodotdev#11876)

kirodotdev#11699 routed backend-test shard 1 to the CodeBuild fleet via a
canary-conditional runs-on that reads needs.changes.outputs.linux_runner_large,
but did not add the job to _EXPECTED_RESOLVER_CONSUMER_JOBS in
test_ci_fleet_routing_expression_parity.py. The census test's added-consumer
assertion now fails on main for every PR (and main itself).

Add the entry with its exact canary expression so the byte-match check also
pins the canary form against silent drift.

Co-authored-by: Helena Stafford <helenars@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI red on main: ci.yml routing parity test fails because backend-test became a resolver consumer in #11699 without updating the fixed expectation

3 participants