Skip to content

fix(cloud): accept the interactive tier keys on the Fargate lane - #11534

Merged
bolichen97 merged 1 commit into
mainfrom
fix/fargate-tier-size-keys-11328
Sep 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/fargate-tier-size-keys-11328

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

What is the problem?

Driven on origin/main, all three EC2 tier keys are refused by the Fargate lane:

DEFAULT_TIER_KEY = balanced
_parse_size('light')     -> ValueError: size 'light' is not a Fargate cpu/memory pair
_parse_size('balanced')  -> ValueError: size 'balanced' is not a Fargate cpu/memory pair
_parse_size('power')     -> ValueError: size 'power' is not a Fargate cpu/memory pair
_parse_size('1024/2048') -> TaskSize(cpu='1024', memory='2048', ephemeral_storage_gib=None)

The RFC's Sizing section promises the opposite, twice: "The existing size keys
keep working", and "the Fargate backend maps the same three keys to CPU and memory
pairs, so a launch that does not name a backend behaves as it does today".
fargate_engine.py never imports sizes at all.

The key refused is the one a caller gets by not choosing:
sizes.DEFAULT_TIER_KEY is balanced.

Why this issue matters to the user

It is latent today, and only for one reason: nothing registers the lane, so
engine_for resolves aws_ec2 alone and no caller can reach the refusal. It
stops being latent in the very change that registers the lane -- at which point a
user who never mentioned Fargate, and whose size key is therefore a tier name,
gets a refusal from a backend they did not ask for. That is why this lands first,
separately, rather than riding along with registration.

How our fix solves it

_tier_as_pair respells a tier key into Fargate's own
"<cpu>/<memory>/<gib>" vocabulary, and _parse_size accepts both spellings.

The shape is derived from sizes.py's tier table, not written out again here:

light     vcpu=4  ram_gb=16  disk_gb=40  ->  4096/16384/40
balanced  vcpu=8  ram_gb=32  disk_gb=60  ->  8192/32768/60
power     vcpu=16 ram_gb=64  disk_gb=80  ->  16384/65536/80

That table is where a tier's vCPU, RAM and disk are decided, and it has been
changed before -- its own comment records a raised ladder while the keys stayed.
Deriving means such a change reaches this lane; a second table here would keep
answering with the retired shape, and nothing would compare the two.

It returns a key, not a finished TaskSize, so the derived numbers run
through exactly the bounds check a hand-written pair does -- against
FARGATE_MEMORY_FOR_CPU, its minimum, maximum and step. A tier whose shape ever
leaves Fargate's own table is then refused by name rather than handed to
RunTask. All three are legal today, which is asserted rather than assumed.

Only the three interactive keys are respelled. light-x86 is a real ladder key
and deliberately not one of them: mapping it would silently pick an architecture
the caller did not ask this lane for.

All four refusal messages name the key the caller passed. The docstring
claims a refusal "names the key the caller passed, never its respelling", and four
branches named only resolved numbers -- cpu-not-in-table, memory-out-of-range, and
both ephemeral-storage branches -- so someone who typed power read back
cpu='16384000' or a bare GiB count, with nothing to search for. Each of the four
is reached the same way, by moving the ladder until a tier derives an illegal
shape, and each now carries the caller's key. That keeps the docstring's claim
universal rather than narrowed to the pairs it happened to cover. This is the
defect this line keeps producing: prose asserting what the adjacent code does not
do, and the tests that force those paths are what find it.

What tests we did

Eight new tests in test/test_fargate_launch_engine.py, twelve collected cases
once the parametrised ones expand; the file goes 48 -> 60 and every existing one
still passes. Siblings run too:
test_fargate_runtask.py, test_fargate_taskdef.py, test_cloud_sizes.py and
test_cloud_launch_job.py -- 284 passed.

Expectations are derived from the ladder, not restated, so a raised ladder moves
the test with the code instead of pinning a retired shape.

Every assertion mutation-verified:

# mutation caught
1 the mapping always declines, so tier keys are refused again yes
2 the shape is HARDCODED instead of derived from the ladder yes
3 GiB conversion uses 1000, so memory leaves Fargate's table yes
4 the disk field is dropped, losing the tier's storage yes
5 an x86 ladder key is respelled too yes
6 the refusal names the respelling instead of the caller's key yes
7 the tier keys are accepted but not listed as legal yes
8 every key is respelled, breaking Fargate's own vocabulary yes
9 the storage-out-of-range refusal drops the caller's key yes
10 the non-whole-GiB refusal drops the caller's key yes

10 mutations, 0 holes.

Row 6 is worth reading, twice over. Its first version pointed at the wrong
test, and the matrix reported it caught -- because the mutation only affects the
refusal path, which a tier key never reaches on today's ladder. So the docstring's
claim was untestable with real inputs. The test that fixes that forces the path by
moving the ladder (a thing that has happened) so a tier derives an illegal shape.
Running it revealed the claim was simply false, which is the code fix above.

The second lesson from that row: a mutation result means nothing while the clean
tree is red.
The matrix printed "8 mutations, 0 holes" at a moment when the new
test failed on an unmutated tree, because a test that reds either way reds under
mutation too. Clean-tree green is now checked before the matrix is believed.

Gates run locally: every scripts/check_*.py the Backend Lint lane invokes, plus
isort, flake8, ruff, black --target-version py310, the comment-history gate and
the brand-name gate.

Any other suggestions on the work

Fixes part of #11328

Pattern harvest

Rule candidate: a docstring sentence asserting a property must have a test that
FORCES the path it describes. Here _parse_size's docstring claimed "a refusal
names the key the caller passed, never its respelling" while two branches named
only the resolved numbers -- and the claim was unreachable with real inputs,
because a tier key cannot fail on today's ladder. Writing a test that moves the
ladder to reach that path is what showed the claim was simply false. A property
whose path no test can reach is a property nothing is checking, and prose is the
worst place to keep it. The same sentence then stayed false in a narrower way:
the two branches the first test reached were fixed while both ephemeral-storage
branches kept naming resolved numbers. A property stated for "a refusal" is a
claim about EVERY refusal in the function, so enumerate the raise sites and cover
them all, or narrow the sentence to the set a test actually forces.

Rule candidate: a mutation matrix means nothing until the clean tree is green.
This one printed "8 mutations, 0 holes" at a moment when the new test failed on an
UNMUTATED tree -- a test that reds either way also reds under mutation, so every
row read as caught. Assert clean-tree pass first, then read the matrix.

Rule candidate: when a test's expectation restates a table the code derives from,
the test agrees with itself rather than with the code. The tier expectations here
are computed from sizes.get_tier for that reason: the ladder has been raised
before with the keys unchanged, and a second copy of the numbers would keep
asserting the retired shape while passing.

@chenmingwei23
chenmingwei23 requested review from a team and hoang-phan98 September 17, 2026 12:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1ed103d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the tier's disk_gb riding into billed Fargate ephemeral storage — the RFC promises "CPU and memory pairs"; the disk field is this PR's own extension.

[DESIGN-REVIEWED] 1ed103d

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The tier's disk maps to ephemeral storage too — the RFC promises "CPU and memory pairs"; confirm the third field is intended, not scope creep.

What this change ships

Inventory (4 items) — 4 justified

Intent: make the Fargate lane accept the three EC2 tier keys before the lane is registered, as the in-progress Fargate RFC promises — a FIX (reproduction in description; tests fail on base; rfc-remote-instance-on-fargate.md is in-progress on the base list and names this shape twice).

  1. light/balanced/power now parse on the Fargate lane, derived from the tier table — justified
  2. A tier's disk size becomes the Fargate ephemeral-storage field — justified
  3. The unusable-size refusal now lists the three tier keys as legal — justified
  4. Four refusal messages now name the key the caller passed instead of resolved numbers — justified

No duplicate mechanism exists (grep vcpu * 1024 / tier→Fargate mapping: only fargate_engine.py); the fix sits at cause level (the parse site never consulted sizes.py); the lane is unregistered (defaults.py:675 resolves aws_ec2 alone), so no shipped default changes.

[FIRST-PRINCIPLES-REVIEWED] 1ed103d

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1ed103d

Verdict parsed from the review's SHA-scoped output markers for commit 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1ed103d40bf74fa70cd46bb15fd95d44371c5e4d: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 3) on this head is main's, not this change's -- measured, not assumed.

The failure is:

kiro_crew.pod.windows.WindowsTaskError: the Task Scheduler pod backend is win32-only; this host is linux.
src/kiro_crew/pod/windows.py:179: WindowsTaskError

Three separate reasons it cannot be this PR:

  1. The diff cannot reach it. This branch touches exactly two files:
    src/kiro_crew/cloud/fargate_engine.py and test/test_fargate_launch_engine.py.
    Neither is imported by, nor imports, kiro_crew.pod.windows.
  2. It is already filed. test_pod_windows_run fails on Linux since #10707: up now reaches windows.require_backend #11517 -- "test_pod_windows_run fails on Linux since
    fix: diagnose an unreachable pod user bus #10707: up now reaches windows.require_backend" -- is open and names the same
    test and the same mechanism.
  3. It is not specific to this head. The same shard is failing on feat(prepare-pr): tell the loop when a PR's green has expired #11508, an
    unrelated pull request of mine, and was cancelled rather than run on feat(cloud): create the Fargate cluster, per-crew roles and log group #11521.

Locally, on this branch: test/test_fargate_launch_engine.py is 59 passed (58
before this change, all still passing), and the sibling suites
test_fargate_runtask.py, test_fargate_taskdef.py, test_cloud_sizes.py and
test_cloud_launch_job.py are 224 passed. The whole Backend Lint gate set,
isort, flake8, ruff, black --target-version py310, the comment-history gate and
the brand-name gate all pass, and the 8-mutation matrix for the new assertions has
zero holes.

Nothing to do on this branch for that lane; it clears when #11517 does.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-tier-size-keys-11328 branch from 338873d to 8846bbc Compare September 17, 2026 16:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-tier-size-keys-11328 branch from 8846bbc to ce8500d Compare September 17, 2026 17:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-tier-size-keys-11328 branch from ce8500d to 82458f2 Compare September 17, 2026 19:07
@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • span=2b08daeadf0d -- FIXED, and not by narrowing the claim.

The finding is real. _parse_size's docstring claims a refusal "names the key
the caller passed, never its respelling", and both ephemeral-storage branches
named only the resolved gib / storage value.

I did not take the suggested remedy of narrowing the sentence to CPU/memory
refusals. The storage branch is reachable by the same route as the CPU one and
carries the same defect: a tier's GiB arrives from the DERIVED respelling, so a
moved ladder makes a caller who typed power read back a GiB count they never
wrote. Narrowing the sentence would document that hole instead of closing it,
in a change whose whole point is that the lane must answer a tier key honestly.

Both branches now name size_key, in the same shape the CPU and memory
branches already use, so the claim holds for all four raise sites in the
function. A new test forces both paths: the non-whole-GiB path with a
hand-written 1024/2048/xx, and the out-of-range path by moving disk_gb one
GiB past Fargate's ceiling while vcpu and ram_gb stay legal so the storage
branch is the one reached. Mutation-verified -- dropping size_key from either
message reds the test, which is matrix rows 9 and 10.

The PR body is updated to the four-branch state, and its rule candidate now
says a property stated for "a refusal" is a claim about every raise site in the
function: enumerate them, or narrow the sentence to the set a test forces.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • span=6be8a1274957 -- ACCEPTED, and deferred to the change that registers the lane.

Agreed that a decision resting on a test docstring and a PR body is not
durable. The reason it is not in a spec here is that this change does not
register the Fargate lane: engine_for('aws_fargate') still raises KeyError,
so there is no owning spec section for the lane's size vocabulary to land in
yet.

It belongs in the change that registers the lane together with a configuration
home for FargateLaunchSpec, which is the remainder of #11328. A spec
paragraph written here would describe a lane no caller can select, and the two
would drift from each other before anything could use either.

Recorded so the next contributor does not read the exclusion as a gap: the
three -x86 ladder keys stay unmapped because mapping them would take the
architecture from the key rather than from the caller, and that is a separate
decision from accepting the three interactive keys.

Driven on main, all three EC2 tier keys are refused by the Fargate lane:

  DEFAULT_TIER_KEY = balanced
  _parse_size('light')    -> ValueError: size 'light' is not a Fargate cpu/memory pair
  _parse_size('balanced') -> ValueError: size 'balanced' is not a Fargate cpu/memory pair
  _parse_size('power')    -> ValueError: size 'power' is not a Fargate cpu/memory pair
  _parse_size('1024/2048') -> TaskSize(cpu='1024', memory='2048', ...)

The RFC's section 9 promises the opposite twice -- "The existing size keys keep
working", and "the Fargate backend maps the same three keys to CPU and memory
pairs, so a launch that does not name a backend behaves as it does today". And
the key refused is the one a caller gets by NOT choosing: sizes.DEFAULT_TIER_KEY
is balanced.

It is latent today only because nothing registers the lane -- engine_for
resolves aws_ec2 alone -- so no caller can reach the refusal. It stops being
latent in the change that registers it, which is why this lands first.

_tier_as_pair respells a tier key into Fargate's own "<cpu>/<memory>/<gib>"
vocabulary, DERIVED from sizes.py's tier table rather than written out again.
That table is where a tier's vCPU, RAM and disk are decided and it has been
changed before; deriving means such a change reaches this lane, while a second
table here would keep answering with the retired shape and nothing would compare
the two. It returns a KEY, not a finished TaskSize, so the derived numbers run
through the same bounds check a hand-written pair does -- a tier whose shape ever
leaves Fargate's table is refused by name instead of being handed to RunTask.

Two refusal messages also now name the key the caller passed. The docstring said
a refusal "names the key the caller passed, never its respelling" while the
cpu-not-in-table and memory-out-of-range branches named only the resolved
numbers, so someone who typed `power` read back `cpu='16384000'` and had nothing
to search for. The test that forces that path is what found it.

Fixes part of #11328
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-tier-size-keys-11328 branch from 82458f2 to 1ed103d Compare September 17, 2026 22:31
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Board note: the inherited reds cleared, and this PR was never changed for them

Rebased onto main's tip 2627b0407c, fix(session-control): authorize a created child on the store it is bound to (#11593). Recording why the board changed so
it does not have to be reconstructed.

Before the rebase, head 82458f2359 read 84 lanes = 67 success + 12 skipped + 5
failure. All five came from main:

lane tests it named
Backend Tests (3.12, 5), (Windows) (5) test_member_session_control.py::TestMemberChildPrivateBinding::test_a_v1_child_writes_no_private_binding, ::test_global_caller_cannot_grant_a_child_private_authority
Backend Tests (3.12, 6), (Windows) (7) test_session_control.py::test_create_refuses_unreadable_delegation_identity[OSError], [ValueError], test_private_creator_cannot_change_memory[public-worker-canonical], [writer-canonical]
Coverage Gate fail-closed cascade off the four above

That attribution came from main's own CI rather than from inference. Two separate
main runs failed exactly those five lanes with exactly those six test ids: run
35254907186 on head 0640c1e0d5 (#11438, which introduced them) and run
35268597516 on head c5842a635, three commits later. This PR saw them only
because pull_request checks run on the merge ref, and two of the six tests do
not exist on the base this branch was cut from. It was tracked in #11563,
#11581, #11583 and #11585.

#11593 is the fix, and all five lanes are success on its head 6eefcae8d.

Nothing in this diff was changed for any of it. The two files are byte-identical
across the rebase, still one commit, +194/-17. Local gates re-run against the new
base: 21 of 21 python gates pass, including
run_scoped_tests.py --surface backend, and test_fargate_launch_engine.py with
test_fargate_runtask.py, test_fargate_taskdef.py, test_cloud_sizes.py and
test_cloud_launch_job.py give 284 passed.

One earlier red is worth naming so nobody rediscovers it. On head 8846bbcebc,
Backend Tests (Windows) (3) failed
test_fairness_lanes.py::test_wait_resume_granted_wakes_on_the_grant_not_on_a_timer,
a loop.time() - t0 < 0.1 wall-clock bound that landed on main in #10521. That
one was also unreachable from this diff and cleared on its own. Shard numbers
cannot be compared across branches here: pytest-split finds no .test_durations
in CI so it splits by test count, and this diff takes
test_fargate_launch_engine.py from 48 collected cases to 60, which moves every
boundary. Attribute a shard red by the test it names, never by the shard number.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@bolichen97
bolichen97 merged commit a311f8c into main Sep 18, 2026
124 of 125 checks passed
@bolichen97
bolichen97 deleted the fix/fargate-tier-size-keys-11328 branch September 18, 2026 04:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants