Skip to content

fix(cloud): name the one Protocol keyword the Fargate engine may omit - #11432

Closed
iamwhatever wants to merge 1 commit into
mainfrom
fix/fargate-signin-conformance
Closed

iamwhatever wants to merge 1 commit into
mainfrom
fix/fargate-signin-conformance

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backend Tests (Windows) is red on main, so every PR that merges current main
inherits it. Two conformance tests in test/test_fargate_launch_engine.py fail:

begin_signin: ['instance_id', 'profile', 'region'] != ['instance_id', 'profile', 'region', 'login_target']

Because the shard matrix is fail-fast, that one failure also cancels a sibling
shard, so a contributor sees two red Backend Tests checks and a failed
PR Readiness for something their branch never touched.

Why it matters

Nobody can tell their own breakage from this one. A red that belongs to the base
teaches contributors to ignore red, which is how a real failure ships.

What changed (motivation → approach → change)

The one keyword the Fargate engine is allowed to leave out is now named, with
the reason, instead of the whole comparison being red about it.

Case Before After
Engine omits begin_signin.login_target 🟥 fail 🟩 pass — the named, documented omission
Engine omits any OTHER Protocol parameter 🟥 fail 🟥 fail
Engine misses a method, renames, reorders, adds a parameter 🟥 fail 🟥 fail
SigninHandle gains a member the handle lacks 🟥 fail 🟥 fail
FargateSigninHandle carries error 🟥 absent 🟩 present, always ""

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

LaunchEngine.begin_signin gained an optional login_target, and SigninHandle
gained error. FargateLaunchEngine implements neither, and launch_job is built
on exactly that: _begin_signin_with_target inspects the signature and calls the
three-argument shape when the keyword is absent, and error is documented as read
through getattr "so older handles still conform". The test compared signatures
for equality and pinned the handle's attribute set with an equality, so a
supported implementation read as a broken one.

The omission is load-bearing, which is why this fixes the test and not the engine.
_check_signin_target_supported decides whether an engine can honour a requested
identity by asking whether begin_signin accepts login_target, and refuses at
preflight when it cannot — before anything is provisioned or billed. A Fargate
container is handed its credential at run time and performs no interactive
sign-in, so making it accept the keyword to satisfy the test would replace that
refusal with a launch that signs the crew in as the wrong identity: the defect the
keyword was added to close. Refusing inside begin_signin instead would move the
failure after provisioning and strand a billing task.

So a module-level SUPPORTED_OMISSIONS names the single (method, parameter)
pair, carrying that reasoning, and the signature comparison accepts the Protocol's
parameter list either whole or minus that one pair. The handle's attribute set
moves to the Protocol's current five members. In
src/kiro_crew/cloud/fargate_engine.py, FargateSigninHandle gains error = "",
which is the truth for a handle that performs no sign-in.

On the ratchet rule

a-ratchet-may-only-tighten is blocking and matches test/**/*.py, so this is
stated up front rather than left to be found.

Neither assertion is loosened. Both stay exact equalities. The signature check
gains one named exception whose reason is in the code beside it, and the handle's
member set moves to its new true value with its docstring — the rule's own
"Allowed" case. Nothing becomes >=, <=, in or issubset, no skip or
xfail appears, and no timing budget moves. A future optional keyword still reds
this test until someone decides, for that keyword, what was decided for this one.

The five mutants below are the evidence, including one that adds a second
optional keyword to the Protocol and confirms the carve-out does not stretch to
cover it.

Tests

The two failing tests are the tests. Verified in this order:

  1. Reproduced both failures locally on the unmodified test file, matching CI's
    assertion text exactly.
  2. Both pass after the change; test_fargate_launch_engine.py and
    test_cloud_launch_job.py are green together (92 tests).
  3. Five mutants, each of which must fail and does: a renamed parameter, a missing
    required parameter, a reordered parameter list, a handle missing a Protocol
    member, and a second optional Protocol keyword the engine omits.

Manual verification

N/A — unit coverage sufficient: the change is entirely within the assertions of
two tests plus one always-empty attribute, and the mutants exercise both the
carve-out and its boundary.

Pattern harvest

Rule candidate: review-prompt
Pattern: a conformance test that pins another module's Protocol with an exact
equality, when that Protocol documents a member or keyword as optional. The
equality then fails on a supported implementation, and the comfortable fix is to
make the implementation accept something it cannot honour.

Related Issues

Fixes #11417

The LaunchEngine Protocol gained an optional `login_target` keyword on
`begin_signin` and an `error` member on SigninHandle. The Fargate engine
implements neither, which `launch_job` explicitly supports:
`_begin_signin_with_target` inspects the signature and calls the three-argument
shape when the keyword is absent, and `error` is documented as read through
`getattr` so older handles still conform. The conformance test compared
signatures for exact equality and pinned the handle's attribute set with an
equality, so both assertions failed on a supported implementation and every
branch merging main inherited the red.

The omission is load-bearing. `_check_signin_target_supported` decides whether
an engine can honour a requested identity by asking whether `begin_signin`
accepts `login_target`, and refuses at preflight when it cannot, before anything
is provisioned or billed. A Fargate container is handed its credential at run
time and performs no interactive sign-in, so accepting the keyword to satisfy the
test would turn that refusal into a launch that signs the crew in as the wrong
identity.

Both assertions stay exact. A module-level SUPPORTED_OMISSIONS names the single
`(method, parameter)` pair this engine may leave off, with the reason, and the
signature comparison accepts the Protocol's parameter list either whole or minus
that one pair; anything else still fails. The handle's attribute set moves to the
Protocol's current five members rather than being loosened, so a member added
upstream still reds the test until the handle carries it. FargateSigninHandle
gains `error = ""`, which is the truth for a handle that performs no sign-in.
@iamwhatever
iamwhatever requested review from a team and bolichen97 September 17, 2026 02:21
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Intent: Get Backend Tests green on main again by naming the one
LaunchEngine keyword the Fargate engine is allowed to omit, so a conformance
ratchet stops firing on a shape the runtime deliberately supports — without
giving the engine a keyword it cannot honour, which would turn a preflight
refusal into a wrong-identity launch.

Not a goal: changing the Protocol, changing what any engine actually does,
teaching the Fargate engine to sign in, or relaxing what the conformance tests
catch.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 33a6d0f810458f73ca37ce42b606557300ef0867. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- test/test_fargate_launch_engine.py:115 -- Protocol conformance ratchet is replaced by an allowlist
allowed = [p for p in want_params if (name, p) not in SUPPORTED_OMISSIONS]
Fargate omits login_target -> allowlist subtraction -> nonconforming signature passes.
Anchor: a-ratchet-may-only-tighten
Fix: Restore exact signature equality and represent unsupported identity targets outside the Protocol signature.
[BLOCK-MERGE] 33a6d0f
[GPT-REVIEWED] 33a6d0f

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

F1 is fenced (annotate-only). I opened the test and the runtime it guards.

The allowlist is pinned to exactly one (method, param) pair — ("begin_signin", "login_target") (test/test_fargate_launch_engine.py:92). The assertion got_params in (want_params, allowed) (test/test_fargate_launch_engine.py:116) still reds on any other deviation: a NEW optional keyword the engine omits matches neither want_params nor allowed (which subtracts only login_target), so it fails — the ratchet does not silently absorb future omissions.

The single permitted omission is independently guarded at runtime, so it cannot produce the "wrong identity" harm the keyword exists to close: _engine_accepts_login_target detects the absence via inspect.signature (src/kiro_crew/cloud/launch_job.py:540-542), and _check_signin_target_supported refuses a non-default identity at preflight, before provisioning (src/kiro_crew/cloud/launch_job.py:555-565, called at :698).

The condition that would turn this loosening into unbounded harm — a new security-relevant keyword slipping through undetected — is one the test's own construction prevents (it reds), and the one shape it does admit is safe by an independent runtime refusal. Complete record, extreme/self-contradicting harm condition → FLAG.

[ADJUDICATION] 33a6d0f total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 33a6d0f
[ADJUDICATION-FENCED] 33a6d0f fenced=1 flagged=1
FLAG F1 test/test_fargate_launch_engine.py:115 -- The allowlist admits exactly one documented (begin_signin, login_target) omission and reds on any other deviation, and that lone omission is independently guarded at runtime by a preflight refusal (launch_job.py:540-565,698), so no nonconforming signature can reach the wrong-identity harm.
[GPT-ADJUDICATED-FENCED] 33a6d0f

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.

  • F1 test/test_fargate_launch_engine.py:115 — The allowlist admits exactly one documented (begin_signin, login_target) omission and reds on any other deviation, and that lone omission is independently guarded at runtime by a preflight refusal (launch_job.py:540-565,698), so no nonconforming signature can reach the wrong-identity harm.

    /ai-review override gpt 33a6d0f810458f73ca37ce42b606557300ef0867: The allowlist admits exactly one documented (begin_signin, login_target) omission and reds on any other deviation, and that lone omission is independently guarded at runtime by a preflight refusal (launch_job.py:540-565,698), so no nonconforming signature can reach the wrong-identity harm.
    

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 33a6d0f810458f73ca37ce42b606557300ef0867: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 33a6d0f810458f73ca37ce42b606557300ef0867 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 33a6d0f

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 33a6d0f810458f73ca37ce42b606557300ef0867 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 33a6d0f

Verdict parsed from the review's SHA-scoped output markers for commit 33a6d0f810458f73ca37ce42b606557300ef0867.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 33a6d0f810458f73ca37ce42b606557300ef0867: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 33a6d0f810458f73ca37ce42b606557300ef0867 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify mutant #5 once — a second optional Protocol keyword must still red the test — since the carve-out's boundary rests on the author's local run.

Both base failures are mechanically real in the checkout: the Protocol (launch_job.py:500-507) takes optional login_target the Fargate engine (fargate_engine.py:649) omits, and SigninHandle declares five members while the base handle carried four — so the two exact-equality pins were red on main, and this unwedges it. The omission is load-bearing as claimed: _check_signin_target_supported (launch_job.py:555) turns the missing keyword into a preflight refusal, itself pinned at test_cloud_launch_job.py:760-780, and launch_job.py:722 already reads error via getattr. Grepped inspect.signature under test/ and LaunchEngine|SigninHandle (4 files): no sibling conformance pin left red. Both rewritten pins stay exact equalities with the reversal evidenced by the runtime's own contract, and the description declares it.

What this change ships

Inventory (4 items) — 4 justified

Intent: FIX — turn main green by teaching two Fargate conformance tests the omission the runtime contract deliberately supports.

  1. Conformance test passes when the engine omits only begin_signin.login_target; every other difference still fails — justified
  2. Test-file constant SUPPORTED_OMISSIONS names that one pair with its reason — justified
  3. Handle-member pin moves from four members to five, adding error — justified
  4. FargateSigninHandle gains error = "", satisfying the Protocol; no runtime behavior change — justified

[FIRST-PRINCIPLES-REVIEWED] 33a6d0f

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: no
mechanism: SUPPORTED_OMISSIONS, a one-pair constant naming the single tolerated (method, parameter).

  • Protocol conformance ratchet is replaced by an allowlist span=8519ad5ec867

Rebutted as not a defect. allowed subtracts exactly ONE hardcoded pair, so a second omitted keyword matches neither want_params nor allowed and the assertion reds.
Proven by mutation rather than by reading: five mutants, five reds — a second optional Protocol keyword the engine omits, a renamed parameter, a dropped required parameter, a reordered list, and a handle missing a Protocol member.
The requested remedy is the unsafe one. Exact equality forces the engine to DECLARE login_target.
_engine_accepts_login_target (launch_job.py:540) reads that very signature to decide whether an engine can honour a requested identity.
A Fargate container is handed its credential at run time and never signs in interactively, so it cannot honour a non-default target no matter what its signature says.
Declaring the keyword therefore makes _check_signin_target_supported (launch_job.py:555, called at :698) stop refusing at preflight, and the launch proceeds as the WRONG identity — the defect that keyword was added to close.
Refusing inside begin_signin instead moves the failure after provisioning and strands a billed task.
Representing the target outside the signature changes another module's compatibility contract, which is a maintainer decision rather than part of this fix.
This lane's own adjudication pass reached the same conclusion and FLAGGED the finding; the security fence keeps it blocking, so only a maintainer can clear it.
Maintainer override reason: exact signature equality would force the engine to declare a keyword it cannot honour, turning the preflight identity refusal at launch_job.py:555 into a wrong-identity launch, while the one-pair allowlist reds on every other deviation, proven by five mutants.

The mutation evidence behind line 2, each run against this head's assertions:

Mutation Result
A second optional keyword added to the Protocol, omitted by the engine fails
Engine parameter renamed fails
Required parameter dropped fails
Parameter list reordered fails
Handle missing a Protocol member fails

I am not posting the override — a repository writer must decide that. The reason
above is offered for that purpose and is stated in my own words after checking
each cited line.

@chenmingwei23

Copy link
Copy Markdown
Contributor

Measurement: main currently has TWO red Fargate tests, and this PR is the only one
of the four rivals that clears both.

Observed on an unrelated PR's Windows shard (run 35175176922, a ledger-only change,
so neither failure is that PR's):

test/test_fargate_launch_engine.py:96
  test_engine_satisfies_the_launch_engine_protocol
  AssertionError: begin_signin: ['instance_id', 'profile', 'region']
                             != ['instance_id', 'profile', 'region', 'login_target']

test/test_fargate_launch_engine.py:141
  test_signin_handle_has_every_signin_handle_protocol_member
  AssertionError: ... Right contains one more item: 'error'

So #11230 landed an engine whose begin_signin omits a keyword the Protocol
declares, and a handle missing the error member. Both are red on main today.

This PR closes the first by NAMING the omission in SUPPORTED_OMISSIONS rather than
by widening the signature, and the second by declaring self.error = "".

Why the naming route matters, measured on origin/main: launch_job.py:698
_check_signin_target_supported runs BEFORE :699 engine.preflight and
:706 engine.provision, and _engine_accepts_login_target (:540) decides by
inspect.signature. While the keyword is absent, a non-default target is refused at
PREFLIGHT -- no task started, nothing billed. Adding the keyword makes that guard
pass, so the refusal moves to the sign-in step with a task already running; #11442's
own docstring says as much ("by the time sign-in runs the task is already billing")
and #11426 has to add a stop-the-task path to compensate.

A Fargate container is handed its credential through task-definition secrets and runs
no interactive sign-in, so no argument to begin_signin can change the identity it
serves under. Keeping the omission and naming it preserves a free refusal; accepting
the keyword buys protocol symmetry and pays for it after provisioning.

Note the other three cannot merge on their current heads either: every open PR is
red on the separate ledger break from #11317 (#11439, #11443).

@chenmingwei23

Copy link
Copy Markdown
Contributor

The twelve reds on this board are no longer real. #11445 merged at 05:43:57Z as
cdd4f132d, which is now main's tip, and it fixes the ledger break every one of them
came from:

test/test_ledger_kinds.py      actor  0 -> 1 occurrence on main
test/test_ledger_types.py      actor 8 / agent 10
test/test_ledger_retention.py  actor 13 / agent 20

This PR's base is still 04727ebe0, from before that merge, so its board is measuring a
main that no longer exists. A rebase onto cdd4f132d (or just a re-run after one) should
clear all twelve without touching the diff.

What remains red on main is only break C, which is what this PR fixes:

test/test_fargate_launch_engine.py:96  test_engine_satisfies_the_launch_engine_protocol
test/test_fargate_launch_engine.py:141 test_signin_handle_has_every_signin_handle_protocol_member

For the record, #11442 is now APPROVED and already rebased onto cdd4f132d, and it
answers the pre-billing objection I raised earlier by consulting an optional
login_target_refusal hook at preflight (I corrected myself on that at #11442
comment 5708994978). So the choice between the two is contract surface, not billing:
this PR is +37/-9 and adds no engine method, that one is ~+251 and adds one optional
method the LaunchEngine contract then carries.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Closing: superseded by #11442, which merged and fixes the same red.

#11442 took the other direction — the engine now accepts login_target — and it
closed the hazard this PR was avoiding rather than walking into it: a
login_target_refusal fails a non-default identity at preflight before anything is
billed, and begin_signin raises fail-closed if one reaches it anyway. That is a
better answer than what this PR proposed, because the refusal no longer depends on
a signature omission at all. It also carries the handle's error member and the
repinned attribute set, so nothing here is left to land.

The conformance test on main is now an exact comparison with no carve-out, which
is the stronger shape. SUPPORTED_OMISSIONS is not needed and should not be
revived.

One note for the record, since the discussion on this PR turned on it: the reason
the engine can safely declare the keyword now is that #11442 added an independent
refusal. Declaring it WITHOUT that refusal would have made
_check_signin_target_supported stop refusing and let a launch proceed as the
wrong identity. Anyone revisiting this area should keep those two changes together.

No code from this branch is being carried forward.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 17, 2026
@bolichen97
bolichen97 deleted the fix/fargate-signin-conformance branch September 20, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backend Tests (Windows) red on main: FargateLaunchEngine.begin_signin lost Protocol conformance

2 participants