Repository navigation
fix(llm): fallback chain walk honors namespace-qualified model ids - #8896
Conversation
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The design analysis holds end-to-end. The wire-spelling split is genuinely required by Design-Verdict: PASS Root-cause fold at the last literal-membership site, with the chain/wire spelling split correctly derived from real consumer contracts. [DESIGN-REVIEWED] 140a34d |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All verification is done. The description's mechanism claims each check out against the base tree, the return value's spelling change is safe (all three callers use it only as a None-check/notice), and my sibling grep ( First-Principles-Verdict: PASS
What this change shipsInventory (5 items) — 5 justifiedIntent: a pinned fallback carrying a stale
Provenance verified against base: the literal skip exists ( [FIRST-PRINCIPLES-REVIEWED] 140a34d |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
|
Re: Opus 4.8 suggestion — import Commit 1 (0a830e7) had exactly that shape and failed Per Happy to reroute if maintainers prefer a different shape, but as the gate stands, the suggested import is the one it rejects. |
|
Backend Tests (3.12, 4) failure — the #8893 flake, third occurrence, not this diff
Root cause is confirmed at source (details in #8893): both ordering tests patch No changes to this PR; requesting a rerun of the failed shard once maintainers deem it appropriate. |
5c645c8 to
35f0e8d
Compare
|
CI note: the |
35f0e8d to
7582844
Compare
|
The
This is the 4th occurrence of the #8893 signature across recent CI runs on unrelated PRs. No code change is needed here; happy to have the shard re-run whenever convenient. |
|
🤖 Kiro Crew [operator: iamwhatever#241cc860]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: the only failing shard is the #8893 flake, since fixed on main; plan is a rebase onto current main, the First-Principles rename of |
…irodotdev#8740) The throttle-fallback walk filtered persisted agent.fallback_model chain entries by literal membership in the advertised set, so a chain entry carrying a stale <namespace>::<bare-id> qualifier was skipped as unadvertised even when the backend serves the bare id, and the walk fell through toward "auto"/chain exhaustion. On a partition that does not serve "auto" the chain exhausts and the original throttle error resurfaces, so the configured fallback never fires. next_fallback_candidate now judges membership and the active-model skip through resolve_pin_spelling (the shared fold from kirodotdev#8616/kirodotdev#8737); an entry absent under BOTH spellings is still skipped, and a verbatim-advertised qualified id is never peeled. The walk still returns the CHAIN's own spelling because FallbackState.next_candidate locates the applied candidate with remaining.index. advance_fallback_candidate re-folds that entry once through _fallback_wire_spelling and uses the ADVERTISED spelling for everything later compared against a served model: the substitute set_model call, the silent no-op witness, fb_state.active/walked, the TURN_FALLBACK_ATTR marker probe_fallback_restore reads, the swap log, and the return value. A qualified spelling in the marker would make the restore probe read the session as having moved off the fallback, clear the sticky state, and skip the slot heal. resolve_pin_spelling is reached through kiro_crew.agent_sdk.drivers.acp, the delegation that exists for the agent-sdk-boundary gate; a direct kiro_crew.acp.client import both adds an ACP edge and rewrites a baselined import line, which the gate refuses. Six behavior pins added; docs/system-specs/modules/model-fallback.md candidate-walk section synced in the same commit. Fixes kirodotdev#8740 Co-authored-by: Joe Guo <zejiangg@amazon.com>
7582844 to
140a34d
Compare
|
🤖 Kiro Crew [operator: iamwhatever#241cc860]: confirming the First Principles PASS headline on
There is also no spelling that regresses for a user. Before this change a qualified chain entry never reached The alternative — keep the config spelling in No code change; recording the confirmation rather than a fix. Local gates are green on this head (mypy 1470 files, isort, flake8, black baseline, comment-history, agent-sdk-boundary, docs-lint, |
|
🤖 Kiro Crew [operator: iamwhatever#241cc860]: this is green and ready for maintainer review — What I changed on @javenciu's behalf, as a co-author — the approach, the design and the tests are unchanged from what was submitted:
Verdicts on the current head: Design Review PASS, First Principles Review PASS, GPT 5.6 no findings, Opus 4.8 no findings, Security Scope Review pass, Internal Content Scan pass. The First Principles PASS headline asked whether the Two CI reds that were not this diff, both resolved without touching the PR's code:
Leaving the |
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: a persisted fallback-chain entry carrying a stale :: qualifier was compared literally against the session's advertised ids, so it was skipped as unadvertised; the fix judges membership through the shared resolve_pin_spelling fold and records/sends the advertised spelling so the restore probe stays in sync. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/model-fallback.md. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
iamwhatever
left a comment
There was a problem hiding this comment.
Read the full diff. The chain/wire spelling split is correct: next_fallback_candidate still returns the CHAIN spelling because FallbackState.next_candidate locates it with remaining.index, while _fallback_wire_spelling folds to the ADVERTISED spelling for the set_model call, the no-op witness, fb_state.active/walked and the TURN_FALLBACK_ATTR marker the restore probe compares against. resolve_pin_spelling tries the full id first and peels only one leading namespace qualifier, so deny-parity holds (absent under both spellings stays skipped) and the post-fold active skip covers a qualified entry that resolves to the failing model. Tests pin all four cases.
…irodotdev#8740) (kirodotdev#8896) The throttle-fallback walk filtered persisted agent.fallback_model chain entries by literal membership in the advertised set, so a chain entry carrying a stale <namespace>::<bare-id> qualifier was skipped as unadvertised even when the backend serves the bare id, and the walk fell through toward "auto"/chain exhaustion. On a partition that does not serve "auto" the chain exhausts and the original throttle error resurfaces, so the configured fallback never fires. next_fallback_candidate now judges membership and the active-model skip through resolve_pin_spelling (the shared fold from kirodotdev#8616/kirodotdev#8737); an entry absent under BOTH spellings is still skipped, and a verbatim-advertised qualified id is never peeled. The walk still returns the CHAIN's own spelling because FallbackState.next_candidate locates the applied candidate with remaining.index. advance_fallback_candidate re-folds that entry once through _fallback_wire_spelling and uses the ADVERTISED spelling for everything later compared against a served model: the substitute set_model call, the silent no-op witness, fb_state.active/walked, the TURN_FALLBACK_ATTR marker probe_fallback_restore reads, the swap log, and the return value. A qualified spelling in the marker would make the restore probe read the session as having moved off the fallback, clear the sticky state, and skip the slot heal. resolve_pin_spelling is reached through kiro_crew.agent_sdk.drivers.acp, the delegation that exists for the agent-sdk-boundary gate; a direct kiro_crew.acp.client import both adds an ACP edge and rewrites a baselined import line, which the gate refuses. Six behavior pins added; docs/system-specs/modules/model-fallback.md candidate-walk section synced in the same commit. Fixes kirodotdev#8740 Co-authored-by: Joe Guo <zejiangg@amazon.com>
Problem / Motivation
A persisted
agent.fallback_modelchain entry that carries a<namespace>::<bare-id>qualifier (the #8521 mismatch class — stored when acatalog advertised the qualified spelling) is silently skipped by the
throttle-fallback walk when the session advertises the bare id.
next_fallback_candidate(src/kiro_crew/llm_helpers.py) filters the chainby literal membership:
so a chain entry the backend fully serves is treated as unadvertised, and the
walk falls through toward
"auto"/chain exhaustion. On a partition that doesnot serve
"auto", the chain exhausts and the original throttle errorsurfaces — the configured fallback never fires. Issue #8740 identified this as
the one literal-membership site left outside the #8616/#8737 fold.
Why it matters
Users who pinned a fallback while a catalog advertised qualified spellings get
no fallback at all at exactly the moment it exists for: sustained throttling
on the primary. The failure is silent (a debug-level skip log), so it reads as
"fallback feature does not work". The same stale-qualifier pins were already
judged worth folding at the substitute-send and picker sites (#8616, #8737);
the fallback walk was the remaining consumer of persisted model values still
comparing literally.
What changed (motivation → approach → change)
Observed symptom: a served-but-qualified chain entry is skipped as
unadvertised (reproducer below — the walk returns
Noneon a chain whose onlyentry the backend serves under its bare spelling).
Root cause:
next_fallback_candidatejudges membership with a literal setlookup, and the issue's own analysis explains why the fold could not be a
one-line predicate swap: the walk's return value feeds two consumers with
conflicting spelling needs.
FallbackState.next_candidatelocates the appliedcandidate with
remaining.index(cand), so it needs the CHAIN's own spelling —but the wire and every later served-model comparison (the
AcpClient.set_modelexplicit-pick guard, the silent no-op witness, theTURN_FALLBACK_ATTRmarker thatprobe_fallback_restorecompares against theserved model, and
fb_state.active/walked) need the ADVERTISED spelling. Aqualified spelling in the marker would make the restore probe read the session
as having moved off the fallback, clear the sticky state, and skip the slot
heal.
The change, keeping the two spellings from ever disagreeing:
next_fallback_candidatejudges membership and the active-model skipthrough
resolve_pin_spelling(the shared fold from Namespace-qualified model ids: sibling comparison sites not covered by #8615 #8616/fix(models): fold namespaced pins at substitute and picker sites (#8616) #8737): full idfirst, one leading
<namespace>::peel on a miss. An entry absent underboth spellings is still skipped (deny-parity), a verbatim-advertised
qualified id is never peeled, and a qualified entry that resolves to the
currently-failing model is skipped post-fold. The function still returns the
chain's own spelling, preserving the
remaining.indexbookkeeping contractand all ten existing behavior pins.
_fallback_wire_spellingis the single home of thechain→wire
translation:
advance_fallback_candidatecomputes it once per candidate anduses it for the in-loop active skip, the
set_modelcall, the no-opwitness,
fb_state.active/walked, the published marker, the swap log, andthe return value. Empty/unknown advertised set falls back to the entry's own
spelling, matching the walk's existing fail-open stance.
docs/system-specs/modules/model-fallback.mdcandidate-walk section syncedin the same commit.
Alternative rejected: returning a
(chain_entry, wire_spelling)tuple fromnext_fallback_candidate. It changes the function's contract and everyexisting pin for the same information the shared fold can recompute
deterministically at the one wire consumer; the smaller surface keeps the fold
authoritative in one place.
Tests
Reproducer, before the fix (tests-only tree at base
235d36a62):After the fix:
6 passed, and the full targeted file142 passed.New pins and what each locks in:
test_namespace_qualified_entry_folds_to_advertised_bare_id— a qualifiedentry the backend serves under its bare id is selected, and the CHAIN
spelling is returned (bookkeeping contract).
test_qualified_entry_peeling_to_active_model_is_skipped— post-fold activeskip: a qualified entry resolving to the failing model cannot help.
test_entry_absent_under_both_spellings_still_skipped— deny-parity: thefold does not weaken the advertised filter.
test_verbatim_advertised_qualified_id_not_peeled— a full-id match wins;peeling never rewrites it.
TestAdvanceFallbackCandidateNamespacedChain::test_qualified_entry_applies_under_advertised_spelling— end to end:
set_modelawaited with the bare spelling, marker(primary, bare),fb_state.active/walkedcarry the bare spelling, andposadvanced past the chain entry itself.TestAdvanceFallbackCandidateNamespacedChain::test_qualified_entry_peeling_to_failing_model_is_skipped— the walk exhausts rather than announcing a no-op swap when the chain names
the failing model under a qualifier.
Neighboring suites green locally:
test_subagent_turn_resilience.py+test_model_selection_scenarios.py+test_api_models_entitlement.py(84 passed),
test_dashboard_chat.py -k fallback(16 passed). mypy, flake8,isort, the black baseline gate, and
docs-lint.shall pass on the changedfiles.
Manual verification
N/A — unit coverage sufficient: the walk, the wire call, the witness, and the
marker are all exercised through the same provider-double shape the existing
fallback tests use, and the seam has no UI or external-service surface.
Related Issues
Fixes #8740
Pattern harvest
Rule candidate: review-prompt
Pattern: "persisted model ids must be compared through the shared spelling
fold, never by literal membership; any value later compared against a SERVED
model must carry the advertised spelling". #8616 → #8737 → this PR closed the
enumeration one site at a time; a review prompt on new
in advertised-stylemembership tests over persisted model values stops the class from re-growing.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Per the template placeholder (CLA text pending): offered under the same terms as my prior merged contributions to this repository (#8835).