Repository navigation
feat(decisions): add the wake judge's LLM lane and per-point provider selection - #12776
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The card reports "Judged by the small model" as active at this HEAD, but nothing can judge — the point's ask site lives in an unlanded sibling PR. WatchEverything that makes this PR's claims true ships in Suggestions
[DESIGN-REVIEWED] 1f413b4 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The lane ships consumer-less: nothing at this HEAD asks Not justified as shipped
What this change shipsInventory (8 items) — 2 justifiedIntent: let the wake judge answer on machines without a Jev key by adding an LLM lane to the decisions seam — an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 1f413b4 |
UX Review (Fable 5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS Judge-row status contradicts the card's frame: active while the switch is off, "Off" while Jev is picked — the blind reader couldn't tell what's running. Watch
Suggestions
[UX-REVIEWED] 1f413b4 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsFINDING. A legitimate bracketed ( FINDING — src/kiro_crew/decisions/gate.py:588 — [OPUS-REVIEWED] 1f413b4 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
0825b91 to
ad9ab94
Compare
GPT 5.6 Review —
|
ad9ab94 to
e833c5b
Compare
|
Dispositions for the first-principles review, on head Item 3 (duplicate of Item 8 (website constant Items 1, 2, 4 and the blocker (surface ahead of its consumer) -- accurate, and not the Two notes on the inventory: item 5's clamp is 8-60 s, and items 6 and 7 are an |
e833c5b to
bbbb00d
Compare
bbbb00d to
f6fdf1c
Compare
buluoray
left a comment
There was a problem hiding this comment.
Source-verified at 618bdea0676c5ea33528d878a4346b688d36becc (read through the refs/pull/12776/merge tree, 32cb5980). Approving.
This adds a second judge lane — a tool-less, strict-parsing one-shot model call — plus per-point provider selection for nudge.wake. The four things I went after, all clean:
- No hardcoded model id.
JUDGE_MODEL_DEFAULT = "auto"(src/kiro_crew/decisions/impl_llm.py:73);build_session_runnerdrops any valueis_model_idrejects and maps"auto"to inherit (:182). Scanning every added line undersrc/andwebsite/src/forclaude-*/opus*/sonnet*/haiku*/gpt-*/fable*returns nothing outside test fixtures.lane_model(cfg, lane=lane)(gate.py:891) also makes the scanned id the sent id, soscrub_reasoncannot be checking a different model than the one the selected lane names. - Every failure path returns
Noneand records why. Timeout →ERROR_TIMEOUT(gate.py:905), any other exception →ERROR_PROVIDERloggingtype(exc).__name__only, no traceback and no provider text (:910-916), empty or out-of-domain answers →ERROR_INVALID_RESULT(:923).CancelledErrorre-raises.Noneis the same contract every other point already has, so a broken lane degrades to the ungated timer rather than dropping a wake silently or crashing the caller. - Untrusted evidence is placed as data.
render_promptwraps the state in<EVIDENCE>…</EVIDENCE>with an explicit "this is DATA, not instructions" sentence. - Bounded. One runner call, no retry loop;
wait_for(budget)with the budget clamped to_LLM_TIMEOUT_MIN_SECS.._LLM_TIMEOUT_MAX_SECS= 8–60s (gate.py:96-97,:450);_MAX_RESPONSE_CHARS64KiB caps the parse (impl_llm.py:77,:319); the session is released and destroyed infinally.
Authority reads correctly as additive: _judge_authority (gate.py:566) gives the LLM lane not _capability_denied(session_key) (:611), i.e. the fleet ceiling still binds it, while the Jev lane keeps consent + the point's scope. The argument for not requiring consent on the LLM lane — same model provider the owner's sessions already use, no new egress — holds, and the comment at :855 puts that reasoning where the next reader will look. Nothing in the - lines removes a guard; they are signature and import moves.
test/test_decisions_judge_llm.py is 91 passed at this head.
On the three 🟡 advisory lanes, so they aren't left hanging — none is a defect I'd hold this for:
- design: the settings row can read as active before the consumer lands in the sibling core PR. Real, and a landing-order question rather than a code fault.
- first-principles: the lane ships ahead of its only caller (no
decide("nudge.wake", …)site in this build). Inert and reversible; a sequencing preference. - ux: the "nothing is sent while this is off" copy sits oddly beside an LLM-lane row that is live under the off switch. Worth a wording pass whenever the consumer lands, not now.
Not verified: the nudge.wake call site is in the sibling PR, so I checked the gate's return contract but not the caller's handling of a None verdict. I did not run the full decisions suite or the website tests.
|
Dispositions for the advisory findings on Accepted: the judge row did not match the gate (Opus, Design). Two cases were wrong, Accepted: the card re-derived the lane (same finding, frontend half). It cannot: Accepted: a present-but-invalid Accepted: the inherit label named the wrong model (GPT). An empty Accepted: the consent surface still said nothing is sent (UX). The card's frame speaks Verification: 212 python tests across the two affected files, 92 card tests, 703 i18n tests A second capture frame is attached with the provider menu open, because the option labels Still open and not reachable from this branch: no production code calls |
2264831 to
9b4ff7c
Compare
|
Dispositions for UX BLOCK, missing evidence: accepted, and the cause was mine. The attached frames were Two items from that review are disclosed in the body rather than fixed here, under a heading Design CONCERNS: both items were accurate and both are fixed. Section 14 still closed with PR Hygiene: fixed. The branch carried three commits where the gate allows one or two. It is Frontend shard 2: mine, fixed. Backend Tests (Windows) (7): inherited, and not curable from this branch. |
9b4ff7c to
292ca74
Compare
|
GPT's upheld finding on F1, feature-map-correctness: right on both clauses, and both were mine. The row said the F2, the consent ratchet: I am leaving it as the adjudication found it. First Principles is BLOCK on the same premise as before, that nothing calls |
|
Board state on Two reds remain and both are main's, not this branch's. Three advisory items from the reviews of this head are fixed and waiting on one push rather |
|
@chenmingwei23 main now carries #12943 (80532c1), which fixes the |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at 292ca74 with Windows compatibility as the primary lens (618bdea was reviewed in full earlier; this is the delta review). Backend delta since 618bdea: handlers/decisions.py now takes the judge row's lane from gate.judge_lane and computes jev_armed with the same fail-closed predicate the gate uses (an unregistered scope reads as no grant, not as none needed), and the LLM lane reads active only when a runner is registered; impl_llm.py rejects a present-but-invalid confidence instead of silently dropping it. Pure logic - no path, file, lock, permission, signal, subprocess or encoding code, nothing that can differ on Windows. Windows evidence: on the earlier head the new/changed decisions tests ran on native Windows (375 passed, 1 pre-existing POSIX skip not added by this PR); on this head every Windows shard is green except shard 7, whose single failure is test_crew_log_exc_info_sites - inherited from main via the merge ref (crew_store.py sites landed by #12303, not touched here) and fixed on main by #12943 (80532c1). Linux shard 7 and Coverage Gate are the same inherited red. All review lanes are green on this head. Approving; merge once a CI round against main >= 80532c1 clears shard 7 (a rebase/push will do it - GitHub pins a rerun to the old merge commit).
…and a card row The judge decision point `nudge.wake` had exactly one provider, `JevOracle`, so a machine without a Jev key could not use the feature at all. This adds a second lane that answers the same protocol from one text-only model call on the provider the session already uses, teaches the gate to choose between them per point, and gives the point its row and its two controls on the Decisions card. - `decisions/impl_llm.py`: `LlmOracle` over an injected runner factory, plus a parser as strict as `_from_wire`: exactly the question ids, choices inside each question's own option set, every number finite in 0..1, a complete distribution that adds to 1, a present-but-invalid `confidence` refused rather than dropped, and prose refused unread. Every raise carries a constant message, because the gate logs the exception class into the decision log and a model may quote the evidence back. - `gate.py`: lane selection for `nudge.wake` only, from `decisions.nudge_wake.provider`. The two lanes are authorized by different things because they send to different places. The Jev lane needs the keystone in full -- consent for the endpoint AND this point's `nudge_evidence` scope, read fail-closed through `_point_scope_granted` rather than `_scope_consented`, which answers yes for a point with no registered scope. The LLM lane needs neither: it adds no destination and no data class, so `provider = llm` plus a `judge` spec on the loop is its authorization. Only the fleet ceiling binds both. `auto` resolves against the Jev side ARMED, so an owner who consented to the endpoint but never granted this point gets the small model instead of a refusal. The provider budget is clamped up for the LLM lane, which cannot answer inside a budget sized for a 100 ms System One call, and both lanes' model ids go through the one existing `scrubbed:provider-model` refusal. - No agent template of its own: the lane runs on the shipped `kirocrew-lite` spec, because a judge template would differ from it only in a model the runner passes per call and a standing prompt the per-call prompt already states. - `decisions.nudge_wake.provider` and `.llm_model`, both writable through the config route's allowlist beside the `decisions.model_route.<tier>` pins, hot-applied per call, with an unknown provider name reading as `auto` so a typo is never a third lane and never stops a boot. On the card the row already appears, because the list projects one row per point in the gate's registry. Three things were missing and are here. - The row had no label: its name and its one line of what it decides sit in the card's two label maps, keyed by the server's id like every other point, with the new strings in every catalog. - The point's panel offered no control for its own two settings. It draws a provider picker (auto / Jev / the small model) and a model picker for the small-model lane, both shaped like `model.route`'s tier pickers and written through the same config route. Neither is gated on consent: the small-model lane needs none, and an owner with no Jev key has to be able to pick it while the switch is off. The panel also states which lane answers and where the evidence goes, because the card's frame speaks for the Jev endpoint and this lane is the exception to it. - The row's status is the gateway's and names the LANE that would answer, resolved through `gate.judge_lane` and carried to the card as `lane`, so the word on screen and the oracle that answers come from one read. A pinned `llm` runs on its own, a pinned `jev` only with the Jev side armed, `auto` on either; the sampled share binds all three and the fleet denial refuses all three. Three capture frames of the judge panel: the small-model lane with consent OFF, the provider menu open where each option's destination is legible before it is picked, and the provider pinned to Jev with the switch off, which is the refuse path. Two existing card expectations move because the projection lists a seventh point: the row-list assertion gains it and the End-key case lands on it. Both are the deliberate edit that fixture exists to force. The `t` catalog chunk's bundle ceiling moves 905 -> 951 KB on the file's own 5% convention, measured against a base build rather than assumed.
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed at 1f413b4 with Windows compatibility as the primary lens. Full review was done at 618bdea and the backend delta to 292ca74 was reviewed and approved earlier (gate.judge_lane-driven row, fail-closed jev_armed, runner-gated LLM row, strict confidence parsing). 1f413b4 changes no src/ file - only docs/feature-map, decisions.md, the capture script and decisionsPreview.test.ts - so nothing Windows-sensitive is added. This head's CI round ran against main >= 80532c1 (the #12943 exc_info fix) and is fully green: 85 success / 7 skipped including Windows backend shards 1-8, Windows IPv6, the Windows E2E boot and pod canary, and readiness: passed. Local Windows evidence from the earlier round: 375 decisions tests passed, 1 pre-existing POSIX skip not added by this PR. Approving and squash-merging.
|
Correction to my approval note above: 1f413b4 did carry one src/ change relative to 292ca74, in decisions/impl_llm.py -- the response ceiling is now enforced per chunk while streaming (an on_chunk bound raising LlmProtocolError past _MAX_RESPONSE_CHARS) instead of only after the whole reply is collected, and the uuid import moved to module level. Reviewed after the merge: pure logic, no filesystem/process/encoding surface, tightens the memory bound, and the head CI round (incl. all Windows shards) covered it. The rest of the 2-dot diff was main movement from the rebase. |
|
Green on Three advisory verdicts remain, with dispositions. First Principles and Design, both CONCERNS on the same point: nothing asks UX, CONCERNS: four copy items, deliberately carried. The row chip truncates its own row title, |
1. What is the problem?
The Jev decision seam has exactly one provider. Every decision point routes through
JevOracle, which calls a paid third-party endpoint, so a decision can only be askedon a machine whose owner holds a Jev key and consented to that endpoint. PR #11490
shipped the seam that way on purpose and said so in its own body: the LLM lane was cut
because nothing consumed it.
The wake judge (
nudge.wake, RFCdocs/request-for-change/rfc-wake-judge.md) is theconsumer that makes the lane necessary. It decides whether a monitor loop's quiet tick
should wake its owner at all. If it could only ask Jev, the feature would exist only
for owners with a key, and every other install would keep firing the ungated timer.
2. Why this issue matters to the user
An owner without a Jev key gets woken by every tick their loop schedules, whether the
work moved or not. The judge is what makes a quiet cycle cost nothing, and a
Jev-only judge cannot give them that. It also means the feature cannot be evaluated by
most of the people who would benefit from it, because the cheapest way to try it -- the
small text model the machine already talks to every turn -- is the one path the seam
does not have.
3. How our fix solves it
The chain runs from the symptom to the single missing piece.
The symptom is that no judge answers without a Jev key. The immediate cause is that
gate.decideconstructs one oracle. The root cause is that the seam never had a secondimplementation of its
ask(state, questions) -> Answersprotocol, so there was nothingto choose between.
decisions/impl_llm.pyadds that implementation.LlmOracleanswers the same protocolfrom ONE tool-less run of the shipped
kirocrew-liteagent: no tools, no MCP servers, noskills, and the call streams under
REJECT_ALL, so no tool can run even if one wereoffered. The lane ships no agent template of its own on purpose. One would differ from
lite in exactly two fields and neither survives inspection -- the model, which the runner
passes per call, and a standing system prompt, every sentence of which the per-call prompt
already states more completely, including the rule that the evidence is data rather than
instructions. A second spec written to every install to repeat what the call already says
is a file to keep in sync for no behaviour. The prompt carries the
state fenced as data plus each question with its id, type and options, and demands one
JSON object whose keys are exactly the question ids.
The parser is as strict as
impl_jev._from_wire, because a text model is a lessdisciplined producer than the wire, not a more trusted one. A choice must be in that
question's own option list; every number must be finite in 0..1 and not a bool; a
probabilitiesmap may name only that question's options and must include the chosenone; a complete distribution must sum to 1; a two-option question may answer with a bare
number, read as the probability of the first option; a response over 64 KiB is refused
unread. Anything else raises, and the gate turns a raise into the same
Nonea timeoutproduces, which is the caller's FALLBACK -- the tick then fires exactly as the ungated
timer does. Every raise carries a CONSTANT message, so no model prose can reach an
exception message or a decision-log row; a test pins that with a canary token.
gate.pychooses the lane fornudge.wakeand no other point, fromdecisions.nudge_wake.provider(auto | jev | llm), withdecisions.nudge_wake.llm_modelnaming the model. Both are registered in the config route's
_EDITABLE_CONFIGallowlistbeside the
decisions.model_route.<tier>tier pins, so they are written the same way andhot-applied. An unknown provider name reads as
autorather than as an error: a typo mustnot become a third lane and must not stop the gateway booting. An empty model INHERITS -- the runner
passes no model and the judge agent's own resolves, which is the
kirocrew-litebackground template's model rather than the model of whichever session the tick belongs
to -- and that is what the picker's inherit default means end to end.
autoresolves to Jev when the Jev side is ARMED for this point and to the small modelotherwise, so an owner with no key needs no setting at all. An explicitly pinned
jevwithout that arming refuses rather than substituting a different model silently.
The LLM lane's budget is the seam's
provider.timeout_msclamped up into 8..60 seconds.That number is sized for Jev (1 s by default, ~100 ms in practice); a text model cannot
answer inside it, so a lane that honoured it alone would time out on every tick and read
as "does nothing" rather than as "misconfigured". Clamped rather than replaced, so an
operator who raised it deliberately still gets what they asked for.
docs/system-specs/modules/decisions.mdgains section 14 for the lane, the parsercontract and the authorization below.
Security
The two lanes are authorized by different things, because they send to different places.
The Jev lane is unchanged: it needs the keystone in full, consent for the configured
endpoint AND this point's own
nudge_evidencescope. That is what a scope means here, acategory of egress to the Jev endpoint. The judge reads the scope fail-closed through a
new
_point_scope_grantedrather than the existing_scope_consented, which answersTruefor a point with no registered scope -- correct forskills.select, which sendsnothing beyond what the main switch records, and wrong here, where the absence of a
registration means nothing has authorized Jev to receive worker transcripts.
The LLM lane needs no consent row, and
provider = llmplus ajudgespec on theloop is its whole authorization:
already send to on every turn. No new endpoint, no new credential, no new egress path.
review comments and ledger events -- creator-only material that the same provider
already received when those sessions ran.
Every failure (refusal, timeout, prose, invalid answer, missing runner) returns
Noneand the tick fires as it does today, so the worst case is one delayed wake, bounded by
the quiet-streak floor.
would have run.
What still binds both lanes is the FLEET ceiling. A managed install that pinned
capabilities.decisionsoff withdrew the seam, not one provider's endpoint, so the LLMlane pays that governed capability probe itself rather than inheriting it from the
keystone read.
4. What tests we did
test/test_decisions_judge_llm.py(new): the parser strictness table, one row per way amodel can be wrong (valid forms, extra key, missing id, prose,
NaN, out-of-range,bool-as-number, a probability map naming a foreign option, an over-ceiling response); a
canary token asserting no model text reaches an exception message; the prompt's fencing
and question rendering; the runner seam including a model id that is not one; the lane
selection matrix over provider x consent; the authorization matrix (LLM with no consent
and no scope, an unregistered scope on each lane, Jev needing both halves,
autofalling back, the fleet ceiling); the budget clamp;
_EDITABLE_CONFIGregistration andthe endpoint and key staying uneditable; and an end-to-end
decideper failure path.test/test_decisions_gate.py: unchanged except one line, and re-run whole.test/test_settingref_schema_fixture.py: the settings-reference fixture for the newconfig section.
main: 194 passed, three files,-n0.isort,blackandflake8clean on all ten files. CI carries the breadth.5. Any other suggestions on the work
nudge.wakedecision point, its collectors, the autonudge hook and its registration in
gate.POINTSwith thenudge_evidencescope -- is a separate PR(
feat/wake-judge-core). The RFC lands separately too. This PR is the second oracle plusthe per-point provider keys, and it is inert until the point is registered.
"nudge.wake"in the pinned pointvocabulary in
test/test_decisions_gate.py(whose own docstring says adding a point is adeliberate edit there) and one constant in
website/src/pages/settings/decisionsPreview.ts(a cross-layer guard requires one perbackend point). Both are also writable by the core PR or by the Decisions card
redesign; whoever lands second drops the duplicate.
projects one row per point from the gate's own registry, so
nudge.wakeneeded no newcard: it needed a label, controls in its panel, and a status the gateway computes from
the provider as well as the keystone. All three are here. The scope registration itself
is not:
nudge_evidencebelongs to the core PR, so until that lands the Jev lanerefuses and only the small-model lane can answer.
test_decisions_config.py::TestDefaults::test_no_field_here_is_a_switchpinned thesentence "every field is a bound or an address -- nothing here grants egress". That
stops being literally true once
decisions.nudge_wake.provider = llmis what runs theLLM lane, so the pin now reads "nothing here opens a destination the session does not
already send to", which is the honest form of the same guarantee and is exactly what the
Security section above argues. The field-name sets in that test and in the round-trip
test also gain
nudge_wake, which is what those pins exist to make deliberate. Namedhere because weakening a pinned invariant should be announced rather than noticed.
wants a cheap judge should get its own key, so no point inherits another's provider.
Screenshots
All three frames come from
node website/scripts/capture-decisions-preview-card.mjson thishead, so every string in them is a string that ships.
The panel with the card's consent switch OFF and the provider on the small model, which is
the state this lane exists for. The row does not borrow the generic active word there: under
a list headed "What Jev decides while this is on" that word would contradict the switch, so
the row names the lane that would answer. Both pickers are reachable with the switch off,
because the small-model lane needs no key and no agreement.
The provider menu open, which is the frame that matters for the choice itself: each option
says where the evidence goes before it is picked, and the collapsed control keeps that
marking rather than truncating it.
The refuse path, reached by pinning Jev while its switch is off. The gate refuses that lane,
so the row reads the plain Off word rather than naming one and the small-model note is
absent, because on this provider the switch above does govern the answer.
The harness asserts these frames rather than only writing them: the provider must read as
the chosen lane, the model must read as inherit, the row must carry the lane words on the
small model and none on a refused Jev, and the panel must be fully inside the frame.
Open review items, carried deliberately
Two advisory findings are disclosed rather than fixed here, because both need user-visible
copy in thirteen catalogs and neither changes what the gate does.
"Jev is a small, fast decision model": both lanes are small models, so the phrase does not
separate them. The egress clause on each option does, which is the safety-relevant half.
A rename belongs in one careful pass over every catalog, not appended to this diff.
check-in it would have judged is spent ungated. Saying so needs one more string on the
point panel.