Skip to content

feat(decisions): add the wake judge's LLM lane and per-point provider selection - #12776

Merged
bolichen97 merged 1 commit into
mainfrom
feat/wake-judge-llm-lane
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/wake-judge-llm-lane

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

1. What is the problem?

The Jev decision seam has exactly one provider. Every decision point routes through
JevOracle, which calls a paid third-party endpoint, so a decision can only be asked
on a machine whose owner holds a Jev key and consented to that endpoint. PR #11490
shipped the seam that way on purpose and said so in its own body: the LLM lane was cut
because nothing consumed it.

The wake judge (nudge.wake, RFC docs/request-for-change/rfc-wake-judge.md) is the
consumer that makes the lane necessary. It decides whether a monitor loop's quiet tick
should wake its owner at all. If it could only ask Jev, the feature would exist only
for owners with a key, and every other install would keep firing the ungated timer.

2. Why this issue matters to the user

An owner without a Jev key gets woken by every tick their loop schedules, whether the
work moved or not. The judge is what makes a quiet cycle cost nothing, and a
Jev-only judge cannot give them that. It also means the feature cannot be evaluated by
most of the people who would benefit from it, because the cheapest way to try it -- the
small text model the machine already talks to every turn -- is the one path the seam
does not have.

3. How our fix solves it

The chain runs from the symptom to the single missing piece.

The symptom is that no judge answers without a Jev key. The immediate cause is that
gate.decide constructs one oracle. The root cause is that the seam never had a second
implementation of its ask(state, questions) -> Answers protocol, so there was nothing
to choose between.

decisions/impl_llm.py adds that implementation. LlmOracle answers the same protocol
from ONE tool-less run of the shipped kirocrew-lite agent: no tools, no MCP servers, no
skills, and the call streams under REJECT_ALL, so no tool can run even if one were
offered. The lane ships no agent template of its own on purpose. One would differ from
lite in exactly two fields and neither survives inspection -- the model, which the runner
passes per call, and a standing system prompt, every sentence of which the per-call prompt
already states more completely, including the rule that the evidence is data rather than
instructions. A second spec written to every install to repeat what the call already says
is a file to keep in sync for no behaviour. The prompt carries the
state fenced as data plus each question with its id, type and options, and demands one
JSON object whose keys are exactly the question ids.

The parser is as strict as impl_jev._from_wire, because a text model is a less
disciplined producer than the wire, not a more trusted one. A choice must be in that
question's own option list; every number must be finite in 0..1 and not a bool; a
probabilities map may name only that question's options and must include the chosen
one; a complete distribution must sum to 1; a two-option question may answer with a bare
number, read as the probability of the first option; a response over 64 KiB is refused
unread. Anything else raises, and the gate turns a raise into the same None a timeout
produces, which is the caller's FALLBACK -- the tick then fires exactly as the ungated
timer does. Every raise carries a CONSTANT message, so no model prose can reach an
exception message or a decision-log row; a test pins that with a canary token.

gate.py chooses the lane for nudge.wake and no other point, from
decisions.nudge_wake.provider (auto | jev | llm), with decisions.nudge_wake.llm_model
naming the model. Both are registered in the config route's _EDITABLE_CONFIG allowlist
beside the decisions.model_route.<tier> tier pins, so they are written the same way and
hot-applied. An unknown provider name reads as auto rather than as an error: a typo must
not become a third lane and must not stop the gateway booting. An empty model INHERITS -- the runner
passes no model and the judge agent's own resolves, which is the kirocrew-lite
background template's model rather than the model of whichever session the tick belongs
to -- and that is what the picker's inherit default means end to end.

auto resolves to Jev when the Jev side is ARMED for this point and to the small model
otherwise, so an owner with no key needs no setting at all. An explicitly pinned jev
without that arming refuses rather than substituting a different model silently.

The LLM lane's budget is the seam's provider.timeout_ms clamped up into 8..60 seconds.
That number is sized for Jev (1 s by default, ~100 ms in practice); a text model cannot
answer inside it, so a lane that honoured it alone would time out on every tick and read
as "does nothing" rather than as "misconfigured". Clamped rather than replaced, so an
operator who raised it deliberately still gets what they asked for.

docs/system-specs/modules/decisions.md gains section 14 for the lane, the parser
contract and the authorization below.

Security

The two lanes are authorized by different things, because they send to different places.

The Jev lane is unchanged: it needs the keystone in full, consent for the configured
endpoint AND this point's own nudge_evidence scope. That is what a scope means here, a
category of egress to the Jev endpoint. The judge reads the scope fail-closed through a
new _point_scope_granted rather than the existing _scope_consented, which answers
True for a point with no registered scope -- correct for skills.select, which sends
nothing beyond what the main switch records, and wrong here, where the absence of a
registration means nothing has authorized Jev to receive worker transcripts.

The LLM lane needs no consent row, and provider = llm plus a judge spec on the
loop is its whole authorization:

  • Destination unchanged. The state goes to the model provider the owner's sessions
    already send to on every turn. No new endpoint, no new credential, no new egress path.
  • Data class unchanged. The evidence is the owner's own children's transcripts,
    review comments and ledger events -- creator-only material that the same provider
    already received when those sessions ran.
  • Worst case bounded. The judge chooses only QUIET against firing and is fail-open.
    Every failure (refusal, timeout, prose, invalid answer, missing runner) returns None
    and the tick fires as it does today, so the worst case is one delayed wake, bounded by
    the quiet-streak floor.
  • Spend-reducing. One short judge call replaces the full agent turn a needless wake
    would have run.

What still binds both lanes is the FLEET ceiling. A managed install that pinned
capabilities.decisions off withdrew the seam, not one provider's endpoint, so the LLM
lane pays that governed capability probe itself rather than inheriting it from the
keystone read.

4. What tests we did

  • test/test_decisions_judge_llm.py (new): the parser strictness table, one row per way a
    model can be wrong (valid forms, extra key, missing id, prose, NaN, out-of-range,
    bool-as-number, a probability map naming a foreign option, an over-ceiling response); a
    canary token asserting no model text reaches an exception message; the prompt's fencing
    and question rendering; the runner seam including a model id that is not one; the lane
    selection matrix over provider x consent; the authorization matrix (LLM with no consent
    and no scope, an unregistered scope on each lane, Jev needing both halves, auto
    falling back, the fleet ceiling); the budget clamp; _EDITABLE_CONFIG registration and
    the endpoint and key staying uneditable; and an end-to-end decide per failure path.
  • test/test_decisions_gate.py: unchanged except one line, and re-run whole.
  • test/test_settingref_schema_fixture.py: the settings-reference fixture for the new
    config section.
  • Result after rebasing onto current main: 194 passed, three files, -n0.
    isort, black and flake8 clean on all ten files. CI carries the breadth.

5. Any other suggestions on the work

  • This PR is the provider lane only. The core of the feature -- the nudge.wake
    decision point, its collectors, the autonudge hook and its registration in
    gate.POINTS with the nudge_evidence scope -- is a separate PR
    (feat/wake-judge-core). The RFC lands separately too. This PR is the second oracle plus
    the per-point provider keys, and it is inert until the point is registered.
  • Two one-line overlaps with sibling PRs. "nudge.wake" in the pinned point
    vocabulary in test/test_decisions_gate.py (whose own docstring says adding a point is a
    deliberate edit there) and one constant in
    website/src/pages/settings/decisionsPreview.ts (a cross-layer guard requires one per
    backend point). Both are also writable by the core PR or by the Decisions card
    redesign; whoever lands second drops the duplicate.
  • The Settings UI is in this PR, on top of the Decisions card redesign. That card
    projects one row per point from the gate's own registry, so nudge.wake needed no new
    card: it needed a label, controls in its panel, and a status the gateway computes from
    the provider as well as the keystone. All three are here. The scope registration itself
    is not: nudge_evidence belongs to the core PR, so until that lands the Jev lane
    refuses and only the small-model lane can answer.
  • This PR rewrites one pinned invariant, deliberately.
    test_decisions_config.py::TestDefaults::test_no_field_here_is_a_switch pinned the
    sentence "every field is a bound or an address -- nothing here grants egress". That
    stops being literally true once decisions.nudge_wake.provider = llm is what runs the
    LLM lane, so the pin now reads "nothing here opens a destination the session does not
    already send to", which is the honest form of the same guarantee and is exactly what the
    Security section above argues. The field-name sets in that test and in the round-trip
    test also gain nudge_wake, which is what those pins exist to make deliberate. Named
    here because weakening a pinned invariant should be announced rather than noticed.
  • The lane is deliberately configured per point rather than globally. A second point that
    wants a cheap judge should get its own key, so no point inherits another's provider.

Screenshots

All three frames come from node website/scripts/capture-decisions-preview-card.mjs on this
head, so every string in them is a string that ships.

The nudge.wake panel with the card's consent switch off: the row reads "Judged by the small model" while every other row reads Off, and both pickers are reachable

The panel with the card's consent switch OFF and the provider on the small model, which is
the state this lane exists for. The row does not borrow the generic active word there: under
a list headed "What Jev decides while this is on" that word would contradict the switch, so
the row names the lane that would answer. Both pickers are reachable with the switch off,
because the small-model lane needs no key and no agreement.

The same panel with the provider menu open: each of the three options states where the evidence goes

The provider menu open, which is the frame that matters for the choice itself: each option
says where the evidence goes before it is picked, and the collapsed control keeps that
marking rather than truncating it.

The same panel with the provider pinned to Jev while the switch is off: the row reads Off and names no lane

The refuse path, reached by pinning Jev while its switch is off. The gate refuses that lane,
so the row reads the plain Off word rather than naming one and the small-model note is
absent, because on this provider the switch above does govern the answer.

The harness asserts these frames rather than only writing them: the provider must read as
the chosen lane, the model must read as inherit, the row must carry the lane words on the
small model and none on a refused Jev, and the panel must be fully inside the frame.

Open review items, carried deliberately

Two advisory findings are disclosed rather than fixed here, because both need user-visible
copy in thirteen catalogs and neither changes what the gate does.

  • "The small model" as the name of this lane collides with the card's own opening line,
    "Jev is a small, fast decision model": both lanes are small models, so the phrase does not
    separate them. The egress clause on each option does, which is the safety-relevant half.
    A rename belongs in one careful pass over every catalog, not appended to this diff.
  • On the refuse path above, the surface says the judge is off but not that the quiet
    check-in it would have judged is spent ungated. Saying so needs one more string on the
    point panel.

@chenmingwei23
chenmingwei23 requested review from a team and cixuuz September 22, 2026 09:57
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 1f413b4f40d5cc4dea99f368eb99aef01cf45e45 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The card reports "Judged by the small model" as active at this HEAD, but nothing can judge — the point's ask site lives in an unlanded sibling PR.

Watch

Everything that makes this PR's claims true ships in feat/wake-judge-core: the nudge.wake ask site, the scope registration, and the quiet-streak floor that bounds the worst case cited to justify rewriting the test_no_field_here_is_a_switch pin ("nothing here grants egress" → "no destination the session does not already send to"). Meanwhile this PR adds nudge.wake to DECISION_POINT_NAMES and its _points row can read active (status = _POINT_ACTIVE if _llm_lane_available()), so if the core PR stalls, main ships a picker and a status word describing a judge that never runs, and a deliberately weakened security pin whose bounding control does not exist anywhere.
Clears when: feat/wake-judge-core lands adjacent (or first), or the row's active status is gated on the point's module being registered rather than on the lane's runner alone.

Suggestions

  • Prefer landing the core PR first: it removes the dead-surface window, resolves the two disclosed one-line overlaps in the cheaper direction, and lets the pin rewrite land in the same commit as the floor it cites.

[DESIGN-REVIEWED] 1f413b4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 1f413b4f40d5cc4dea99f368eb99aef01cf45e45 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1f413b4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1f413b4f40d5cc4dea99f368eb99aef01cf45e45: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 1f413b4f40d5cc4dea99f368eb99aef01cf45e45 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The lane ships consumer-less: nothing at this HEAD asks nudge.wake, its licensing RFC is draft on base, and #11490 cut this exact lane for exactly that.

Not justified as shipped

  • Items 1–5 — zero consumers: the description's harm ("an owner without a Jev key gets woken by every tick") is not removed here — grep nudge\.wake|JUDGE_POINT over src/ hits only this PR's own definitions in gate.py, sections.py, handlers/; there is no decisions/points/nudge_wake.py and no decide/is_enabled caller, the ask site "arrives with the core wake-judge change" (this PR's own feature-map hunk). The PR's spec quotes feat(decisions): use Jev for sampled skill selection #11490: "the LLM lane was cut because nothing consumed it" — that condition still holds verbatim at this HEAD, and the authority cited for the no-consent stance (gate.py: "RFC rfc-wake-judge, Providers") reads draft on the base RFC list, a proposal not a decision.
    Subtraction: defer — land this beneath or with the ask-site change instead of ahead of it.
    Clears when: a merged consumer calls decide("nudge.wake", …) on the base (push or rebase after), or a maintainer decides the seam lands first.
  • Item 6 — rides along: the Jev half of the selector is dead in this build — _POINT_SCOPES registers 3 scope kinds (tool_args, compaction, memory_text), none for nudge.wake, so a pinned jev always refuses and auto always resolves LLM.

What this change ships

Inventory (8 items) — 2 justified

Intent: let the wake judge answer on machines without a Jev key by adding an LLM lane to the decisions seam — an ADDITION.

  1. Decisions card gains a seventh row, nudge.wake, whose status names the answering lane — zero consumers: no tick asks the point
  2. A key-free LLM judge lane (one tool-less kirocrew-lite call, strict parser, 8–60 s budget, runner wired at DashboardState boot) — zero consumers
  3. New config key decisions.nudge_wake.provider (auto|jev|llm, default auto), hot-applied — zero consumers
  4. New config key decisions.nudge_wake.llm_model (empty = inherit) — zero consumers
  5. Provider and model pickers in the point's panel, usable with the consent switch off — zero consumers
  6. A pinned jev always refuses and auto always picks the LLM lane in this build — rides along: the nudge_evidence scope it arms on is registered by the follow-up change
  7. Judge row carries a server-supplied lane word; _points now also reads the fleet denial — justified
  8. i18n strings (13 locales), settings-registry entries, screenshot-harness rows, and a measured bundle-budget raise (905→951 KB per the documented 5% convention) — justified

[FIRST-PRINCIPLES-REVIEWED] 1f413b4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — 🟡 CONCERNS

UX-level review of 1f413b4f40d5cc4dea99f368eb99aef01cf45e45 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

Judge-row status contradicts the card's frame: active while the switch is off, "Off" while Jev is picked — the blind reader couldn't tell what's running.

Watch

  • The one row that runs while the master switch is off sits under the unchanged heading "What Jev decides while this is on:", and pinning "Jev — sends the evidence to the address above" with the switch off yields a plain "Off" chip (shot-03). The reader: "I cannot tell from looking whether this 'Quiet check-ins' thing is actually doing anything in either picture, or what 'Off' means on that row when a judge is picked." Every card visit, persistent, misread state rather than task failure. Smallest fix: mirror decisions_judge_lane_note for the pinned-Jev-while-off case ("Jev is picked but answers only once the switch above is on").
  • "The small model" collides with the intro's "Jev is a small, fast decision model" — the reader: "both are described as models that judge things… I couldn't tell you what 'the small model' actually is." Rename the decisions_judge_provider_llm option around what distinguishes it, e.g. "Your own model provider — no extra service".

Suggestions

  • decisions_judge_model_inherit "Keep the judge agent's own model" names an entity never introduced here (reader: "I don't truly know what 'the judge agent' is"); say "Default judge model" or similar.
  • The "Judged by the small model" chip truncates its own row title to "Quiet check-ins: w…" (shot-01); shorten the chip (e.g. "Small model") so the row name survives.

[UX-REVIEWED] 1f413b4

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 1f413b4f40d5cc4dea99f368eb99aef01cf45e45 — this comment is updated in place on each push.

Review details

FINDING. A legitimate bracketed ([1m]) judge model silently disables the new LLM lane.

FINDING — src/kiro_crew/decisions/gate.py:588 — lane_model returns the operator's llm_model verbatim (return configured or JUDGE_MODEL_DEFAULT), so an advertised 1M-window id such as "claude-opus-4-8[1m]" — offered by the new judge picker (judgeModelOptions = modelsQ.data.map(m => m.name)), accepted by the config pattern ^[A-Za-z0-9._\-\[\]]*$ and by _validate_role_model — reaches scrub_reason(..., model="claude-opus-4-8[1m]"), where is_model_id rejects the brackets (MODEL_ID_RE = \A[A-Za-z0-9][A-Za-z0-9._-]{0,63}\Z) and returns ERROR_SCRUBBED_MODEL; decide then writes a scrubbed:provider-model row and returns None before _oracle/build_session_runner's graceful "invalid id → inherit" is ever reached, so every wake tick fires ungated and the LLM lane silently does nothing → Fix: in lane_model (LLM branch) normalize the window suffix off the configured id before it reaches the scrub (as acp/client.py:6977 does with stripped.replace("[1m]", "")), so the judge's accepted grammar matches the picker/config grammar the rest of the model-selection surface advertises.

[OPUS-REVIEWED] 1f413b4

Verdict parsed from the review's SHA-scoped output markers for commit 1f413b4f40d5cc4dea99f368eb99aef01cf45e45.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1f413b4f40d5cc4dea99f368eb99aef01cf45e45: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from 0825b91 to ad9ab94 Compare September 22, 2026 10:35
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 0825b9131c874dbf156519b5c74875948c00a36c; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0825b9131c874dbf156519b5c74875948c00a36c: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from ad9ab94 to e833c5b Compare September 22, 2026 11:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for the first-principles review, on head e833c5ba50.

Item 3 (duplicate of kirocrew-lite) -- applied. Both halves of the claim check out:
build_session_runner passes model= per call, and render_prompt states every sentence
of the template's system prompt more completely, including the
evidence-is-not-instructions rule. The template therefore differed from lite in nothing
that survives inspection. _install_judge_agent, _JUDGE_SYSTEM_PROMPT and
JUDGE_AGENT_FILENAME are deleted and the lane runs on kirocrew-lite, so agent.py and
agent_files.py are byte-identical to main again and have left the diff. The tool-less
property never rested on the template: the call streams under REJECT_ALL.

Item 8 (website constant DECISIONS_NUDGE_WAKE_POINT) -- not applied; the subtraction
would break a committed test.
The constant is not there for an importer.
test/test_settingref_schema_fixture.py::TestDecisionsSettingCrossLayer::test_the_frontend_and_backend_name_the_same_points
asserts set(DECISION_POINT_NAMES) - frontend == {tool.risk}, and gives its reason in its
own failure message: "a point is an egress path -- name it in decisionsPreview.ts so
some surface shows what was sent." A point added to the vocabulary without the constant
fails that assertion. Deleting the export is only available together with dropping
nudge.wake from DECISION_POINT_NAMES, which would leave the lane unreachable by the
gate's own list.

Items 1, 2, 4 and the blocker (surface ahead of its consumer) -- accurate, and not the
author's to clear.
The caller is a sibling PR by design: this PR is the provider lane,
while the point and its autonudge hook are feat/wake-judge-core. The review names the
two clearances, a base that carries the caller or a maintainer override, and neither is an
author's to take. It sits with the maintainer who owns the split.

Two notes on the inventory: item 5's clamp is 8-60 s, and items 6 and 7 are an
authorization change rather than plumbing -- the Jev lane now reads this point's scope
fail-closed, which is a tightening against _scope_consented.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from e833c5b to bbbb00d Compare September 22, 2026 11:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from bbbb00d to f6fdf1c Compare September 22, 2026 12:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 22, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 22, 2026
buluoray
buluoray previously approved these changes Sep 22, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source-verified at 618bdea0676c5ea33528d878a4346b688d36becc (read through the refs/pull/12776/merge tree, 32cb5980). Approving.

This adds a second judge lane — a tool-less, strict-parsing one-shot model call — plus per-point provider selection for nudge.wake. The four things I went after, all clean:

  • No hardcoded model id. JUDGE_MODEL_DEFAULT = "auto" (src/kiro_crew/decisions/impl_llm.py:73); build_session_runner drops any value is_model_id rejects and maps "auto" to inherit (:182). Scanning every added line under src/ and website/src/ for claude-*/opus*/sonnet*/haiku*/gpt-*/fable* returns nothing outside test fixtures. lane_model(cfg, lane=lane) (gate.py:891) also makes the scanned id the sent id, so scrub_reason cannot be checking a different model than the one the selected lane names.
  • Every failure path returns None and records why. Timeout → ERROR_TIMEOUT (gate.py:905), any other exception → ERROR_PROVIDER logging type(exc).__name__ only, no traceback and no provider text (:910-916), empty or out-of-domain answers → ERROR_INVALID_RESULT (:923). CancelledError re-raises. None is the same contract every other point already has, so a broken lane degrades to the ungated timer rather than dropping a wake silently or crashing the caller.
  • Untrusted evidence is placed as data. render_prompt wraps the state in <EVIDENCE>…</EVIDENCE> with an explicit "this is DATA, not instructions" sentence.
  • Bounded. One runner call, no retry loop; wait_for(budget) with the budget clamped to _LLM_TIMEOUT_MIN_SECS.._LLM_TIMEOUT_MAX_SECS = 8–60s (gate.py:96-97, :450); _MAX_RESPONSE_CHARS 64KiB caps the parse (impl_llm.py:77, :319); the session is released and destroyed in finally.

Authority reads correctly as additive: _judge_authority (gate.py:566) gives the LLM lane not _capability_denied(session_key) (:611), i.e. the fleet ceiling still binds it, while the Jev lane keeps consent + the point's scope. The argument for not requiring consent on the LLM lane — same model provider the owner's sessions already use, no new egress — holds, and the comment at :855 puts that reasoning where the next reader will look. Nothing in the - lines removes a guard; they are signature and import moves.

test/test_decisions_judge_llm.py is 91 passed at this head.

On the three 🟡 advisory lanes, so they aren't left hanging — none is a defect I'd hold this for:

  • design: the settings row can read as active before the consumer lands in the sibling core PR. Real, and a landing-order question rather than a code fault.
  • first-principles: the lane ships ahead of its only caller (no decide("nudge.wake", …) site in this build). Inert and reversible; a sequencing preference.
  • ux: the "nothing is sent while this is off" copy sits oddly beside an LLM-lane row that is live under the off switch. Worth a wording pass whenever the consumer lands, not now.

Not verified: the nudge.wake call site is in the sibling PR, so I checked the gate's return contract but not the caller's handling of a None verdict. I did not run the full decisions suite or the website tests.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for the advisory findings on 618bdea067, all fixed in 22648313bc.

Accepted: the judge row did not match the gate (Opus, Design). Two cases were wrong,
both because the row derived its own answer about which lane runs. A pinned jev read
active whenever the keystone permitted, but gate._point_scope_granted fail-closes a point
with no registered scope, so decide refuses that lane and the row named one that cannot
run. A pinned llm read active with no registered runner, where every ask raises. The row
now resolves the lane through gate.judge_lane itself, with jev_armed computed from the
same fail-closed predicate the gate uses, and a pinned llm consults runner availability
the way the Jev side consults its grant.

Accepted: the card re-derived the lane (same finding, frontend half). It cannot: auto
resolves against the Jev lane being armed for this point, which needs that point's own
evidence scope, and the card holds no reader for a scope this build does not register. The
row now carries lane and the card reads it.

Accepted: a present-but-invalid confidence was discarded silently (GPT). Absent stays
legal, since the field is optional and the gate treats it as unstated. A present value that
is not a probability now raises, like every other malformed field in that parser.

Accepted: the inherit label named the wrong model (GPT). An empty llm_model resolves
the judge agent's own model, not the model of the session the tick belongs to. The picker
has its own label, and the spec and this PR's description now say the same thing.

Accepted: the consent surface still said nothing is sent (UX). The card's frame speaks
for the Jev endpoint, and every other point is covered by it. The panel now states, on the
one point that is the exception, which lane answers and where the evidence goes, drawn only
when that lane is in fact the live one.

Verification: 212 python tests across the two affected files, 92 card tests, 703 i18n tests
with I18N_BASE_REF set, tsc -b and ESLint clean, the bundle gate passing at 927,584 B
against the ceiling this branch raised with that measurement recorded, and the
comment-history and feature-map gates clean.

A second capture frame is attached with the provider menu open, because the option labels
are the reader's basis for the choice and the closed control shows only one of them.

Still open and not reachable from this branch: no production code calls
decide("nudge.wake"), because the caller is in #12787.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from 2264831 to 9b4ff7c Compare September 22, 2026 23:41
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Dispositions for 22648313bc, all in 9b4ff7c337.

UX BLOCK, missing evidence: accepted, and the cause was mine. The attached frames were
older than the strings they were supposed to show, so the blind read tested copy this PR had
already replaced. All three frames are re-captured on this head and re-attached, including the
one that did not exist before: the provider pinned to Jev while the card's switch is off, which
is the refuse path. The harness now fails that frame if the row names a lane there or if the
small-model note appears, so the two states cannot be photographed wrongly again.

Two items from that review are disclosed in the body rather than fixed here, under a heading
that says so. The naming collision is real: "the small model" does not separate this lane from
Jev, whose own intro line calls it "a small, fast decision model". What separates them on screen
is the egress clause each option carries, which is the safety-relevant half, so the rename is a
copy pass over thirteen catalogs and belongs in one careful change rather than appended to this
one. The second is the refuse path itself: the surface says the judge is off, and does not say
that the check-in it would have judged is spent ungated. That needs one more string on the point
panel, in the same pass.

Design CONCERNS: both items were accurate and both are fixed. Section 14 still closed with
the row and pickers arriving later and the row reading off until then, which two commits of
this PR had already falsified. It now describes what ships: the row's labels, both pickers and
why they stay reachable with the switch off, and the status rule as the gateway's, naming the
lane that would answer. The quiet-streak-floor claim is scoped where the second item asked: the
floor is enforced around the autonudge loop's own ask site and lands with the point's
registration, not here.

PR Hygiene: fixed. The branch carried three commits where the gate allows one or two. It is
one commit now, whose tree is identical to 22648313bc plus this cycle's three files.

Frontend shard 2: mine, fixed. readPoints gained the lane field and three cases in
decisionsPreview.test.ts pin the row shape exactly, which is what that block is for. They
carry the field now, with one case added for the lane being read through rather than derived.

Backend Tests (Windows) (7): inherited, and not curable from this branch.
test_crew_log_exc_info_sites.py is not in this branch's tree at all; it exists on origin/main,
which this head is 32 commits behind. Its assertion names frames in
src/kiro_crew/apps/builtins/issue_radar/backend/crew_store.py, a file this diff does not touch
and which is byte-identical between this head and main. CI runs the merge ref, so the test comes
from the base together with the code it flags, and a rebase moves that pair forward rather than
resolving it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/wake-judge-llm-lane branch from 9b4ff7c to 292ca74 Compare September 22, 2026 23:59
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

GPT's upheld finding on 9b4ff7c337 is fixed in 292ca74efa.

F1, feature-map-correctness: right on both clauses, and both were mine. The row said the
LLM lane's evidence "never leaves the machine the session already runs on". It does leave:
build_session_runner runs a real session, so the evidence reaches the configured model
provider. What is true is the claim the Security section actually rests on, so the row now
says the lane opens no DESTINATION, because the evidence goes to the provider the session's
own turns already go to. The pickers paragraph also said an empty llm_model inherits the
session's own model; it inherits the judge agent's. Both clauses corrected, feature-map gate
passes with the base ref set.

F2, the consent ratchet: I am leaving it as the adjudication found it. _keystone_statuses
excludes the judge row on purpose, documented at the helper: that file's subject is what the
keystone decides, and the judge's status also depends on whether a runner is wired, which the
file pins unavailable. Asserting the judge's lane-dependent status there would couple the
consent ratchet to runner wiring, which is the coupling the exclusion exists to avoid. The
row is pinned elsewhere instead: _points_off, the three provider cases and the
fleet-denial case in test_decisions_judge_llm.py, and the projection tests.

First Principles is BLOCK on the same premise as before, that nothing calls
decide("nudge.wake") in this build. That consumer is #12787's, still open, so this branch
cannot clear it.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Board state on 292ca74efa: all six review lanes green, no block marker, PR Hygiene, Bundle
Size Gate, Screenshot Evidence and every frontend shard green. First Principles moved to PASS.

Two reds remain and both are main's, not this branch's. Backend Tests (3.12, 7) and
Backend Tests (Windows) (7) fail on
test_crew_log_exc_info_sites.py::test_no_exc_info_site_anywhere_in_kiro_crew_holds_a_crew_log_handle,
which is not in this branch's tree: it comes from the base, and it names frames in
src/kiro_crew/apps/builtins/issue_radar/backend/crew_store.py, where origin/main carries
ten exc_info occurrences and this diff carries no lines at all. #12939 records main as red on
it and #12943 is the fix, still open. Coverage Gate fails behind those two shards rather than
on coverage.

Three advisory items from the reviews of this head are fixed and waiting on one push rather
than a second re-roll of six green review lanes: the feature map called nudge.wake the sixth
point where the registry holds seven, import uuid moved to module scope with the session-layer
import kept deferred and documented, and the 64 KiB response ceiling now counts per chunk as the
response streams, so it bounds what is retained instead of what is parsed, with one test pinning
the mid-stream refusal. They ship with the rebase once #12943 lands.

@bolichen97

Copy link
Copy Markdown
Contributor

@chenmingwei23 main now carries #12943 (80532c1), which fixes the test_crew_log_exc_info_sites red that shard 7 (Linux + Windows) and Coverage Gate inherit here. A rerun stays pinned to the old merge commit, so a rebase onto current main and push is what clears the last three reds; everything else on 292ca74ef is green and approved.

bolichen97
bolichen97 previously approved these changes Sep 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 292ca74 with Windows compatibility as the primary lens (618bdea was reviewed in full earlier; this is the delta review). Backend delta since 618bdea: handlers/decisions.py now takes the judge row's lane from gate.judge_lane and computes jev_armed with the same fail-closed predicate the gate uses (an unregistered scope reads as no grant, not as none needed), and the LLM lane reads active only when a runner is registered; impl_llm.py rejects a present-but-invalid confidence instead of silently dropping it. Pure logic - no path, file, lock, permission, signal, subprocess or encoding code, nothing that can differ on Windows. Windows evidence: on the earlier head the new/changed decisions tests ran on native Windows (375 passed, 1 pre-existing POSIX skip not added by this PR); on this head every Windows shard is green except shard 7, whose single failure is test_crew_log_exc_info_sites - inherited from main via the merge ref (crew_store.py sites landed by #12303, not touched here) and fixed on main by #12943 (80532c1). Linux shard 7 and Coverage Gate are the same inherited red. All review lanes are green on this head. Approving; merge once a CI round against main >= 80532c1 clears shard 7 (a rebase/push will do it - GitHub pins a rerun to the old merge commit).

…and a card row

The judge decision point `nudge.wake` had exactly one provider, `JevOracle`, so a
machine without a Jev key could not use the feature at all. This adds a second lane
that answers the same protocol from one text-only model call on the provider the
session already uses, teaches the gate to choose between them per point, and gives
the point its row and its two controls on the Decisions card.

- `decisions/impl_llm.py`: `LlmOracle` over an injected runner factory, plus a
  parser as strict as `_from_wire`: exactly the question ids, choices inside each
  question's own option set, every number finite in 0..1, a complete distribution
  that adds to 1, a present-but-invalid `confidence` refused rather than dropped,
  and prose refused unread. Every raise carries a constant message, because the
  gate logs the exception class into the decision log and a model may quote the
  evidence back.
- `gate.py`: lane selection for `nudge.wake` only, from
  `decisions.nudge_wake.provider`. The two lanes are authorized by different things
  because they send to different places. The Jev lane needs the keystone in full --
  consent for the endpoint AND this point's `nudge_evidence` scope, read fail-closed
  through `_point_scope_granted` rather than `_scope_consented`, which answers yes
  for a point with no registered scope. The LLM lane needs neither: it adds no
  destination and no data class, so `provider = llm` plus a `judge` spec on the loop
  is its authorization. Only the fleet ceiling binds both. `auto` resolves against
  the Jev side ARMED, so an owner who consented to the endpoint but never granted
  this point gets the small model instead of a refusal. The provider budget is
  clamped up for the LLM lane, which cannot answer inside a budget sized for a
  100 ms System One call, and both lanes' model ids go through the one existing
  `scrubbed:provider-model` refusal.
- No agent template of its own: the lane runs on the shipped `kirocrew-lite` spec,
  because a judge template would differ from it only in a model the runner passes
  per call and a standing prompt the per-call prompt already states.
- `decisions.nudge_wake.provider` and `.llm_model`, both writable through the config
  route's allowlist beside the `decisions.model_route.<tier>` pins, hot-applied per
  call, with an unknown provider name reading as `auto` so a typo is never a third
  lane and never stops a boot.

On the card the row already appears, because the list projects one row per point in
the gate's registry. Three things were missing and are here.

- The row had no label: its name and its one line of what it decides sit in the
  card's two label maps, keyed by the server's id like every other point, with the
  new strings in every catalog.
- The point's panel offered no control for its own two settings. It draws a provider
  picker (auto / Jev / the small model) and a model picker for the small-model lane,
  both shaped like `model.route`'s tier pickers and written through the same config
  route. Neither is gated on consent: the small-model lane needs none, and an owner
  with no Jev key has to be able to pick it while the switch is off. The panel also
  states which lane answers and where the evidence goes, because the card's frame
  speaks for the Jev endpoint and this lane is the exception to it.
- The row's status is the gateway's and names the LANE that would answer, resolved
  through `gate.judge_lane` and carried to the card as `lane`, so the word on screen
  and the oracle that answers come from one read. A pinned `llm` runs on its own, a
  pinned `jev` only with the Jev side armed, `auto` on either; the sampled share
  binds all three and the fleet denial refuses all three.

Three capture frames of the judge panel: the small-model lane with consent OFF, the
provider menu open where each option's destination is legible before it is picked,
and the provider pinned to Jev with the switch off, which is the refuse path.

Two existing card expectations move because the projection lists a seventh point:
the row-list assertion gains it and the End-key case lands on it. Both are the
deliberate edit that fixture exists to force. The `t` catalog chunk's bundle
ceiling moves 905 -> 951 KB on the file's own 5% convention, measured against a
base build rather than assumed.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 1f413b4 with Windows compatibility as the primary lens. Full review was done at 618bdea and the backend delta to 292ca74 was reviewed and approved earlier (gate.judge_lane-driven row, fail-closed jev_armed, runner-gated LLM row, strict confidence parsing). 1f413b4 changes no src/ file - only docs/feature-map, decisions.md, the capture script and decisionsPreview.test.ts - so nothing Windows-sensitive is added. This head's CI round ran against main >= 80532c1 (the #12943 exc_info fix) and is fully green: 85 success / 7 skipped including Windows backend shards 1-8, Windows IPv6, the Windows E2E boot and pod canary, and readiness: passed. Local Windows evidence from the earlier round: 375 decisions tests passed, 1 pre-existing POSIX skip not added by this PR. Approving and squash-merging.

@bolichen97

Copy link
Copy Markdown
Contributor

Correction to my approval note above: 1f413b4 did carry one src/ change relative to 292ca74, in decisions/impl_llm.py -- the response ceiling is now enforced per chunk while streaming (an on_chunk bound raising LlmProtocolError past _MAX_RESPONSE_CHARS) instead of only after the whole reply is collected, and the uuid import moved to module level. Reviewed after the merge: pure logic, no filesystem/process/encoding surface, tightens the memory bound, and the head CI round (incl. all Windows shards) covered it. The rest of the 2-dot diff was main movement from the rebase.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Green on 1f413b4f40: 85 lanes success, 7 skipped, zero failures, zero in flight, PR Readiness
success, no block marker. The two shard-7 reds were main's own exc_info violation and cleared
on the rebase onto #12943.

Three advisory verdicts remain, with dispositions.

First Principles and Design, both CONCERNS on the same point: nothing asks nudge.wake at this
head.
Accepted as accurate. The ask site, the scope registration and the quiet-streak floor all
live in the core wake-judge change, which the body names as a dependency. Design's alternative
remedy is to gate the row's active status on the point's module being registered rather than on
the lane's runner; that would be a probe for another change's module and becomes dead code the
moment it lands, so the honest options are the ordering ones: land the core change first or
adjacent. That is a maintainer's call rather than something this branch can fix, and it is the
one item this PR cannot close by itself.

UX, CONCERNS: four copy items, deliberately carried. The row chip truncates its own row title,
"the small model" does not separate this lane from Jev, the pinned-Jev-while-off row says only
"Off", and the inherit label names a judge agent the reader has not met. All four are copy in
thirteen catalogues plus a new string on the point panel. They are disclosed in the body under
"Open review items" and belong in one careful pass rather than appended here: the last three
pushes each cost a cycle to a locale style gate, and none of these four changes what the gate
does. Every one is reachable without re-opening the lane's design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants