Skip to content

fix(chat): suppress leaked token budget banners - #7626

Open
rubencu wants to merge 2 commits into
kirodotdev:mainfrom
rubencu:fix/suppress-weighted-token-banner
Open

rubencu wants to merge 2 commits into
kirodotdev:mainfrom
rubencu:fix/suppress-weighted-token-banner

Conversation

@rubencu

@rubencu rubencu commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop a provider's private You have N weighted tokens left banner from being shown, stored or delivered as the assistant's answer, without hiding a legitimate answer or echo.

The kiro-cli backend behind this installation's dashboard sessions emits the exact You have N weighted tokens left control banner to interactive chat consumers as if it were the assistant's answer (73 assistant rows across 8 of this host's dashboard transcripts between 2026-09-01 and 2026-09-17 start with it, 52 of them banner-only; verbatim excerpts are in this PR's discussion): as a whole message, glued to the reply's first word, or on its own line before the reply. Dashboard chat persisted it, mirrored it to linked channels and treated it as the turn's answer. Text alone cannot tell that artifact from an answer or echo the user asked for.

Why it matters

A banner-only turn ends with no answer while looking finished, so the user's request silently stalls. Stripping too much is the opposite failure: a false positive deletes content the user asked for. Recovery therefore needs structural provenance, not text matching, and the capacity-request fail-open has to see what the user actually said in the turn, including a steer they typed mid-turn.

Not a goal

  • No per-variant file attribution for regenerated answers (variant file snapshots, a regeneration recovery owner that settles a continuation into a parked selector row, a steer-boundary flush of that owner). Those changes are not part of this PR.
  • No Autopilot / orchestrator change: chat Autopilot was removed on main (chore: remove Autopilot plan/stage loop #15362, chore(frontend): remove Autopilot mode #15359).
  • No change to quoted, embedded, suffixed or echoed banners, or to any turn whose human text names tokens, a budget, capacity, a quota or the context window: they stay visible.
  • No new surface, setting, counter or user-facing control.
  • Dashboard chat only. The banner is recognised where the dashboard runner classifies a completed segment; the other consumers of the provider stream (the Slack gateway, cron dispatch, workflows, side chat, apps) still pass it through, and moving recognition into the ACP layer beside control_notice is tracked as Recognise the provider token-budget banner in the ACP layer beside control_notice #18970.
  • No suppression where the turn's evidence is not whole, which today is most long sessions. The ordinary-tail strip runs only when every source the model could echo is readable: no earlier tool row written before this change (it carries no completeness mark), no restored prefix outside the 500-row live window, no prompt image in this or an earlier turn (an image file the prompt as sent names, through a $skill body or another prepend, counts too, and the slot keeps that for every later turn until the gateway restarts), and no result that was cut, held an image or left out a field. Replayed on this head over this host's 2026-09-01 to 2026-09-17 transcripts, it strips 1 of the 8 banner-only tails that end a turn after visible output on the stored rows, and at most 2 of 8 once the new writer has marked the rows: 4 sit in one 21,713-row session past the live window and 2 follow an earlier prompt screenshot. Those banners stay visible; that is the cost of failing open, recorded here at @bolichen97's request (review of 2026-10-17). The rounds since add evidence sources (the prompt as sent, every consumed steer, a peer's included, count digits a tool result carries as JSON escapes, and the bodies an earlier skill load recorded) and not-whole marks (an MCP sibling field, a free-text status, an image file the prompt as sent names, a skill-load row with no readable body); each can only keep more of those tails visible, so 1 and 2 of 8 stay upper bounds.

What changed (motivation → approach → change)

Symptom → cause. The runner took any leading text as the answer, so a provider-authored budget banner became a transcript row, a channel reply and the turn's result.

Grammar (chat_utils.strip_provider_budget_banner). One leading banner with a plain or comma-grouped count, ending at a line break, at the end of the text, or directly before an uppercase letter. Recognition proves the shape, not who wrote the bytes.

Authority (chat_utils.classify_provider_budget_banner, called once in the settle phase before answer detection, persistence, Stop hooks and delivery). The checks that keep the text visible run first, on every turn kind: a typed transient retry keeps a repeated answer, any message from the session's own human that names tokens, a budget, capacity, a quota or the context window keeps the bytes visible (chat_utils.mentions_provider_budget is a topic test, not a wording list: no verb, subject or question shape is required and whitespace is folded first; seven review rounds of ever-new remaining-token wordings showed a list of question shapes cannot be complete, and failing open on a turn that mentions tokens costs one visible leaked banner, the behaviour before this change on exactly those turns, while a missed wording deletes an answer the person asked for, so a message about an API token or a project budget fails open too), and a banner whose count appears in the human's own words this turn (the message and each steer they typed, in any language: a translation request holding the figure names no topic word the gate knows), in what the model read as input this turn, whoever wrote it (the prompt as sent, so an @prompt body or an injected payload, and every steer the turn consumed, a peer's included; count evidence only, never the topic gate), in a tool result delivered this turn or in any row of an earlier turn, whoever wrote it (its text, or a tool row's recorded output; the same number, however the digits are grouped, in any case or wording: the text may hold the banner sentence or only the figure) stays visible as a possible echo, reformatting or repeat of text the turn can see; a count that appears nowhere the turn can see is the provider's own figure. Only then may a typed provider-budget recovery strip the prefix and keep the answer, and an ordinary turn strip a banner-only final tail after it already showed visible output and only while the echo evidence is whole (Evidence completeness and What the capacity gate reads, under Change detail).

Owning specs: docs/system-specs/modules/session.md (Provider-budget banner suppression) and docs/system-specs/modules/providers.md (transient retry provenance).

Backwards compatibility

Compatible: suppression applies only to the recognized leading banner under the authority rule above; the queue drain drops only entries this PR tags; every other reply, row and queue entry is unchanged.

Reader: src/kiro_crew/dashboard/chat_turn/segment_flush.py:_flush_segment -- chat -- a banner-only segment persists no assistant row and the continuation's answer is persisted instead (test_banner_only_final_segment_is_suppressed_and_continued_once, test_banner_only_file_changes_use_current_turn_placeholder)
Reader: src/kiro_crew/dashboard/chat_turn/segment_flush.py:_turn_settle_answer -- chat -- Stop hooks and cross-surface delivery read the classified assistant_text, which is the stripped answer, or empty on a recovering turn (Stop-hook order on that turn: test_banner_recovery_precedes_stop_hook_without_replacing_visible_variant)
Reader: src/kiro_crew/dashboard/chat_turn/steer_queue.py:_requeue_unconsumed_steers -- chat -- an acknowledged steer the turn never consumed requeues with the origin its caller reported and releases the record (test_acked_steer_the_turn_never_consumed_settles_at_the_turn_end)

Tests

Rebased on main 32a43272d, two commits, author kept. On this head: the 288 test modules that touch the changed modules (12,990 passed, 16 skipped), the src/i18n frontend tests (1,163 passed, I18N_BASE_REF=origin/main), isort, Flake8 and mypy on the changed files. Earlier validation, on 5295103662: Main split _run_chat into phase owners under dashboard/chat_turn/, so this change's runner hunks moved with the code they edit: the banner classification and the banner-only recovery arm are _classify_turn_banner and _settle_provider_budget_banner in chat_turn/segment_flush.py, called from _turn_settle_answer; the steer-consumed evidence is in chat_turn/steer_queue._on_steer_consumed; the provenance arguments sit on the recovery sites in acp_recovery.py, stop_recovery.py, turn_failures.py and model_fallback.py; the not-landed guards in stop_recovery.py and turn_stats.py; the Stop-hook insert index in turn_spec_hooks.py. Main already holds a steer's origin records until it settles, so this change's edits to chat_delivery.py, steer_queue._settle_consumed_steers and their tests were dropped as landed. Main has since moved steering_outcome and steering_advertised into acp/steering_wire.py itself, so this change's own copy (acp/steering_answers.py) was dropped on rebase. Checked on this head: the PR's test modules, every test_chat*, test_acp*, test_dashboard_chat*, test_provider*, test_steer*, test_session_*, test_tool_result*, test_discord*, test_state_parts*, test_link_screen* and facade module (594 modules: 23,708 passed, 17 skipped); 7 cases fail the same way on main on this host (agent-env and launch goldens, private-group CLI name, two composition guards fixed here). isort, Flake8, mypy, comment-history, brand-name, line-cap, docs-lint and the flake-pattern ratchet pass.

  • test/test_provider_budget_banner_evidence_owner.py (17 tests): both bounds by count and size, each unreadable source making the evidence incomplete, rows read only when asked, an overflowed record keeping a banner-only tail visible, and a provider-recovery turn that streams the banner then an answer (line-separated and glued) saving the answer. Without the raw-copy cut both end-to-end cases save the withheld notice.

  • test/test_provider_budget_banner.py (189 tests): grammar table, capacity-gate, classifier and tool-result echo tables, end-to-end turns for preservation, suppression, the one-shot continuation, the continuation under YOLO and session trust, the stop-generation snapshot, the pre-stream trust race, the loop bound, the transient owner, regeneration and the Stop-hook order.

  • test_capacity_gate_reads_a_steer_consumed_after_its_ack (7 cases) drives the real delivery order: steer_into_running_turn gets the RPC acknowledgement, then the stream echoes steering_consumed. The origin record survives the acknowledgement, a consumed user capacity request keeps the exact answer visible, a non-capacity user steer still suppresses, a peer's steer that carries the banner's count keeps the answer as an echo, a peer naming tokens without the count still suppresses, and the record is released afterwards. test_acked_steer_the_turn_never_consumed_settles_at_the_turn_end covers the turn-end settle.

  • test_capacity_request_steered_into_a_provider_recovery_keeps_the_banner (3 cases): a capacity request the human steers into a running provider recovery keeps the exact text; with no human request, or with the recovery's own continuation text matching the topic grammar, the banner is still stripped.

  • test_remaining_token_questions_keep_a_banner_only_tail_after_visible_output (26 cases): with prior visible output, where a banner-only tail is otherwise metadata, every remaining-token wording the review rounds produced (subject or none, modifiers, the ability form, a qualifier before tokens, the inverted auxiliary, a quantity phrase, the yes/no form, wrapped or double-spaced) and the other senses of the words (how many tokens did the prompt use, API tokens, project budget) keep the answer; only a message that names none of the topic words (finish the remaining tasks, how many files are left, count the lines in this file) still strips. test_remaining_token_question_keeps_its_answer_after_narration_and_a_read (8 cases) drives the end-to-end turn: narration, a usage.json read whose result holds only a number, then the banner-shaped answer, which stays visible with no continuation queued. test_banner_tail_after_narration_reads_this_turns_tool_results (4 cases) covers the echo evidence end to end, including a file whose line groups the digits (8,154) and one in lower case, with a request to repeat the line capitalized and without commas: the reply carries the file's number and stays visible. test_repeating_an_earlier_answer_after_a_read_stays_visible: an earlier turn's banner-shaped answer, then Check README, then repeat your previous answer exactly with a read that holds no figure; the repeat stays visible because the count is in an earlier assistant row. test_repeating_a_line_from_any_earlier_row_stays_visible (2 cases): the line came from the human's earlier message, or from an earlier tool's output, and this turn prints it after an edit; both stay visible.

  • test_a_cut_tool_result_keeps_an_ordinary_banner_only_tail_visible (4 cases): with prior visible output and a result whose visible head lacks the phrase, whole evidence strips and a cut result keeps the tail; typed recovery strips either way. test_banner_tail_after_a_cut_read_stays_visible drives the turn end to end: narration, a read whose result the parser cut past the phrase, the banner-shaped echo stays visible, no continuation. test_banner_echoed_from_stderr_stays_visible does the same for a shell result whose banner sits in stderr, with the event built by the real dispatch builder and provider translation of the envelope. test_recovery_turn_keeps_a_banner_shaped_echo_when_its_evidence_is_not_whole runs that envelope on a provider-budget recovery turn: the typed strip stands down and the echo stays. test_an_earlier_cut_tool_row_keeps_a_later_echo_visible: an earlier turn's cut result, marked on its row, makes a later turn's evidence incomplete and its banner-shaped repeat stays visible. test_the_count_in_the_humans_own_words_is_echo_evidence (5 cases): the figure in the human's own request, in French, Spanish or English, keeps the reply; words without it, or another figure, do not. test_capacity_gate_reads_a_steer_consumed_after_its_ack gained user-translation-count and peer-translation-count: a French translation steer, consumed after flushed narration, keeps the banner-shaped answer whoever typed it (a peer's too, since this round).

  • Evidence that is not whole, from the GPT round on 2e68fbf15c: test_an_earlier_prompt_image_keeps_a_later_transcription_visible (2 cases: an earlier user row attached the image, or typed its path) keeps a later banner-shaped transcription visible; test_a_container_or_number_beside_stdout_is_omitted_output and test_every_parser_marks_a_stdout_display_that_leaves_out_a_container mark a stdout-only Json display that leaves out a nested object, a list or a number (a boolean, a null or an empty container is not output); test_every_parser_marks_a_json_wrapped_mcp_image and test_a_raw_output_mcp_image_is_marked_by_both_streaming_parsers (2 cases) mark an MCP image block inside a Json item, a bare rawOutput envelope or one under result. Each is red without its fix.

  • From the GPT round on a19a4f949d. F1: test_a_turn_input_carrying_the_count_is_echo_evidence_and_never_the_topic_gate (4 cases) and test_an_expanded_prompt_carrying_the_count_keeps_its_answer_visible (2 cases: an @prompt body that carries the count keeps the answer, one without it leaves the strip), with the peer cases of the steer test above. F2: test_a_status_field_is_set_aside_only_while_it_reads_as_a_run_status (12 cases), test_every_parser_marks_a_free_text_status_beside_stdout, test_the_dispatch_builder_marks_an_mcp_field_its_text_display_drops and test_a_result_wrapped_mcp_text_display_marks_what_it_drops (3 cases). Mutation: dropping turn_inputs from the runner's call turns peer-capacity-text, peer-translation-count and prompt-carries-count red; dropping the MCP sibling mark turns the dispatch-builder test red.

  • From the GPT round on 5b7b891a25. F1 (test/test_provider_budget_banner_sent_prompt_image.py): a $skill body that names a real screenshot keeps the later transcription visible on every turn, an image an @prompt body sent stays evidence on a later turn, and a restored skill-load row keeps its image as evidence; a skill whose example path names no file still lets the tail be stripped, and only a path a regular file backs counts. F2 (test/test_tool_result_escaped_digit_evidence.py): count digits a tool result carries as JSON escapes (grouped, double-escaped, astral, after a no-break space) are evidence through the dispatch builder, the streaming parser and the session-file read-back, and the requested translation stays visible end to end; longer, decimal and lone numbers stay negative. Each fails without its fix.

  • From the GPT round on 79d63418e0. F1 (test/test_provider_budget_banner_skill_body_evidence.py): an earlier $reference body that carries the sentence keeps the next turn's repeat visible, and a skill-load row with no body leaves the evidence incomplete; a body without the figure still lets the tail be stripped. The first two fail without the fix. The same round's CI found two repository guards this change had not updated: the chat_utils facade's manifest now names the two JSON-escape helpers it forwards, and the link-screen baseline records the prompt-image check as a refusing site.

  • test/test_nothing_to_do_directive.py: test_a_banner_only_tail_after_the_quiet_end_keeps_the_quiet_end drives the real applier: narration, nothing_to_do, then a banner-only tail; the banner stays out of the transcript and the quiet end stands (no provider-artifact stop reason, no continuation, no notice, the turn is not read as interrupted). test_post_token_transient_retry_mints_its_owner_and_keeps_the_tail proves the transient owner from the production path: output, a transient 5xx after it, the one-shot CONTINUE queued under RecoveryProvenance.TRANSIENT_RETRY, the replayed row carrying that owner, and the owned retry keeping a banner-shaped tail an ordinary turn strips (red if the producer drops the tag or the consumer ignores it).

  • test/test_tool_result_truncation_evidence.py (43 tests): providers.acp_turns._to_llm_event forwards the flag (the production seam between parser and runner; test_acp_provider.py's field-parity test pins it too); each of the three parsers records the cut only when a bound removed text (under, at and over the bound; the read-back's per-part cut on text, stdout and dumped JSON parts; its join cut), and the exact shape the runner must not misread (a banner line as the last line past the bound).

  • test_discord_resumed_busy.py: test_an_acked_but_unconsumed_channel_steer_requeues_as_a_channel_turn (a Discord steer the client accepted and the turn never consumed requeues with both marks and commands off) and the accepted-steer reconciliation test now pins both records held; test_chat_runner_coverage_pending.py::TestSteerLifecycle::test_a_settled_steer_releases_both_origin_records.

  • test_banner_tail_after_narration_reads_this_turns_tool_results (2 cases): narration is flushed at a read's tool boundary, then the model repeats the file. The banner tail stays visible when that tool result carries its phrase, and is stripped and continued once when it does not. test_banner_phrase_in_a_tool_result_is_an_echo_candidate (6 rows) is the classifier table for the same rule.

  • TestPostTokenRecoveryPurge (5 tests, test/test_chat_runner_coverage_queue.py): drain-time purge on a queued correction and a resolved Stop; under auto-approve both owners of the shared continuation text keep their queue place and dispatch in order.

  • test_unattended_slot_guardrails.py (36 tests) passes.

  • test_discord_resumed_busy.py and test_session_broadcast.py now assert the origin record is held for the pending steer.

  • No frontend file changed, so no frontend run.

  • Mutation: putting the provider-recovery strip back before the capacity check turns test_capacity_request_steered_into_a_provider_recovery_keeps_the_banner[consumed-capacity-steer] red, with three classifier rows; counting the turn's own runner-authored text as human authority again turns its [recovery-text-names-capacity] case red.

  • Mutation: deleting the tool-result check turns test_banner_tail_after_narration_reads_this_turns_tool_results[file-echo] red, with three rows of the echo table.

  • Mutation: disabling the banner grammar match turns 46 of the 113 banner tests red, among them test_strip_is_narrow, test_banner_only_final_segment_is_suppressed_and_continued_once and test_capacity_gate_reads_a_steer_consumed_after_its_ack.

  • Mutation: deleting the origin record on the acknowledgement again turns all five real-sequence authority tests red, including test_capacity_request_steered_into_a_provider_recovery_keeps_the_banner[consumed-capacity-steer], which asserts only the visible answer.

  • Provider transcript: provider-banner-transcript.md holds verbatim excerpts of real dashboard transcripts, 73 assistant rows across 8 sessions from 2026-09-01 to 2026-09-17 that start with the banner: 52 banner-only (45 of them after visible output earlier in the same turn), 5 line-separated, 15 glued to an uppercase letter, and 1 glued to a Markdown heading, which the grammar does not recognize. A gate-by-gate replay on this head runs classify_provider_budget_banner over those turns with every argument computed from the stored rows as the runner computes it (a turn starts at a user, nudge, inject or subagent row; 500-row live window). 8 banner-only rows end their turn after visible output (the earlier count was 7; the eighth is in the rotated archive copy of one session); the rest are segments followed by more output in the same turn, which the runner keeps visible. On the stored rows, which predate the outputTruncated mark, 1 of the 8 strips: each of the other 7 has an unmarked earlier tool row in the live window, so the evidence reads as incomplete and the banner stays visible. With the marks the new writer records, reconstructed from each stored result's length (a result's stderr is not recoverable from the stored row, so this is an upper bound), 2 of the 8 strip: 4 are in one 21,713-row session, where the restored prefix outside the live window keeps the banner visible, and 2 follow an earlier prompt screenshot in the live window. No other gate fired on any of the 8: no token or budget topic in the human's words, no count in this turn's tool results or in any earlier row of the live window, and no result over 8,000 characters.

  • Black, isort, Flake8, mypy on the changed modules, docs lint, comment-history, brand-name, subprocess-encoding and git diff --check passed.

  • From @bolichen97's review of 2026-11-03. Ask 1: test/test_provider_budget_banner_prompt_chip_evidence.py keeps a banner-shaped reply visible after an earlier @prompt chip (tagged and legacy untagged), with a control where no chip is present and the tail is still stripped, and pins the prompt_load tag on the chip row. Ask 2: test/test_tool_result_truncation_evidence.py marks a dropped resource, resource_link or audio block in both streaming parsers, with a control where an empty text block stays whole. Ask 4: test_a_turn_with_no_banner_probes_no_file shows dispatch records the path and probes nothing until a banner arrives; test_the_screen_is_the_prompt_builders_own shows a path written as code is not an image. The exact-text condition stays covered by test_ordinary_read_only_near_miss_is_preserved.

Change detail

Evidence owner. Per the maintainer's call on 2026-11-01 (option (a): keep the echo evidence in this PR), the evidence has one owner, chat_utils_turn_text.TurnEchoEvidence. The settle phase builds it only once the banner's shape matched, so a turn with no banner walks no rows and probes no file. Which prompt paths count as images is the prompt builder's own scan and screen, moved out of build_prompt_blocks into the leaf module kiro_crew.image_path_screen so the dashboard reuses it without importing the ACP layer. Dispatch records those paths lexically (remember_sent_image_refs, at most 64 per slot); they are probed on a worker thread only once a reply opens with the banner (_probe_sent_prompt_images). A file removed after it was sent reads as no image. This is an interim fix: the filter's docstring points to #18970 for the ACP-layer recognition. The prompt as sent, the consumed steers and the human's own words are bounded there (retain_turn_evidence, extend_user_authority: TURN_EVIDENCE_MAX_ITEMS texts, TURN_EVIDENCE_MAX_CHARS characters); a text left out marks the evidence incomplete, so the banner stays visible. When a typed recovery strips the banner and keeps the answer, _drop_segment_head cuts the banner off the slot's raw segment copy too, so the flush saves the answer instead of the withheld notice. src/kiro_crew/providers/acp_turns.py forwards the parsers' tool_output_truncated flag onto the provider event the runner reads.

Evidence completeness. The three tool-result parsers (the dispatch terminal-frame builder, the client's streaming update parser, the kiro-cli session-file read-back, which also bounds each part) now record on the event (AcpEvent.tool_output_truncated) when their display bound cut the text, when a Json item's stdout-only display left out another output field the tool returned (stderr, or a status that does not read as a run status; an exit code or a status word is not output), or when the dispatch builder's display of a pure-text MCP envelope left out structuredContent or another output-bearing field, the ACP provider forwards it, and every new completed tool row records explicit completeness as meta.outputTruncated: false or true; a missing mark on a legacy row means unknown rather than whole. Any content block the parsers render no text for (an image, an embedded resource, a resource_link, audio, an unknown type) marks the result not whole too (tool_call_content_omits_text); only an empty text block omits nothing. Prompt images, an earlier @prompt expansion (the slot keeps only its "Loaded prompt" chip, now tagged meta.kind: prompt_load; an older untagged chip is matched by its text) and a restored frozen prefix outside the 500-row live window also make echo evidence incomplete. When any source is incomplete, a banner-only tail stays visible because the number may sit in unavailable text or pixels, and the typed-recovery strip stands down for the same reason.

What the capacity gate reads. The capacity gate reads the turn's own text unless it is runner- or app-authored (a recovery's continuation text never counts), plus every steer the turn consumed that this session's human typed. A message another session injected (session_send) or a scheduled job delivered is read like typed text: an injected capacity phrase keeps a banner visible, the pre-suppression behaviour, and loses nothing. The user-origin flag is not the gate because a queue entry restored across a restart carries no such flag (a documented security property of the durable queue), so a human request replayed from it would lose its answer. The tool-result text is read off this turn's own tool rows (meta.output: the redacted result, at most its first 8,000 characters).

Consumed-steer authority. The steer RPC acknowledges before kiro-cli echoes steering_consumed, and steer_into_running_turn deleted the steer's _steer_user_origin record on that acknowledgement, so the consumption echo could never read it. The record is now held while the steer is pending, read by the turn at the consumption echo, and released by _settle_consumed_steers; a steer the turn never consumed is released by the turn-end requeue, which reads the same record. The home is unchanged (_ChatSlot._steer_user_origin); only its release moved from the acknowledgement to the settle. The channel mark (_steer_channel_origin) is held and released in the same lockstep: the requeue reads the two as a pair, so a channel human's acknowledged, unconsumed steer re-enters the queue as a channel turn (commands off, channel authority), never as a dashboard human's turn.

Recovery provenance. A banner-only turn sets the stop reason provider_budget_artifact, persists no row, clears the streamed banner with an empty assistant frame, and queues one continuation tagged RecoveryProvenance.PROVIDER_BUDGET_ARTIFACT on the same conversation. The approval mode does not gate it: under YOLO, session trust and scoped unattended trust the continuation is queued and runs exactly as the post-token transient retry does, because it re-drives the same request that retry re-drives and no harm separates a banner-triggered continuation from a 5xx-triggered one. The entry arms the continuation replay record, so Stop, rebind and user input purge it like the promise-only and compaction continuations. The post-token transient retry carries RecoveryProvenance.TRANSIENT_RETRY for the same text. Requeues within one recovery episode carry the tag forward; a Stop-hook continuation queues behind the recovery. A regeneration whose prior answers are pending still persists its selector row, so those answers survive. An applied terminal directive (nothing_to_do) outranks the recovery: a banner arriving after it is activity after the sanctioned quiet end, so the banner stays out of the transcript but no stop reason, continuation or notice follows and the quiet end stands as a landed turn (docs/decisions/2026-10-03-every-agent-turn-ends-with-text-or-nothing-to-do.md).

Scope amendment (review round 16). Two parts of the earlier cut are gone: the auto-approve authority gate for the banner recovery (_provider_recovery_is_authorized, _drop_unauthorized_provider_recovery, its drain call and the two notices), because no threat separates a banner-triggered continuation under auto-approve from the 5xx-triggered one that already runs there; and the remaining-token wording list in the capacity gate, replaced by the topic test above after seven review rounds each found a wording it missed.

Manual verification

N/A — deterministic tests drive _run_chat and steer_into_running_turn with scripted provider streams for every banner shape and recovery path. The one UI change is the give-up notice row, which now carries a notice token and renders localized copy.

Give-up notice row (provider_budget_artifact_give_up), drawn by the real transcript renderer on website/capture/gateway-notice.html (dark theme). It names no button: on a turn that already showed output no recovery button appears, so it asks the person to send the request again. English, then German:

Give-up notice, English

Give-up notice, German

Related Issues

Follow-up #18970: recognise the banner in the ACP layer beside control_notice so every consumer of the event stream gets the same classification. Maintainer decision (bolichen97, 2026-11-01, option (a)): the echo evidence stays in this PR and its cost is accepted; #18970 follows after this lands and adds only the ACP-layer detection and the Slack test.

Sibling sweep

Sibling: src/kiro_crew/slack -- tracked in #18970

The same provider stream also reaches cron dispatch, workflows, side chat and apps; #18970's recognition beside control_notice covers every consumer.

Pattern harvest

Rule candidate: review-prompt
Pattern: recovery entries that share text must carry a typed owner; never key stripping or queue policy on text equality.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@rubencu
rubencu requested review from a team and bolichen97 September 1, 2026 13:26
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 1, 2026
@rubencu
rubencu force-pushed the fix/suppress-weighted-token-banner branch from be5851a to 748cada Compare September 1, 2026 13:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 782c64e891c2b6141118f39a78b3ef8be375b74f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fail-open behaviour and its low hit rate are recorded in the description and were accepted in an earlier maintainer review. A revert undoes all of it.

Notes

  • By the PR's own replay, about 1–2 of the 73 observed banner rows change. Two cases never strip or recover on an ordinary turn: a reply that is only the banner with no earlier visible output, and a banner glued to the reply's first word. classify_provider_budget_banner returns (text, False) for both. So for most observed cases, the "request silently stalls" harm in ## Why it matters is still there.
  • Every new tool row now persists meta.outputTruncated, and AcpEvent gains tool_output_truncated, a field that crosses the ACP layer. Both exist only to serve this text heuristic. If Recognise the provider token-budget banner in the ACP layer beside control_notice #18970 moves recognition beside control_notice, they become contracts nothing reads, so keep them easy to remove.
  • Releasing _steer_user_origin and _steer_channel_origin together, and only at settle, also fixes a separate bug: an acked channel steer being promoted to a dashboard turn. That fix would be easier to review and revert as its own PR.

[DESIGN-REVIEWED] 782c64e

Description read

[DESCRIPTION-READ] 9819bafd7a013945c10fac80d0233f024299e7f96d6cec4dbc28c2314ef340ee

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 782c64e891c2b6141118f39a78b3ef8be375b74f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The goal is in scope and the ACP-layer move is declared and deferred (#18970). Most of the size goes to hiding the banner, and the author's own replay says that hiding fires rarely.

Notes

  • Item 1 — symptom-level: the banner is recognised at the dashboard settle phase, but AcpEvent.control_notice (acp/types.py:1051) already exists to mark provider text as "not the answer". 23 of the 24 non-ACP, non-test src/ files matching EVENT_TEXT_CHUNK (Slack, cron, workflows, subagents, channel.py, apps) still pass the banner through. This is declared ("tracked as Recognise the provider token-budget banner in the ACP layer beside control_notice #18970") and accepted as deferred.
  • Item 4 — oversized: the evidence-completeness sources exist so a strip never deletes an answer, yet the author says the strip "strips 1 of the 8 banner-only tails … at most 2 of 8". The harm the description names ("a banner-only turn ends with no answer … silently stalls") goes away without hiding anything, through the existing compaction-notice pattern: control_notice plus subtracting it from the answer in should_continue_after_compaction. Subtraction: drop tool_output_truncated, _sent_prompt_image*, the prompt_load kind and the skill-body and escaped-digit evidence. Keep "not an answer → continue once" and leave the banner visible. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (9 items) — 6 justified

Intent: stop a leaked You have N weighted tokens left provider banner from standing in for the assistant's answer in dashboard chat. This is a FIX.

  1. A leading banner is recognised and stripped from the stored, delivered and mirrored answer in dashboard chat only — symptom-level
  2. A turn that is only the banner continues once automatically, on the same session — justified
  3. New give-up notice when that continuation is unavailable or already spent (13 locales) — justified
  4. The banner stays visible whenever the evidence is incomplete (tool-output truncation mark on AcpEvent/LLMEvent, prompt-image probe, prompt_load chip kind, skill bodies) — oversized
  5. Any message that names tokens, a budget, capacity, a quota or the context window keeps the banner visible — justified
  6. A count found anywhere the turn could see keeps the banner visible as a possible echo — justified
  7. Queued recoveries carry RecoveryProvenance tags instead of being compared by text — justified
  8. A steer's origin record is now held past the acknowledgement until the steer is consumed — justified
  9. Image-path screen moved out of prompt_blocks into image_path_screen.py, plus a post_turn_reply re-export in chat_runner — rides along

[FIRST-PRINCIPLES-REVIEWED] 782c64e

Description read

[DESCRIPTION-READ] 1ac82d28e13fce49440c4450f771b1c2f02cf0f2521eb262cfe7931adb8b0b33

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 782c64e891c2b6141118f39a78b3ef8be375b74f via the fork AI-review pipeline; updated in place on each push.

Adjudication upheld 1 of 2 blocking finding(s); those must be resolved before merging.

BLOCKING -- src/kiro_crew/dashboard/chat_utils_turn_text.py:1361 -- Tool-call inputs are missing from echo evidence

str(meta.get("output") or ""),
Trigger: An earlier completed file write records You have 8154 weighted tokens left only in meta.input, with number-free output and outputTruncated: false. In a short conversation with otherwise complete evidence, “Check README, then repeat the sentence you wrote” produces narration, a number-free README result, and that exact sentence. _turn_settle_answer → _classify_turn_banner → transcript_texts() ignores the recorded input while complete returns true.
Outcome: The requested echo is removed before persistence and delivery, and an unnecessary continuation replaces it.
Anchor: defect
Fix: Include tool-call inputs in echo evidence and fail open when their recorded content is incomplete or unavailable.

BLOCKING -- src/kiro_crew/dashboard/chat_utils_turn_text.py:1064 -- Equivalent numeric spellings cause legitimate translations to be suppressed

if _names_whole_number(_drop_digit_grouping(_fold_unicode_digits(candidate)), count):
Trigger: A translation request contains Il vous reste 08154 jetons pondérés; after narration is flushed at a tool boundary, the model returns You have 8154 weighted tokens left. With otherwise complete, number-free evidence, _classify_turn_banner → _carries_count rejects 08154 as evidence for 8154. Indian grouping such as ८,१५,४०० normalized to 815400 fails through the same path.
Outcome: The legitimate translation is classified as provider metadata and removed instead of being persisted and delivered.
Anchor: defect
Fix: Recognize equivalent whole-number representations, including zero-padding and non-Western digit grouping, conservatively preserving ambiguous replies.

[BLOCK-MERGE] 782c64e
[GPT-REVIEWED] 782c64e

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — rung MEDIUM, and the harm is silent: a reply the user asked for gets deleted with nothing shown. The tool row stores the full redacted input (chat_turn/tool_rows.py:158, which caps it via chat_utils.py:130-148). TurnEchoEvidence.transcript_texts/tool_results yield only content/output/skill bodies (chat_utils_turn_text.py:1344-1361), and classify_provider_budget_banner strips the reply once complete holds (:1149-1155). The only recovery is the continuation (segment_flush._settle_provider_budget_banner, patch :1368), which replaces the reply.
The real fix is cheap: yield one more recorded field, and treat a … [truncated at marker as incomplete. Harm outweighs that cost.

F2 — rung LOW. All of these must hold: the banner shape matches (:853-878); the asker's text names no token, budget, capacity or quota word (:1124); the source spells the count zero-padded or in lakh grouping, which _names_whole_number (:1032) and _drop_digit_grouping (:918-932) do not fold; evidence is complete (:1145); and the reply is banner-only after visible output (:1149). Lakh grouping only matters for counts of 100,000 or more, and zero-padding a count is contrived. Recovery is a single continuation, and if that also yields the banner, a visible give-up notice telling the user to resend (patch segment_flush.py :1368-1410).
The fix is an open-ended spelling chase: European 8.154, Swiss 8'154, numbers in words, and so on. Each one adds permanent folding code for a near-nonexistent path, so the cost exceeds the harm.

[ADJUDICATION] 782c64e891c2b6141118f39a78b3ef8be375b74f total=2 uphold=1 downgrade=1
UPHOLD F1 src/kiro_crew/dashboard/chat_utils_turn_text.py:1361 reason=harm-warrants-remedy
DOWNGRADE F2 src/kiro_crew/dashboard/chat_utils_turn_text.py:1064 reason=disproportionate-remedy
[GPT-ADJUDICATED] 782c64e891c2b6141118f39a78b3ef8be375b74f

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 782c64e891c2b6141118f39a78b3ef8be375b74f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 782c64e

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 1, 2026
@rubencu
rubencu force-pushed the fix/suppress-weighted-token-banner branch from 748cada to c7e954e Compare September 1, 2026 14:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 1, 2026
@rubencu
rubencu force-pushed the fix/suppress-weighted-token-banner branch from c7e954e to e00588a Compare September 1, 2026 15:50
@rubencu

rubencu commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for an earlier head, kept for the record (no longer a ledger record):

  • File-change attribution — fixed in e00588a63. Banner-only opted-in suppression now creates a non-broadcast empty assistant placeholder whenever the current turn has file changes. _attach_turn_stats and _flush_file_changes therefore bind to that current-turn row, never the preceding assistant message. test_banner_only_file_changes_use_current_turn_placeholder reproduces the screenshot shape and asserts the preceding row receives no file_changes metadata.

"Banner-only tool turns misattribute file changes"

  • Text-only normal-turn suppression — rebutted as an intentional safety boundary. The previous review correctly established that content-only matching can delete a legitimate requested answer. The matcher is therefore restricted to observed structural provenance: terminal tool activity, orchestrator execution, or the synthetic recovery turn; it also defers whenever the user's prompt contains weighted tokens left. test_mid_turn_exact_text_is_preserved and test_explicit_requested_text_is_never_suppressed lock both preservation paths. The reported screenshots are tool turns with file-change cards, so the observed defect remains covered without reintroducing content-only answer deletion.

"text-only normal turns ... banner still persists"

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 1, 2026
@rubencu

rubencu commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Maintainer rerun requested for the current-head GPT 5.6 Review (e00588a63bac81069ef88789884a49a4e12209c2). The check ran from 17:38:03Z to 18:43:18Z, then concluded failure with title review incomplete, zero annotations, and no current-head finding comment. All CI/coverage checks and the Opus, Design, First-Principles, and UX reviews are green; local GPT also passed this exact revision. Please rerun the fork GPT stage-2 review. This is a rerun request only—not an /ai-review override.

@rubencu

rubencu commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for an earlier head, kept for the record (no longer a ledger record):

"The whole defense is one exact regex ... make sure the upstream leak is reported/tracked"

  • Stage Stop/rollback semantics — rebutted as required in-scope correctness. The new same-conversation stage continuation adds await and capture boundaries where a Stop can land. A monotonic Stop snapshot and rollback of only the uncaptured current round are required to prevent ignored cancellation, false stage completion, or a skipped stage on the next Go. Focused regressions cover Stop during context preparation, initial turn, continuation, subagent wait, and capture, plus interrupted stage-2 resume and repeated-artifact rollback. Prior stages, results, escalation history, and failure history remain preserved.

"The stage-loop Stop-generation snapshot + abort_round rollback changes cancellation/capture semantics"

"a banner leaked on a plain no-tool chat turn ... still lands verbatim"

@rubencu

rubencu commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for an earlier head, kept for the record (no longer a ledger record):

"Point patch, 5+ unfixed siblings"

  • Stop-generation guards and abort_round — rebutted as inseparable from safe stage recovery. A banner-only stage now continues synchronously on the same conversation under one absolute deadline. Without Stop checks around every await/capture boundary, or without removing the uncaptured current round, the new recovery path can ignore Stop, capture a false result, or skip the interrupted stage on resume. The implementation narrows rollback to that stage key and timeout clock while preserving prior completed stages/results, escalations, and task-failure history; focused tests cover each cancellation boundary and stage-2 resume.

"Item 9 is a second fix inside this one"

@rubencu

rubencu commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for an earlier head, kept for the record (no longer a ledger record):

  • Owned provider-artifact stop reason — fixed in 28d6bbb1b (reviewed tree 120aef957). The orchestrator now imports _STOP_REASON_PROVIDER_BUDGET_ARTIFACT from chat_runner and uses it at all three terminal/continuation gates. A future stop-reason rename therefore cannot silently desynchronize stage capture.
  • abort_round contract documentation — fixed in 28d6bbb1b (reviewed tree 120aef957). The docstring now names every pre-capture abandonment class: stop, timeout, error, empty result, capture failure, and teardown. This matches all eight orchestrator call sites and preserves the retry-not-skip invariant.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 3, 2026
@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

The cost stands as described: parser marks, persisted row fields and the echo-evidence plumbing, for at most 2 of the 8 tails that end a turn after visible output. Cutting it is outcome (b) of the scope call, and keeping it is (a).
The same-span stall on chat_runner.py (at least 12 review rounds) is why this round makes no further instance change. Next step: the #18970 scope call in the note on this PR. This record supersedes comment 6294751147.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Recognition beside AcpEvent.control_notice is #18970's scope. Under (b) this PR shrinks to the typed-recovery strip and the notice, and #18970 adds the ACP-layer recognition. Under (a) the runner-local classifier ships as the interim fix.
The chat_runner.py span has stalled (at least 12 review rounds), so neither shape is built until the call is made. Next step: the #18970 scope call in the note on this PR. This record supersedes comment 6294751648.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Opus adjudication downgraded this one as disproportionate. It is a new instance on a span that has raised one in at least 12 review rounds, so under the same-span stall rule the next change is the restructure. Here that is the scope call: (a) keep echo evidence in this PR, with its sources gathered in one chat_utils_turn_text owner, or (b) move it to #18970.
Next step: the #18970 scope call in the note on this PR.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Adjudication upheld this one under the blocking rule a-bound-bounds-every-field-it-retains: _turn_input_texts and _user_authority_text keep growing with consumed steers after their admission records are released. The defect is real, and it exists only because of the echo-evidence machinery the scope call decides.
Under (a) the next round bounds both by count and size in the chat_utils_turn_text owner and marks evidence incomplete on overflow, as the finding suggests. Under (b) both fields are deleted with the evidence.
Either way it is one change, made after bolichen97 answers. This span has raised a new instance in at least 12 review rounds, so a patch now would be one more round of the same stall.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Opus adjudication downgraded this one: it needs a decoded answer that is the banner sentence word for word, with no count or capacity word anywhere in the evidence. The fix it asks for, provider-origin evidence before an ordinary answer is suppressed, is what #18970's ACP-layer recognition beside control_notice would supply.
Next step: the #18970 scope call in the note on this PR. Under (b) the ordinary-tail strip this finding targets leaves this PR.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author
  • The suppression gate leaves the motivating cases visible (span=ecd99a636ed1): a design gate for the maintainer; not redesigned in this round.

The lane proposes demoting a recognised banner to a host notice row instead of deleting it, which would retire the echo-evidence machinery. That is a different design for this PR, and the lane's own note puts recognition beside control_notice (#18970). The verdict on this head is PASS, so the item does not block.
Next step: the #18970 scope call in the note on this PR. Demotion fits either outcome; whether it lands here or in #18970 is bolichen97's call.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 the round on b72cbd31a5 has finished. I'm not patching more instances here until you settle the #18970 scope call. GPT's chat_runner.py span has raised a new instance in at least 12 review rounds, well past the three-round same-span stall rule, and the Design and First Principles lanes point at the same call.

There are two outcomes, and I'll build whichever you pick:

The disposition comments above name this call for each open finding. Nothing is built until you answer.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 64576241e0 as f8d0f5c75b, because the PR had become conflicting. Main's #23050 adds _turn_full_reply_text to dashboard/chat_runner.py where this change adds _STOP_REASON_PROVIDER_BUDGET_ARTIFACT. Main's function stays as it is and this change's constant follows it; every other per-file patch ID is unchanged from b72cbd31a5. No code changed beyond the conflict, and the scope question for bolichen97 (note 6332872266) still stands.

Checked on this head: the body's validation set (500 modules: 19,727 passed, 25 skipped, 4 xfailed, with test_slack_mirror_full_reply_23022.py from main's #23050 among them), the facade-mirror, link-screen, dashboard-files and chat-runner base-surface guards (443 passed), the script gates, the flake-pattern and wait-discipline ratchets, and the single-commit push guard. The body's validation paragraph names this head.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

The same scope question as spans ac8efac2c02d and 531a1e59b88f (dispositions 6332870443 and 6332870736), reworded on this rebased head. The open call, put to bolichen97 in 6332872266, is whether the evidence machinery stays in one chat_utils_turn_text owner, or the PR is cut to the typed-recovery strip and the notice and the evidence work moves to #18970. Nothing is built or removed until that is settled.
Next step: a maintainer settles that scope call. A human accepting the cost here is the lane's own third way to clear it.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author
  • The stall the PR names is not fixed (span=e035e88d612c): a design gate for the maintainer; not redesigned in this round.

The same point as spans 72cd27a9c4ea and ecd99a636ed1 (dispositions 6332869478 and 6332871942), reworded on this rebased head: a banner-only first answer is left to #18970's recognition at the ACP layer beside control_notice, which reaches every consumer. Narrowing this PR's Goal to the tail case, or moving the work to #18970, is the scope call put to bolichen97 in 6332872266.
Next step: a maintainer settles that scope call; this PR changes no design meanwhile.

@rubencu

rubencu commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Rebase onto main 5c32cf32ba to clear the conflict main's #22914 made: f8d0f5c75b -> d50cbda3a3.

#22914 makes _flush_segment return the row it persists, and its held-run seam (_cut_segment) reads that row. This change adds strip_provider_banner to the same signature, and its banner-only branch persists nothing. Two overloads keep main's dict[str, Any] return for every caller that does not opt in. An opted-in call is typed dict[str, Any] | None, and the banner-only branch now returns None explicitly. The base-surface pin takes the implementation's signature. Runtime behaviour is unchanged, and the one caller that opts in (the provider-budget branch in chat_runner.py) ignores the return. This change's own lines in chat_runner.py are unchanged; that file's patch ID moves only with #22914's context.

Nothing else changed while the scope question to bolichen97 (comment 6332872266) is open. The gate dispositions 6342072133 and 6342072496 were written for f8d0f5c75b.

Checked on 9598cb91cc (the same per-file patches on 66b3698653; main's one newer commit touches none of this change's files): the PR's own test modules (916 passed), the broad set (501 modules: 19,766 passed, 25 skipped, 4 xfailed), the repository-wide guards (443 passed), the 24 script gates including mypy, the ratchets, and the single-commit push guard, which passes again on d50cbda3a3. In the broad set, eight cases in main's new test_chat_runner_redact_across_tool_boundary.py fail on this host. They fail the same way on main 66b3698653. The body's validation paragraph is regenerated.

@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] @rubencu re your question in #7626 (comment): Thanks for waiting. Option (a): keep the echo evidence in this PR. #18970 will follow after this lands and only add the ACP-layer detection and the Slack test.

AI-assisted, posted on behalf of bolichen97

@bolichen97

Copy link
Copy Markdown
Contributor

Maintainer decision on scope: this PR is accepted as an interim fix for leaked token budget banners. #18970 is an open issue with no PR yet, and waiting on it would block a user-visible fix indefinitely. Two conditions:

  1. The PR description or a code comment at the filter must say this is interim and point to Recognise the provider token-budget banner in the ACP layer beside control_notice #18970 for the ACP-layer fix.
  2. The filter must drop only text it matches exactly as a budget banner, never ordinary assistant text, and a test must show that ordinary assistant text containing similar words is kept.

On the GPT 6.1 lane: its findings at f8d0f5c were downgraded and no longer block. Both describe a case where a legitimate, requested sentence that looks like a banner could be stripped (an earlier expanded @prompt body, and an image whose file was deleted before a resumed session). That falls under condition 2, so please either close those two paths or fail open there, with tests.

@iamwhatever

Copy link
Copy Markdown
Contributor

Addressed the Opus 5.5 findings on 5295103 in 30713f7 (rebased on latest main, one commit).

  • segment_flush.py:1153, redactor reset on the kept path: fixed. _wsred.reset() now runs only on the banner-only path, where everything it withholds is banner. On the kept path the answer branch's final _flush_text_stream emits the held tail, so chunk-built replies end with the answer's last word. Test: test_kept_answer_streams_its_last_word (no trailing punctuation, trailing period, one-word answer); all three fail without the fix.
  • stop_recovery.py:477, _recovering_provider_artifact on a refused continuation: fixed. When _settle_provider_budget_banner queues nothing and the turn already showed visible output, the flag is cleared, so the turn settles as landed (post_turn_reply, budget reset, consolidation, record_success). A banner-only turn that showed nothing stays unacted, which test_repeated_banner_does_not_loop still pins. Test: test_refused_continuation_after_visible_answer_settles_as_landed; fails without the fix.
  • segment_flush.py:1234, give-up notice without a token: fixed. New GATEWAY_NOTICE_PROVIDER_BUDGET_ARTIFACT_GIVE_UP (provider_budget_artifact_give_up), with its NOTICE_FAMILIES / COPY_KEY / LEGACY rows in gatewayNotice.ts, copy in all 12 catalogs plus regenerated en-XA, and the parity pins moved by one. Test: test_give_up_notice_carries_its_token; fails without the fix.

Also fixed the Black 26.10.1 formatting in test/test_provider_budget_banner_evidence_owner.py. Local runs: banner, notice-parity and nearby chat-turn suites (404 passed), gatewayNotice.test.tsx and the i18n vitest suites, Black gate, isort, flake8, mypy on the touched modules.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at f0a9148. The description still names #18970 as the follow-up that moves recognition into the ACP layer, which is the first of the two conditions this was accepted on. The second condition is that the filter hides only the exact provider banner text, with a test for it. Two upheld GPT 6.1 findings show real reply text that only repeats the banner wording still being hidden, so that condition does not hold yet.

1. Blocking (GPT 6.1, upheld): an earlier expanded prompt counts as complete evidence. In chat_utils_turn_text.py, TurnEchoEvidence.complete returns true even though _expand_prompt_mention_off_loop keeps only a loading chip for an earlier @reference prompt, not its text. So when a later turn asks for a translation of a banner-like sentence that came from that earlier prompt, _classify_turn_banner reads the evidence as whole and hides the translation the user asked for. Please mark evidence incomplete when a prompt's expanded text was not kept, so the filter falls back to showing the reply.

2. Blocking (GPT 6.1, upheld): embedded text resources are dropped without marking the evidence incomplete. In acp/_dispatch_parts/tool_result_events.py, a completed tool result with a text block plus a wrapped resource keeps only the text block and leaves tool_output_truncated false. If the banner-like sentence was inside the resource, the turn reads as having seen the whole output and the reply is hidden. Please set tool_output_truncated (or the equivalent) whenever a resource or other block is omitted.

A test for each case, where reply text repeats the banner wording and must stay visible, would cover the condition.

3. Blocking (UX Review): the give-up notice names a button that is not there. The notice says to press Continue, but the composer's recovery button reads Resume, and on a turn that already showed output no recovery button appears at all. Please make the notice match what the user can actually do.

4. Blocking (First Principles): _prompt_names_an_inlinable_image re-implements the image-path screen from acp/prompt_blocks.py and already differs from it (no user-span limit, no suffix check), and it runs a file probe through asyncio.to_thread on every turn, although the spec says a turn with no banner probes no file. Please reuse the builder's screen and run the probe only when a banner is present.

CI: two backend tests fail on both Linux and Windows: test_chat_runner_composition_contract.py::test_no_owner_captures_a_name_tests_rebind_on_the_runner (stop_recovery) and test_leaked_toolcall_notice.py::TestTheBoundaryScanRunsBeforeTheReset::test_every_mid_segment_text_reset_clears_the_raw_copy. Both touch modules this PR changes (stop_recovery.py, segment_flush.py). Frontend Tests (3) fails hiStyle.test.ts ("addresses the reader as तुम, never आप"), so please check the new Hindi strings. Coverage fails because of those shards.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at f401b5f. Besides the points in my last review, this repository now asks for structured matching instead of regex when deciding which text to hide or flag, and this head recognizes the provider banner with new patterns:

  • dashboard/chat_utils_turn_text.py: _PROVIDER_BUDGET_BANNER_RE, _PROVIDER_BUDGET_TOPIC_RE, _DIGIT_GROUPING_RE, _COUNT_RE, _JSON_UNICODE_ESCAPE_RE, and a per-count pattern built with re.escape(count).
  • acp/_dispatch_parts/tool_result_events.py: _RUN_STATUS_CODE.

These patterns decide whether reply text is hidden from the user, and the upheld GPT 6.1 findings already show real reply text that matches them. Requested:

  1. Recognize the banner by exact comparison: keep the provider's banner strings as fixed templates, split a candidate line into its fixed parts and its numeric fields, compare the fixed parts exactly, and check each numeric field with str.isdigit() (allowing only the grouping characters you list, explicitly).
  2. Read the run status code from the tool result's typed fields, if the result carries one, instead of matching text. If it does not, compare the known prefixes (exit status, exit code) exactly and parse the number with int().
  3. Better still, and in line with the follow-up named in the description (#18970), recognize the banner in the ACP layer from the provider's own event type, so no reply text has to be matched at all.

The two evidence-completeness points from my last review still apply. First Principles is red on this head, and CI is still running, so I have not read those results yet.

@iamwhatever

Copy link
Copy Markdown
Contributor

Fix round on f401b5fe72 (rebased on main 32a43272d, two commits, author kept) for @bolichen97's review of 2026-11-03:

  1. An earlier @prompt expansion now marks echo evidence incomplete. The chip row is tagged meta.kind: prompt_load, and an older untagged chip is matched by its text. Test: test/test_provider_budget_banner_prompt_chip_evidence.py.
  2. A tool result that drops any non-text block (resource, resource_link, audio, unknown) sets tool_output_truncated in both streaming parsers (tool_call_content_omits_text). Tests in test/test_tool_result_truncation_evidence.py.
  3. The give-up notice names no button. It asks the person to send the request again, in every locale. New screenshots are in the description.
  4. The image-path screen is now the prompt builder's own (kiro_crew.image_path_screen, shared with build_prompt_blocks). Dispatch records paths with no file probe, and they are probed off the loop only once a reply opens with the banner.
  5. CI reds fixed: post_turn_reply resolves through the runner, the banner-only branch clears assistant_text after its flush, and the Hindi string uses तुम forms.

GPT 6.1 and Opus 5.5 pass on this head. First Principles still blocks on scope ("Mechanism cost far exceeds the measured effect"). Its clear condition is a human accepting the ≤2/73 coverage, which matches the maintainer decision of 2026-11-02. I have not posted an override. If a maintainer agrees, this is the text to post:

/ai-review override first-principles f401b5fe7206ec5b72e2fb2a48130ad0ecc0cac3 Accepted as an interim fix per the 2026-11-02 scope decision; ACP-layer recognition for every consumer is #18970.

AI-assisted, posted on behalf of iamwhatever

@iamwhatever

iamwhatever commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor
  • Integral decimal evidence causes legitimate answers to be discarded (span=f8ab89034b60)

fixed in a58a733: _carries_count now reads an integral decimal (8154.0, 8154.00) as the same whole number, so {"remaining":8154.0} in a tool result keeps the answer visible. 8154.05, 8154.2 and 18154.0 still do not match. Test: test_an_integral_decimal_names_the_same_whole_number, with a control where 8154.5 leaves the tail stripped.

AI-assisted, posted on behalf of iamwhatever

@iamwhatever

Copy link
Copy Markdown
Contributor

@rubencu, every check and all AI reviews (GPT 6.1, Opus 5.5, Design, UX, First Principles) now pass on a58a733d. PR Readiness has one item left: your disposition comment 6183890234 (target=gpt head=48e4b0564e) has no span= ID, and the gate checks every writer's record on its own (_review_contract.py:1516).

Could you add a span to its title bullet? pr_findings.py --rounds prints no span for head 48e4b0564e, because that GPT review was replaced in place. The GPT span it prints for chat_utils_turn_text.py, where the word-to-digit finding sits, is f8ab89034b60. So the bullet would read:

- **Word-to-digit rewrite can match the provider-banner shape** (span=f8ab89034b60) accepted-residual

Once it is added, anyone can re-dispatch readiness on a58a733dde810419d9088d7bc874c5200e821a04. Thank you!

AI-assisted, posted on behalf of iamwhatever

@rubencu

rubencu commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@iamwhatever Thanks for tracking that down. I neutralized comment 6183890234 instead of adding a span. It now starts with "Disposition for an earlier head, kept for the record (no longer a ledger record):", the same edit the other legacy records on this PR got on 2026-10-29, so neither the readiness gate nor the adjudication ledger reads it as a record. Main's pr_status.py --disposition-gate now reports 0 violations on a58a733dde.

I left out span=f8ab89034b60 because at the head that record judged (48e4b0564e) the code was still in chat_utils.py, and f8ab89034b60 is the path-scoped identity of every GPT blocking finding in chat_utils_turn_text.py. An accepted-residual record carrying it could downgrade a later GPT block in that file. The record claimed no span, so no current-head finding loses a disposition.

The head conflicts with main again (src/kiro_crew/dashboard/chat_turn/turn_ledgers.py, test/test_transient_notice_parity.py), so readiness re-evaluates on the next push.

@bolichen97

Copy link
Copy Markdown
Contributor

Two things on a58a733, to save a round trip:

  1. The branch now conflicts with main. Could you rebase onto the latest main and resolve the conflicts?
  2. My change request at f401b5f still applies at this head. The banner is still recognized with regex: _PROVIDER_BUDGET_BANNER_RE, _PROVIDER_BUDGET_TOPIC_RE, _DIGIT_GROUPING_RE, _COUNT_RE and _JSON_UNICODE_ESCAPE_RE in dashboard/chat_utils_turn_text.py, the per-count re.compile(...) there, and _RUN_STATUS_CODE in acp/_dispatch_parts/tool_result_events.py. Please replace them as described there (fixed templates with exact comparison, typed status fields, or the ACP event type), and include that in the rebased head.

We will review it again once both are done and CI is green.

@rubencu

rubencu commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@iamwhatever Are you taking this round for bolichen97's request above, the rebase plus the exact-comparison matcher from their change request? We'll stay off the branch until you answer, so our pushes don't collide with yours. If you'd rather hand it back, say so and we'll do it in one push.

@iamwhatever

Copy link
Copy Markdown
Contributor

@rubencu yes, we took this round. It is pushed now at 71d93bb80; please stay off the branch until CI settles.

@bolichen97 both asks are done:

  1. Rebased onto latest main. Conflicts resolved in chat_turn/segment_flush.py (main's agent-started arm runs first, so an agent-started turn ends as it stands and never queues the banner continuation), chat_turn/turn_ledgers.py (both the BACKGROUND-start key and the recovery provenance are kept) and test_transient_notice_parity.py (main and this PR each add one notice token, so the pins move by one more).
  2. No regex is left in the banner path. _PROVIDER_BUDGET_BANNER_RE, _PROVIDER_BUDGET_TOPIC_RE, _DIGIT_GROUPING_RE, _COUNT_RE, _JSON_UNICODE_ESCAPE_RE, the per-count re.compile and _RUN_STATUS_CODE are gone. The banner is the fixed text You have <count> weighted tokens left, compared exactly, with the count checked as ASCII digits (plain, or comma groups of three). The topic gate compares whole words from a fixed set. The exit-code status compares the exit status / exit code prefixes exactly and checks one to three digits. Behaviour is unchanged: a scratch differential run of 400,000 random inputs gave the same result as the old patterns for every function. New test: test/test_provider_budget_banner_exact_match.py (exact strips, similar assistant text kept, whole-word topics, exit codes, escapes, and a check that none of these functions uses re).

AI-assisted, posted on behalf of iamwhatever

@iamwhatever

Copy link
Copy Markdown
Contributor

Fix round on ac6c3bb02 for the AI reviews of 1e7422052 (still two commits, author kept):

  • GPT F1, quadratic decode on a long backslash run: fixed. When no escape starts in a backslash run, the whole run is kept as written in one step, so _decode_json_unicode_escapes stays linear. Test: test_a_long_undecodable_backslash_run_decodes_in_linear_time (200,000 backslashes + uZZZZ).
  • GPT F2, non-object json result dropped the record: fixed at the shared helper. _json_result_holds_image returns False for a non-dict payload, so an array data is serialised as output again. Test: test_read_back_parser_keeps_a_non_object_json_result.
  • UX BLOCK, give-up notice after a shown answer: fixed. When the turn already showed visible text, it settles as landed and gets no give-up notice. The notice now appears only on a turn that showed nothing. The settle and capacity-gate tests now pin that.

AI-assisted, posted on behalf of iamwhatever

rubencu and others added 2 commits November 6, 2026 10:41
Some model backends emit their private "You have N weighted tokens
left" reminder as answer text. Observed transcripts show it as a whole
message or fused to the reply's first word with no separator, and it
was persisted, mirrored to channels and spoken as the answer.

Recognize the exact leading banner at a line break, end of text, or
glued to an uppercase letter, then classify authorship separately.
The checks that keep the text visible run first, on every turn kind:
a typed transient retry, an explicit capacity request from the
session's own human, and a banner whose phrase appears in a tool
result delivered this turn. Only then does a typed provider recovery
strip the prefix and keep the answer, and an ordinary turn strip a
banner-only tail after visible output; quoted, embedded and suffixed
banners stay visible. A banner-only turn recovers once on the same
conversation with a continuation tagged
RecoveryProvenance.PROVIDER_BUDGET_ARTIFACT, queued only while tool
approval is interactive and re-checked at drain whenever such an
entry is queued; the post-token transient retry carries
TRANSIENT_RETRY for the same text, so ownership never comes from text
equality.

The capacity gate reads only what the session's own human wrote: the
turn's own text unless it is runner- or app-authored, plus the steers
that human typed and the turn consumed. A steer's user-origin record
is kept past the steer RPC's acknowledgement, which arrives before the
consumption echo, and is released when the steer settles: by the
consumption settle, or by the turn-end requeue for a steer the turn
never consumed.

A banner-shaped answer may echo text the turn can see, so the strip runs
only when the evidence it reads is whole. A tool result is not whole
when the display bound cut it, when a Json item's stdout-only display
left out another output field (stderr, a nested object or list, a
number), or when it held an image block, bare or inside an MCP
envelope. A prompt image is unreadable evidence on the turn that sent
it and on every later turn, since the model's conversation keeps it.

What the model read as input this turn is count evidence whoever wrote
it: the prompt as sent (an @prompt body, an injected payload) and every
steer the turn consumed, a peer's included; only the topic gate stays
limited to the session's own human. The dispatch builder's display of
a pure-text MCP envelope drops structuredContent and any other
output-bearing sibling, so such a result is not whole, and a status
field beside stdout is set aside only while it reads as a run status.

An earlier turn's skill load sent its whole body to the model, so the
bodies its row recorded are echo evidence like the row's own text, and
a skill-load row that kept no readable body leaves the evidence
incomplete.

The prompt-image check joins the link-screen baseline as a refusing
site: it screens each candidate the way build_prompt_blocks does and
skips every link or junction it cannot vet, and what follows is only an
is_file() probe, so it holds no chain.

The echo evidence has one owner, chat_utils_turn_text.TurnEchoEvidence,
which the settle phase builds only once the banner's shape matched, so a
turn with no banner walks no rows and probes no file. The prompt as sent,
the consumed steers and the human's own words are kept within a count and
size bound there; a text left out marks the evidence incomplete and the
banner stays visible. When a typed recovery strips the banner and keeps
the answer after it, the banner is cut off the raw segment copy too, so
the flush saves the answer instead of the withheld notice.
@iamwhatever

iamwhatever commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Update on 782c64e89 (rebased on latest main, two commits):

  • Opus 5.5 BLOCK at segment_flush.py:1308: fixed. An agent-started turn has no prompt to continue, so the banner arm never ran for it. When such a turn already showed its answer and ended on a stripped banner tail, the recovering flag is now cleared, so the turn settles as landed (reply notification, budgets, success). Test: test_agent_started_turn_with_a_shown_answer_settles_as_landed, which fails without the fix.
  • The frontend parity pin moved to 66 tokens (main and this PR each add one).

Still blocked on a maintainer: First Principles now BLOCKs on the premise (recognise the banner as an ACP control_notice instead of in the dashboard). @bolichen97 accepted this PR on 2026-11-02 as an interim fix, with #18970 tracking the ACP-layer move. If that decision stands, this is the text to post:

/ai-review override first-principles 782c64e891c2b6141118f39a78b3ef8be375b74f symptom-level: accepted as the interim dashboard fix per the 2026-11-02 maintainer decision; ACP-layer recognition via control_notice is tracked in #18970.

AI-assisted, posted on behalf of iamwhatever

@iamwhatever

Copy link
Copy Markdown
Contributor

@rubencu handing this one back: we are off the branch, so your pushes will not collide with ours. Where it stands at 782c64e: bolichen97's change request is addressed (no regex left in the banner path; exact template, digit check, exit-status prefixes compared exactly), and the Opus BLOCK is fixed. The GPT 6.1 review on that head raised two new findings in the same echo-evidence matcher (#7626 (comment)): tool-call inputs (meta.input) are not part of the echo evidence, and a zero-padded or Indian-grouped count is not recognised as the same number. We stop after one fix round per area, so those two and the rebase (it now conflicts with main) are yours. Backend Tests (3.12, 4) and Coverage Gate were also red on that head.

@rubencu

rubencu commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 @iamwhatever thanks for handing this back. One design question before the next push.

The open GPT 6.1 finding (F1 in the review on 782c64e) is one more case of a wider class. The matcher deletes a banner-only reply when it cannot find the count in the evidence it reads, but Kiro Crew never sees everything the model read: the prompt's prefix context is not stored with the user row, and kiro-cli adds context of its own. More evidence narrows the risk but cannot remove it, and this PR has had many rounds of that.

Proposal: never delete a banner-only reply. Show it as a collapsed notice row instead. It would not be shown or stored as the answer and not delivered to channels, and the one continuation still runs. A legitimate reply that happens to match the banner then stays readable. The evidence code (count folding, escape decoding, the ACP truncation flags, the prompt-image probe) can then be removed, which makes this PR much smaller.

Does a collapsed notice row meet the goal ("not shown, stored or delivered as the assistant's answer")? If yes, the next push is the rebase plus that change. If not, I'll make the narrower fix instead: every recorded tool field counts as evidence, and a truncated field makes the evidence incomplete.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

drive-to-green PR claimed by drive-to-green pipeline fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants