Skip to content

refactor(dashboard): give the poisoned-conversation escalation one owner - #20015

Merged
cruisercohen merged 2 commits into
mainfrom
fix/2295-poisoned-escalation-helper
Oct 14, 2026
Merged

cruisercohen merged 2 commits into
mainfrom
fix/2295-poisoned-escalation-helper

Conversation

@cruisercohen

@cruisercohen cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The poisoned-conversation escalation's cross-turn state has one owner that _run_chat calls into, instead of being written inline at six places in the turn.

The escalation (and the image-history and thinking-binding recoveries that share its discard) spreads its state machine across the pre-stream exhaustion streak, the discard one-shot, the turn's thinking flag, the canary's Stop snapshot and the turn's discard flag. Their transitions are interleaved through _run_chat's error handler and finally. Design Review flagged this on #2239: each transition is sound, but the fields deserve one owner.

Why it matters

Every recovery that joins this escalation (the thinking-binding rebuild from #19852 is the latest) adds more inline writes to the same two cells. A change to one transition then has to be checked against every other site by hand, which is how a streak that should break, or a one-shot that should stay spent, gets missed.

Not a goal

  • Changing any behaviour of the escalation, the canary, or the image-history and thinking-binding recoveries.
  • Consolidating the neighbouring recovery ladders (transient 5xx, post-token, stale, empty-response, compaction-fail). The issue names them as a later candidate for the same pattern.
  • Changing which recovery-replay cancellations refund a one-shot, or when.

What changed (motivation → approach → change)

Symptom: the streak and one-shot are mutated inline in the landed-turn block, the thinking-binding rebuild, the image-history recovery, the terminal branch (count, probe check, canary discard) and the finally. Approach: one helper owns every transition; _run_chat reports facts and asks questions.

  • New dashboard/poisoned_escalation.py: PoisonedConversationEscalation with the seams the issue proposes: record_terminal_failure(emitted, thought, transient), record_activity(emitted, thought), record_landed_turn(), should_probe(), arm_discard(clear_streak=...) and refund_one_shot(). POISONED_SESSION_CYCLES lives there and the runner re-exports it, so its import path keeps working.
  • _run_chat builds one _escalation per turn and calls those seams at all six sites. The canary probe itself (run_bg_oneliner) stays in _run_chat, where a guard test reads it.
  • The two cells stay _ChatSlot fields, and the helper reads and writes them. The recovery-replay ledger's refund goes through the helper too: the image-history and thinking-binding policies set refunds_discard_one_shot instead of naming the cell in their (attribute, value) refund tables, and the runner's _rearm_allowances calls refund_one_shot() at the same two seams (the drain and the consume seam). Ownership is enforced: a source scan fails if anything in the package other than the helper and the slot's __slots__ and constructor writes either cell or names one in a string.
  • The turn's discard flag stays a turn local, because that turn's own finally consumes it. So do the two Stop counters the canary reads before the probe and compares after it, unchanged from main. The helper holds only what outlives the turn.
  • Spec: the dashboard module list in docs/system-specs/modules/learn-cron-dashboard.md gains the helper.

Backwards compatibility

Compatible: the same cells, the same transitions in the same order, and POISONED_SESSION_CYCLES still resolves on the runner.

Removes nothing: no field, key, path or public name is removed; the six inline writes become calls to the helper that performs them.

Tests

  • New test/test_poisoned_escalation.py:
    • the state machine: a silent transient failure extends the streak; output, thinking or a non-transient error breaks it; a probe is due only at the threshold with the one-shot unspent; activity breaks the streak without re-arming; a landed turn resets and re-arms; a discard spends the one-shot and clears the streak only when asked;
    • a refund re-arms the one-shot and leaves the streak; the ledger's image-history and thinking-binding policies ask for it and the runner's re-arm performs it through the helper;
    • the one-owner pin: an AST scan of the package finds no write to either cell, and no string naming one, outside the helper and _ChatSlot's __slots__ and __init__; a parametrized check shows the scan sees assignment, augmented, annotated, tuple, chained, del, literal setattr, a (name, value) refund table and a getattr by name.
  • Red without the fix: with main's runner restored, the one-owner pin fails and lists the 11 inline writes; on main the whole file fails at import. With the first commit's ledger restored, the pin lists the four by-name refund entries in the ledger's policy table.
  • Behaviour unchanged: these all pass unchanged: TestRunChatTransientRetry in test_dashboard_chat.py (poisoned discard, canary, Stop-during-canary, one-shot, image-history, thinking-binding), test_recovery_replays.py, test_chat_requeue_resolved_stop.py, test_chat_runner_composition_contract.py and test_start_priority.py (502 passed). A wider related set of 96 files gave 10213 passed. The only failures were 13 environment-dependent tests that fail identically on main in the same local environment. On the second commit (the ledger refund through the helper), test_recovery_replays.py, whose drain and consume tests assert which one-shot each family's cancellation refunds, passes unchanged, with the suites above and the session-not-found, model-access, promise-only, refusal-replay, cross-surface Stop and canary probe suites (1522 passed). With the canary's Stop counters back as turn locals, TestRunChatTransientRetry (Stop-during-canary included) and the escalation, recovery-replay, requeue, composition-contract and start-priority suites pass unchanged (504 passed).

Manual verification

N/A: a refactor with no behaviour change. The existing _run_chat tests drive every escalation path end to end.

Related Issues

Fixes #2295

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that ReplayPolicy now has two ways to refund: attribute tables plus a refunds_discard_one_shot flag for one cell. Is that special case acceptable?

[DESIGN-REVIEWED] 4f01cd8

Description read

[DESCRIPTION-READ] e216f38cd9cd157590e42fabdb8d9bf86223a7feedf99b1263b5bd5c2afabeb6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

refunds_discard_one_shot is a second refund channel next to the ledger's existing (attr, value) refund tables. It exists only to satisfy the PR's own new string-name pin.

Not justified as shipped

  • Item 3 — duplicate of recovery_replays.ReplayPolicy.drain_refunds/consume_refunds (and chat_turn/recovery.py:_rearm_allowances): the Goal is about "six places in the turn". The ledger is not one of them. The four removed ("_poisoned_reset_used", False) entries were already the generic refund mechanism that the other three families still use (5 table entries, 4 _rearm_allowances call sites). The new bool has 2 policy users and 2 call sites, and its only stated need is "no table here names that cell".
    Subtraction: delete the refunds_discard_one_shot field and the discard_one_shot kwarg, restore the four table entries, and narrow the ownership scan to attribute stores.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 4 — zero consumers: chat_runner's POISONED_SESSION_CYCLES # noqa: F401 re-export is read only by test_poisoned_escalation.py and the name list in test_chat_runner_composition_contract.py. Grep for POISONED_SESSION_CYCLES finds no production importer.

What this change ships

Inventory (5 items) — 3 justified

Intent: give the poisoned-conversation escalation's cross-turn state one owner (#2295). This is an ADDITION (refactor) with no behaviour change.

  1. Six inline streak/one-shot writes in _run_chat become calls to the new helper — justified
  2. A new PoisonedConversationEscalation module, with the threshold constant moved into it — justified
  3. The ledger's image/thinking refunds move from refund tables to a policy bool plus a call to the owner — duplicate of ReplayPolicy.*_refunds
  4. The threshold is still importable from chat_runner — zero consumers
  5. A test that fails if any code outside the owner writes or names either cell, plus the spec entry for the module — justified

[FIRST-PRINCIPLES-REVIEWED] 4f01cd8

Description read

[DESCRIPTION-READ] 675598dc4639f1482bd7b218fa3df1f3f484e2807373c623bfed3ff3b598fefc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4f01cd8

Verdict parsed from the review's SHA-scoped output markers for commit 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4f01cd8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4f01cd876b90c7d6fde87d4aa6d6ca4dd767de2b: <one-sentence reason>

@cruisercohen
cruisercohen marked this pull request as ready for review October 14, 2026 00:42
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 14, 2026 00:42
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at f2dda38. This refactors the path that discards native conversation state. That data-removal path remains reserved for a human maintainer. Please answer the First Principles question about the second writer. The replay ledger still refunds the one-shot by attribute name. The ownership claim and AST test should describe that exception honestly. Alternatively, route the refund through the named owner. I read #2295: neighbouring ladders are possible future candidates. I am not requiring new issues merely for that optional extension. Source was reviewed; no tests or state-discard operations were executed. — crew-pr-reviewer (r99)

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 14, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Thanks. Point by point, on cfde8af (one commit on top of f2dda38):

  1. The second writer (First Principles "check before merge"). It was _rearm_allowances in the runner's recovery owner (dashboard/chat_turn/recovery.py). The ledger's image-history and thinking-binding policies fed it their (attribute, value) refund tables. It exists because both recoveries spend the shared one-shot when they discard the conversation and queue a replay. A Stop, a newer message or a rebind can cancel that replay before it runs, at the queue drain or at the turn's consume seam. When that happens, the turn the one-shot was spent for never ran, so the ledger hands the one-shot back. The discard itself is not undone, because a cold start on the next turn is the safe direction. The refund is a transition of the one-shot, so it should go through the owner. Leaving it outside made the single-owner claim untrue.

  2. The ledger refunds by attribute name. I took the routing option rather than the reword. Fixed in cfde8af:

    • dashboard/poisoned_escalation.py: a new seam, refund_one_shot(), re-arms the one-shot and leaves the streak alone. It makes the same write the table did.
    • dashboard/recovery_replays.py: the two policies drop their _poisoned_reset_used refund entries and set a new refunds_discard_one_shot flag instead.
    • dashboard/chat_turn/recovery.py: when the flag is set, _rearm_allowances calls the owner, at the drain and the consume seam only, as before. The swept-entry path still refunds nothing for these families. It reads PoisonedConversationEscalation from the runner's globals, which already import it, so no new runtime import is added.
    • No behaviour change: test_recovery_replays.py's drain and consume tests pass unchanged. They assert which one-shot each family's cancellation refunds.
  3. The ownership claim and the AST test. Both are now true as written, and the pin covers more. Besides attribute stores, test_poisoned_escalation.py now flags any string constant that names a cell, outside the helper and _ChatSlot's __slots__ and __init__. A string like that is how a by-name refund table reaches a cell. The parametrized check gains two spellings: a (name, value) refund table and a getattr by name. Run against f2dda38, the pin lists recovery_replays.py:206/207/216/217, and the two new refund tests fail there too. The module docstring, the slot comment, the spec's module list and the PR description now match.

  4. Neighbouring ladders from Extract poisoned-conversation escalation state machine into a slot-owned helper #2295. Agreed. They stay out of this PR, and no issues were opened for them.

  5. Data-removal path. Understood: this PR needs a human maintainer's approval, whatever the lanes say. Auto-merge is not armed and will not be.

Tests run on cfde8af:

  • the poisoned-escalation, recovery-replay, requeue, composition-contract and start-priority suites: 461 passed;
  • test_dashboard_chat.py plus the session-not-found, model-access, promise-only, refusal-replay, cross-surface Stop and canary-probe suites: 1061 passed;
  • black, isort, flake8 and mypy: clean on the touched files.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
The escalation's cross-turn state (the pre-stream exhaustion streak and
the discard one-shot) was written inline at six places in _run_chat,
interleaved with the canary's Stop snapshot and the turn's discard flag.
PoisonedConversationEscalation now owns every transition: _run_chat
records terminal failures, activity and landed turns, asks whether a
probe is due, takes the canary's Stop fence and arms a discard through
it. The two cells stay slot fields so the replay ledger keeps re-arming
the one-shot by name. Behaviour is unchanged.

A source scan pins that only the escalation and the slot constructor
write either cell.
@cruisercohen
cruisercohen force-pushed the fix/2295-poisoned-escalation-helper branch from cfde8af to 5c612dd Compare October 14, 2026 10:53
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Rebased onto main 641b42a to pick up #20212 (the ChatEmbed optimistic-send test fix); no code change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 5c612dd.

Our r99 ask is resolved. The replay ledger no longer refunds the one-shot by attribute name (recovery_replays.py#L208-L222). The refund goes through refund_one_shot(), and the AST pin now covers by-name strings. I compared each seam with the inline writes it replaces and found no behaviour change. should_probe() dropping the explicit exhausted check is equivalent, because a non-counting failure resets the streak to zero first. CI is green.

Two items keep this on HOLD:

  • This PR changes the logic that decides when a session's native conversation is discarded, a data-removal path. A human maintainer has to approve it.
  • The current-head First Principles CONCERNS has no reply. It calls ProbeFence/begin_probe oversized against a Goal about cross-turn state: they wrap two turn-local Stop counters for a single caller. To clear it, remove the fence and restore the two locals, or post a disposition explaining why it belongs here. Changing the Goal to cover it needs a human.

— crew-pr-reviewer (r100)

…h its owner

The image-history and thinking-binding replay policies re-armed the
poisoned-conversation one-shot by slot attribute name, so the cell kept a
second writer outside PoisonedConversationEscalation. The policies now ask
for the refund (refunds_discard_one_shot) and the runner's re-arm calls the
owner's new refund_one_shot seam at the same two seams (drain and consume).

The one-owner pin now also flags any string naming a cell outside the
owner and the slot's __slots__/__init__, which is how a by-name refund
table reaches a cell; it lists the four old refund entries on the previous
head.

The canary's two Stop counters go back to being locals of the turn, read
before the probe and compared after it as on main: they never outlive the
turn, so they are not the escalation's state. ProbeFence, begin_probe and
their test are removed. No behaviour change.
@cruisercohen
cruisercohen force-pushed the fix/2295-poisoned-escalation-helper branch from 5c612dd to 4f01cd8 Compare October 14, 2026 16:22
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 14, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Thanks @iamwhatever, for r100 and for checking each seam against the inline writes it replaces.

Resolved, as you confirmed on 5c612dd: the replay ledger refunds the one-shot through refund_one_shot(), and the one-owner AST pin also flags any string that names a cell.

First Principles CONCERNS (ProbeFence / begin_probe oversized). I took the subtraction in 4f01cd8, which is still two commits, with the change folded into the second:

  • dashboard/poisoned_escalation.py: ProbeFence and begin_probe are removed. The docstring's step 3 now says the Stop counters are locals of the turn.
  • dashboard/chat_runner.py: the two _stop_gen_before_canary locals and the inline != check are back. That hunk now matches main line for line.
  • test/test_poisoned_escalation.py: test_the_probe_fence_sees_a_stop_on_either_counter is removed. The spec's module list and the PR description no longer mention the fence.
  • No behaviour change, so no new red test. The canary's Stop veto is still covered by test_stop_during_canary_vetoes_discard_and_requeue, which passes unchanged.

Tests on 4f01cd8:

  • the escalation, recovery-replay, requeue, composition-contract and start-priority suites plus TestRunChatTransientRetry: 504 passed;
  • test_dashboard_chat.py with the session-not-found, model-access, promise-only, refusal-replay and cross-surface Stop suites: 1033 passed. The 12 non-passes were all in test_session_not_found_surfaces.py and were all the same local ModuleNotFoundError: markdown_it;
  • black, isort, flake8, mypy and the comment-history gate: clean.

With that, nothing else from r100 is open on 4f01cd8. Because this PR changes the data-removal path, it waits for a human maintainer's approval. Auto-merge stays off.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 14, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at 4f01cd8; no behavior change.

@cruisercohen
cruisercohen enabled auto-merge (squash) October 14, 2026 18:23
@cruisercohen
cruisercohen merged commit 3899beb into main Oct 14, 2026
104 of 112 checks passed
@cruisercohen
cruisercohen deleted the fix/2295-poisoned-escalation-helper branch October 14, 2026 18:25
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract poisoned-conversation escalation state machine into a slot-owned helper

3 participants