Skip to content

fix(auto-research): settle a campaign whose loop stood down instead of reviving it - #21241

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/research-standdown-revival
Oct 26, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/research-standdown-revival

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Research Lab campaign whose worker keeps failing stops and says so, instead of running doomed cycles forever.

AutoNudge stands a loop down after too many failed cycles (consecutive_failures) or failed session starts (session_start_failures) in a row. For a Research Lab campaign that stand-down never holds. Within one watchdog poll (5 s) the loop is active again, the campaign stays running, and the next doomed cycle is armed. The user sees a campaign that is "running" but makes no progress and never reports why.

Why it matters

The stand-down exists to end exactly this silence: a worker whose session cannot start, or whose every cycle fails, keeps spending model turns (and, for unattended campaigns, auto-approved tool calls) with nothing to show for it. Every Research Lab campaign is exposed, because the watchdog re-arms every inactive campaign loop.

Not a goal

  • Changing the stand-down thresholds or the AutoNudge failure counters themselves.
  • Refusing revival of a stand-down inside AutoNudgeService.update. A stand-down is documented as re-armable by a directive (test_the_stand_down_is_re_armable); the defect is this caller re-arming it unasked.
  • Issue Radar, which has the same caller-side defect and is handled separately in fix(issue-radar): keep a crew's failure stand-down from being revived #20602.

What changed (motivation → approach → change)

Symptom → cause. campaign/watchdog.py _watchdog_loop diverts only an autonudge_stop tombstone into settlement. Every other inactive loop falls through to:

# Re-arm the autonudge loop if a prior app-disable deactivated it
if svc is not None and loop is not None and not loop.active:
    await svc.update(loop.id, active=True)

That branch is meant to undo _suspend_research_loops_while_disabled, but it also catches AutoNudge's own failure stand-downs. AutoNudgeService.update treats any inactive→active flip as a fresh run and zeroes consecutive_failed_cycles and consecutive_start_failures (autonudge_service/mutations.py), so the five-strike stand-down is erased on every poll.

The First Principles review of #20602 flagged this: "The Research Lab watchdog revives the same failure stand-downs the same way and stays unfixed." #20602 deliberately stays Issue-Radar-only. This PR is the Research Lab half.

Change.

  • agent_mode.FAILURE_STANDDOWN_REASONS names the two stand-down reasons.
  • The watchdog routes a loop stopped for either reason through the same path as the stop tombstone (_settle_campaign_from_watchdog). That path already waits for a running worker turn, defers one poll for a freshly resumed run, and removes the spent loop once the campaign is terminal.
  • _stalled_campaign_verdict returns FAILED with "The research worker kept failing and stood down. Resume to try again." for a stand-down. It ranks below a verified finding (still COMPLETE) and above a stale worker_done.json marker, which must not turn repeated failure into a "worker ended the loop" STOPPED.
  • _prepare_loop_launch also removes a leftover stand-down record before a resumed run is published. Without that, if settlement failed to remove the loop, the watchdog would settle the new run on the previous run's evidence. This mirrors the existing tombstone cleanup.

Resume is unchanged: from FAILED it relaunches through _launch_loop, which arms a fresh loop with fresh streaks.

Left to the re-arm on purpose. approval_stalled, structural_terminal and model_refused still go through the watchdog's re-arm. None of them has a streak that the inactive→active flip zeroes, so reviving one does not erase its bound: the next cycle that meets the same condition stops the loop again. Their remedy is also a person (an approval, a corrected goal or context), not a worker that keeps failing, so settling the campaign FAILED would misreport them. This answers the First Principles CONCERNS on 973753b3c1.

Docs (second commit, 371df3cc25). The _stalled_campaign_verdict docstring now has the stand-down bullet: FAILED, ranked below a verified finding and above the tombstone and the marker. The Research Lab watchdog section of docs/system-specs/modules/learn-cron-dashboard.md now covers the stand-down settlement, the stand-down records removed at launch, and why the three reasons above stay with the re-arm.

Backwards compatibility

Compatible: no config key, API field, route or stored value changes shape. A campaign whose loop stood down now ends FAILED instead of staying running, and Resume keeps working.

Removes nothing: the only record removed is a stood-down research loop of a campaign that has just been settled terminal, the same removal the stop tombstone already gets, and Resume re-arms a new one.

Tests

test/test_auto_research_watchdog.py::TestWatchdogFailureStandDown (the watchdog tests moved there when #21269 split test_auto_research.py):

  • test_stand_down_settles_campaign_instead_of_reviving[CONSECUTIVE_FAILURE_REASON|SESSION_START_FAILURE_REASON] uses a real AutoNudgeService: arm a research loop, set both streaks, stand it down with the reason, and run two watchdog polls. It asserts the campaign is FAILED with the stand-down message, the loop is removed rather than revived, and the streak was never reset.
  • test_stand_down_with_done_marker_is_still_a_failure: a stale worker_done.json plus a readable finding does not turn a stand-down into STOPPED.
  • test_launch_clears_a_stand_down_record: a resumed run removes the previous run's stand-down record.

Negative control: with only campaign/watchdog.py and campaign/agent_mode.py reverted to origin/main, all 4 tests fail. The two real-service cases fail with assert 'running' == 'failed': the loop was revived and the campaign kept running.

Focused run (Windows, py3.13): the 4 new tests pass. Across test_auto_research*.py plus the in-package auto_research tests: 544 passed. The 2 failures and 2 errors also fail with this change reverted, from a local cp950 locale decode and a missing event_loop fixture in this environment, so they are unrelated. flake8 and isort are clean on the touched files.

The watchdog tests pace the loop through one helper, _drive_watchdog_polls. It is the single place in the file that replaces the loop's asyncio.sleep, and the two TestWatchdogStopTombstone tests use it too, so test_flake_pattern_ratchet.py counts no new shape (60 passed).

On the rebased head (973753b3c1, one commit on da762d16d6), test_auto_research_watchdog.py, test_auto_research_worker_loop.py and test_auto_research.py give 147 passed and 1 error. The error is test_cancellation_waits_for_persisted_loop_removal, which fails identically on a clean main checkout here (fixture 'event_loop' not found, a local pytest-asyncio mismatch).

On 371df3cc25 (docs and a docstring only): black, flake8, comment-history and brand gates are clean, and test_auto_research_watchdog.py gives 18 passed and the same local-only event_loop error.

Manual verification

N/A — unit coverage sufficient: the regression test drives the real AutoNudgeService and the real watchdog loop over a real campaign DB, which is the whole path involved. No UI or external service is touched.

Related Issues

Closes #21254

Part of the stand-down revival class addressed for Issue Radar in #20602, and requested by that PR's First Principles review.

Pattern harvest

Rule candidate: review-prompt
Pattern: a reconciler that re-arms "any inactive loop" with update(active=True) must first exclude stop reasons it does not own. A revival resets failure streaks, so an unconditional re-arm erases another component's stand-down.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 16, 2026 13:52
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 76c005c9aa418641fd1b349d74c977c34ed86ecb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 76c005c9aa418641fd1b349d74c977c34ed86ecb via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the stand-down now goes through the existing tombstone settlement path, and the PR keeps the re-armable contract in AutoNudgeService.update intact.

Notes

[DESIGN-REVIEWED] 76c005c

Description read

[DESCRIPTION-READ] 4246f7de0867cc7f45304c80810901e36d7efeaf7714c52e06bd2296c7e60782

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 76c005c9aa418641fd1b349d74c977c34ed86ecb via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix works at the mechanism level. It does this by adding names to a list of reasons the re-arm skips, and five other system stop reasons are still revived.

Notes

  • Item 1 — symptom-level: the re-arm exists only to undo _suspend_research_loops_while_disabled, which writes the default MANUAL_STOP_REASON through svc.update(loop.id, active=False). I counted _TERMINAL_BOUND_REASONS (autonudge_service/model.py:261, 7 members): this PR excludes 2 and leaves 5 to the re-arm. The description argues for keeping 3 of them (approval_stalled, structural_terminal, model_refused) and says nothing about cycle_cap or runtime_budget. Subtraction: replace the two-reason exclusion with a single check that only re-arms a loop whose stopped_reason is MANUAL_STOP_REASON.

What this change ships

Inventory (7 items) — 6 justified

Intent: a Research Lab campaign whose worker keeps failing stops and reports why, instead of re-arming forever. FIX.

  1. Watchdog no longer revives a loop that stood down after repeated failures — symptom-level
  2. A stood-down campaign now ends FAILED with "Resume to try again" — justified
  3. The spent stood-down loop is removed during settlement — justified
  4. A stale worker_done.json no longer turns a stand-down into STOPPED — justified
  5. Resume removes the previous run's stand-down record — justified
  6. New FAILURE_STANDDOWN_REASONS constant, used at 3 sites — justified
  7. The pacing sleep in two existing tombstone tests now goes through a shared helper — justified

[FIRST-PRINCIPLES-REVIEWED] 76c005c

Description read

[DESCRIPTION-READ] 0ecca8316f5900b9a9d17d76d7d9603ce5294edc7c18bfd5332d5b0b1006bb15

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 76c005c9aa418641fd1b349d74c977c34ed86ecb via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 76c005c

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 76c005c9aa418641fd1b349d74c977c34ed86ecb via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 76c005c

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @leonlaiyc, this is a clear write-up and the fix is at the right place.

The fix is correct for #21254 on 1764e45: a loop stood down for consecutive_failures or session_start_failures now goes down the settle path instead of reaching the app-disable re-arm, and the campaign ends FAILED. Locally, the 4 TestWatchdogFailureStandDown tests fail with watchdog.py and agent_mode.py reverted (the two real-service cases with 'running' == 'failed') and pass with the change, and test_auto_research*.py plus the in-package auto_research tests pass (648). The app-disable re-arm still works, because _suspend_research_loops_while_disabled only touches active loops, so a stood-down loop keeps its reason through a disable and re-enable.

One change is needed before this can be approved:

  1. Backend Tests (3.12, 2) is red on this head, and it reproduces locally. test/test_flake_pattern_ratchet.py::test_no_changed_file_gains_a_flaky_shape fails with test/test_auto_research.py: K3-stdlib-clock-patch 3 > 2. The new monkeypatch.setattr(h.asyncio, "sleep", _run_two_polls) in test_stand_down_settles_campaign_instead_of_reviving is a third patch of asyncio.sleep in that file. Either share one poll-driving helper with the two existing tests that patch it the same way, or carry a reason on that line as the ratchet asks, for example # flake-ok: stands in for the watchdog poll interval. With that comment added, the ratchet and the 4 new tests pass locally (62).

Non-blocking, for a follow-up rather than this PR: the First Principles lane's point holds in the code. The re-arm still revives every inactive research loop whose reason is not on the new list, including model_refused and structural_terminal, which the dashboard fire path sets on message loops. A revival advances the loop's generation, so the stop verdict no longer applies and the next cycle is sent again. Re-arming only on the manual reason that the suspend path writes would cover them, but it would also send cycle_cap, runtime_budget and approval_stalled down the settle path, which needs verdicts of its own, so it is a separate design change. A sentence in the description saying those reasons are left out on purpose would settle the lane.

I approved the waiting fork workflow runs for this head. Several backend shards were still running when I posted this. Happy to re-review when the head moves.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 17, 2026
cruisercohen
cruisercohen previously approved these changes Oct 19, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @leonlaiyc. I re-reviewed 69000bd, and the change I asked for on 1764e45 is done. _drive_watchdog_polls is now the one place in test_auto_research.py that patches asyncio.sleep, and the two existing tests use it with the same behaviour as before, including the worker-finish step on the second poll. Locally on this head:

  • test_no_changed_file_gains_a_flaky_shape passes. With the previous test file it fails with the same 3 > 2 count as CI.
  • test_auto_research.py passes (277).
  • The 4 TestWatchdogFailureStandDown tests still fail with watchdog.py and agent_mode.py reverted to the base, and pass with the change.

The branch now conflicts with main. Main split test_auto_research.py in #21269 (merged after this branch was cut), so the new tests and the helper need to move into whichever file now holds these classes. Please rebase onto current main. The red lanes on this head are not from the diff: Brand Name Gate and SAST flag test_dev_fleet_app.py, which this PR does not touch, and the backend shards were skipped behind Fast Gate, so they first run on the rebased head.

The description sentence on the reasons the re-arm leaves out is still optional. A push dismisses this approval, and I will re-approve on the rebased head once CI is green.

@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Thanks @cruisercohen. Rebased onto da762d16d6 as one commit, 973753b3c1. Since #21269 split test_auto_research.py, TestWatchdogFailureStandDown and _drive_watchdog_polls now live in test_auto_research_watchdog.py, beside TestWatchdogStopTombstone, whose two tests use the helper as before (the worker-finish step on poll 2 included). Main's per-test flake-ok markers on the two inline sleep patches went away with them, and the helper's single patch carries one. test_flake_pattern_ratchet.py: 60 passed. Source changes are identical to the head you approved.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Oct 19, 2026
cruisercohen
cruisercohen previously approved these changes Oct 19, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after the squash and rebase; the change is unchanged and its tests now live in the split file.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES at 973753b.

The code fix is right. A loop that stood down for consecutive_failures or session_start_failures now settles the campaign FAILED instead of being re-armed (watchdog.py#L222). The documentation no longer matches it, though, and the review lanes flagged this on this head with no reply:

  1. The _stalled_campaign_verdict docstring (L204-L215) still says a marker plus a readable finding gives STOPPED and that everything else is FAILED 'unchanged'. Add a bullet saying a failure stand-down gives FAILED, ranks above the tombstone and the marker, and ranks below a verified finding.
  2. The owning spec (learn-cron-dashboard.md, Research Lab watchdog) does not mention the stand-down settlement or the stand-down records removed at launch. Update it in this PR.
  3. The First Principles CONCERNS on this head is unanswered. Add a sentence to the description saying that approval_stalled, structural_terminal and model_refused are left to the re-arm on purpose, or post a disposition.

— crew-pr-reviewer (r112)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 19, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Addressed r112 at 371df3cc25 (docs only, no code change):

  1. _stalled_campaign_verdict docstring: new bullet saying a consecutive_failures / session_start_failures stand-down gives FAILED, ranks below a verified finding, and ranks above the tombstone and the worker_done.json marker.
  2. learn-cron-dashboard.md (Research Lab watchdog): documents the stand-down settlement, and that launch removes stand-down records along with the tombstone and marker.
  3. First Principles CONCERNS: the description now says why approval_stalled, structural_terminal and model_refused stay with the re-arm. None has a streak that the inactive→active flip resets, so a revival re-stops on the next cycle that meets the same condition. Their remedy is also a person, not a failing worker, so FAILED would misreport them.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving on 371df3c: the new commit is the docstring, spec and description changes r112 asked for, and they match the code; the fix commit is unchanged from 973753b. The only backend reds (test_chat_follow_default_slot_kinds, and Coverage Gate behind it) also fail on main at 6432cad.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
leonlaiyc and others added 2 commits October 26, 2026 10:25
…f reviving it

AutoNudge stands a loop down after too many failed cycles or session
starts in a row. The Research Lab watchdog re-armed every inactive
campaign loop that was not an autonudge_stop tombstone, and a revival
zeroes both failure streaks, so a campaign whose worker kept failing ran
doomed cycles forever and the stand-down never held.

Treat consecutive_failures and session_start_failures like the stop
tombstone: settle the campaign as FAILED (Resume to retry) and remove the
spent loop. A resumed run also clears a leftover stand-down record so the
watchdog cannot settle the new run on the old run's evidence.

The watchdog tests pace the loop through one helper,
_drive_watchdog_polls, the single place that replaces its sleep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stalled-campaign verdict docstring and the Research Lab watchdog
spec still described a stand-down as revivable. State that a
consecutive_failures or session_start_failures stand-down settles the
campaign FAILED, ranking below a verified finding and above the stop
tombstone and the worker_done marker; that launch clears stand-down
records with the tombstone; and why approval_stalled,
structural_terminal and model_refused stay with the re-arm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever
iamwhatever force-pushed the claude/research-standdown-revival branch from 371df3c to 76c005c Compare October 26, 2026 10:25
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto current main (9bf7e31) as 76c005c, with no conflicts and no code change. The earlier backend reds came from test_chat_follow_default_slot_kinds, which main fixed in #22433. They need a rebase because a rerun reuses the old merge commit.

The three asks from the r112 CHANGES_REQUESTED review on 973753b are already done in 371df3c (now 76c005c):

  1. The _stalled_campaign_verdict docstring has the stand-down bullet (FAILED, below a verified finding, above the tombstone and the marker).
  2. The Research Lab watchdog section of learn-cron-dashboard.md covers the stand-down settlement and the records removed at launch.
  3. The description says why approval_stalled, structural_terminal and model_refused stay with the re-arm.

Locally on this head, test_auto_research_watchdog.py and test_flake_pattern_ratchet.py give 79 passed. The push dismissed the approval, so this needs a re-approval, and the stale r112 review needs a dismissal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 76c005c9aa418641fd1b349d74c977c34ed86ecb.

A loop that stood down for consecutive_failures or session_start_failures now settles the campaign through the existing settlement path instead of being re-armed (watchdog.py#L424-L427). The campaign ends FAILED with a Resume hint, below a verified finding and above the tombstone and marker (L227-L233). The docstring, spec and description now cover everything my r112 review asked for.

I checked the First Principles note on cycle_cap and runtime_budget. Research loops are added with no runtime budget, so runtime_budget cannot occur. The watchdog settles the campaign COMPLETE once the cycle count reaches max_cycles (L511), so leaving cycle_cap with the re-arm does not bring back the silent loop. Every lane is clean and every check on this head is green.

— crew-pr-reviewer (r127)

@iamwhatever
iamwhatever merged commit 1861c3e into kirodotdev:main Oct 26, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Research Lab watchdog re-arms a campaign loop that AutoNudge stood down for repeated failures

3 participants