Repository navigation
fix(auto-research): settle a campaign whose loop stood down instead of reviving it - #21241
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the stand-down now goes through the existing tombstone settlement path, and the PR keeps the re-armable contract in Notes
[DESIGN-REVIEWED] 76c005c Description read[DESCRIPTION-READ] 4246f7de0867cc7f45304c80810901e36d7efeaf7714c52e06bd2296c7e60782 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix works at the mechanism level. It does this by adding names to a list of reasons the re-arm skips, and five other system stop reasons are still revived. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: a Research Lab campaign whose worker keeps failing stops and reports why, instead of re-arming forever. FIX.
[FIRST-PRINCIPLES-REVIEWED] 76c005c Description read[DESCRIPTION-READ] 0ecca8316f5900b9a9d17d76d7d9603ce5294edc7c18bfd5332d5b0b1006bb15 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
cruisercohen
left a comment
There was a problem hiding this comment.
Thanks @leonlaiyc, this is a clear write-up and the fix is at the right place.
The fix is correct for #21254 on 1764e45: a loop stood down for consecutive_failures or session_start_failures now goes down the settle path instead of reaching the app-disable re-arm, and the campaign ends FAILED. Locally, the 4 TestWatchdogFailureStandDown tests fail with watchdog.py and agent_mode.py reverted (the two real-service cases with 'running' == 'failed') and pass with the change, and test_auto_research*.py plus the in-package auto_research tests pass (648). The app-disable re-arm still works, because _suspend_research_loops_while_disabled only touches active loops, so a stood-down loop keeps its reason through a disable and re-enable.
One change is needed before this can be approved:
Backend Tests (3.12, 2)is red on this head, and it reproduces locally.test/test_flake_pattern_ratchet.py::test_no_changed_file_gains_a_flaky_shapefails withtest/test_auto_research.py: K3-stdlib-clock-patch 3 > 2. The newmonkeypatch.setattr(h.asyncio, "sleep", _run_two_polls)intest_stand_down_settles_campaign_instead_of_revivingis a third patch ofasyncio.sleepin that file. Either share one poll-driving helper with the two existing tests that patch it the same way, or carry a reason on that line as the ratchet asks, for example# flake-ok: stands in for the watchdog poll interval. With that comment added, the ratchet and the 4 new tests pass locally (62).
Non-blocking, for a follow-up rather than this PR: the First Principles lane's point holds in the code. The re-arm still revives every inactive research loop whose reason is not on the new list, including model_refused and structural_terminal, which the dashboard fire path sets on message loops. A revival advances the loop's generation, so the stop verdict no longer applies and the next cycle is sent again. Re-arming only on the manual reason that the suspend path writes would cover them, but it would also send cycle_cap, runtime_budget and approval_stalled down the settle path, which needs verdicts of its own, so it is a separate design change. A sentence in the description saying those reasons are left out on purpose would settle the lane.
I approved the waiting fork workflow runs for this head. Several backend shards were still running when I posted this. Happy to re-review when the head moves.
cruisercohen
left a comment
There was a problem hiding this comment.
Thanks @leonlaiyc. I re-reviewed 69000bd, and the change I asked for on 1764e45 is done. _drive_watchdog_polls is now the one place in test_auto_research.py that patches asyncio.sleep, and the two existing tests use it with the same behaviour as before, including the worker-finish step on the second poll. Locally on this head:
test_no_changed_file_gains_a_flaky_shapepasses. With the previous test file it fails with the same3 > 2count as CI.test_auto_research.pypasses (277).- The 4
TestWatchdogFailureStandDowntests still fail withwatchdog.pyandagent_mode.pyreverted to the base, and pass with the change.
The branch now conflicts with main. Main split test_auto_research.py in #21269 (merged after this branch was cut), so the new tests and the helper need to move into whichever file now holds these classes. Please rebase onto current main. The red lanes on this head are not from the diff: Brand Name Gate and SAST flag test_dev_fleet_app.py, which this PR does not touch, and the backend shards were skipped behind Fast Gate, so they first run on the rebased head.
The description sentence on the reasons the re-arm leaves out is still optional. A push dismisses this approval, and I will re-approve on the rebased head once CI is green.
69000bd to
973753b
Compare
|
Thanks @cruisercohen. Rebased onto |
cruisercohen
left a comment
There was a problem hiding this comment.
Re-approving after the squash and rebase; the change is unchanged and its tests now live in the split file.
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at 973753b.
The code fix is right. A loop that stood down for consecutive_failures or session_start_failures now settles the campaign FAILED instead of being re-armed (watchdog.py#L222). The documentation no longer matches it, though, and the review lanes flagged this on this head with no reply:
- The
_stalled_campaign_verdictdocstring (L204-L215) still says a marker plus a readable finding gives STOPPED and that everything else is FAILED 'unchanged'. Add a bullet saying a failure stand-down gives FAILED, ranks above the tombstone and the marker, and ranks below a verified finding. - The owning spec (learn-cron-dashboard.md, Research Lab watchdog) does not mention the stand-down settlement or the stand-down records removed at launch. Update it in this PR.
- The First Principles CONCERNS on this head is unanswered. Add a sentence to the description saying that
approval_stalled,structural_terminalandmodel_refusedare left to the re-arm on purpose, or post a disposition.
— crew-pr-reviewer (r112)
|
Addressed r112 at
|
…f reviving it AutoNudge stands a loop down after too many failed cycles or session starts in a row. The Research Lab watchdog re-armed every inactive campaign loop that was not an autonudge_stop tombstone, and a revival zeroes both failure streaks, so a campaign whose worker kept failing ran doomed cycles forever and the stand-down never held. Treat consecutive_failures and session_start_failures like the stop tombstone: settle the campaign as FAILED (Resume to retry) and remove the spent loop. A resumed run also clears a leftover stand-down record so the watchdog cannot settle the new run on the old run's evidence. The watchdog tests pace the loop through one helper, _drive_watchdog_polls, the single place that replaces its sleep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stalled-campaign verdict docstring and the Research Lab watchdog spec still described a stand-down as revivable. State that a consecutive_failures or session_start_failures stand-down settles the campaign FAILED, ranking below a verified finding and above the stop tombstone and the worker_done marker; that launch clears stand-down records with the tombstone; and why approval_stalled, structural_terminal and model_refused stay with the re-arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
371df3c to
76c005c
Compare
|
Rebased onto current main (9bf7e31) as 76c005c, with no conflicts and no code change. The earlier backend reds came from The three asks from the r112 CHANGES_REQUESTED review on 973753b are already done in 371df3c (now 76c005c):
Locally on this head, |
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at 76c005c9aa418641fd1b349d74c977c34ed86ecb.
A loop that stood down for consecutive_failures or session_start_failures now settles the campaign through the existing settlement path instead of being re-armed (watchdog.py#L424-L427). The campaign ends FAILED with a Resume hint, below a verified finding and above the tombstone and marker (L227-L233). The docstring, spec and description now cover everything my r112 review asked for.
I checked the First Principles note on cycle_cap and runtime_budget. Research loops are added with no runtime budget, so runtime_budget cannot occur. The watchdog settles the campaign COMPLETE once the cycle count reaches max_cycles (L511), so leaving cycle_cap with the re-arm does not bring back the silent loop. Every lane is clean and every check on this head is green.
— crew-pr-reviewer (r127)
Problem / Motivation
Goal: A Research Lab campaign whose worker keeps failing stops and says so, instead of running doomed cycles forever.
AutoNudge stands a loop down after too many failed cycles (
consecutive_failures) or failed session starts (session_start_failures) in a row. For a Research Lab campaign that stand-down never holds. Within one watchdog poll (5 s) the loop is active again, the campaign staysrunning, and the next doomed cycle is armed. The user sees a campaign that is "running" but makes no progress and never reports why.Why it matters
The stand-down exists to end exactly this silence: a worker whose session cannot start, or whose every cycle fails, keeps spending model turns (and, for unattended campaigns, auto-approved tool calls) with nothing to show for it. Every Research Lab campaign is exposed, because the watchdog re-arms every inactive campaign loop.
Not a goal
AutoNudgeService.update. A stand-down is documented as re-armable by a directive (test_the_stand_down_is_re_armable); the defect is this caller re-arming it unasked.What changed (motivation → approach → change)
Symptom → cause.
campaign/watchdog.py_watchdog_loopdiverts only anautonudge_stoptombstone into settlement. Every other inactive loop falls through to:That branch is meant to undo
_suspend_research_loops_while_disabled, but it also catches AutoNudge's own failure stand-downs.AutoNudgeService.updatetreats any inactive→active flip as a fresh run and zeroesconsecutive_failed_cyclesandconsecutive_start_failures(autonudge_service/mutations.py), so the five-strike stand-down is erased on every poll.The First Principles review of #20602 flagged this: "The Research Lab watchdog revives the same failure stand-downs the same way and stays unfixed." #20602 deliberately stays Issue-Radar-only. This PR is the Research Lab half.
Change.
agent_mode.FAILURE_STANDDOWN_REASONSnames the two stand-down reasons._settle_campaign_from_watchdog). That path already waits for a running worker turn, defers one poll for a freshly resumed run, and removes the spent loop once the campaign is terminal._stalled_campaign_verdictreturnsFAILEDwith "The research worker kept failing and stood down. Resume to try again." for a stand-down. It ranks below a verified finding (stillCOMPLETE) and above a staleworker_done.jsonmarker, which must not turn repeated failure into a "worker ended the loop"STOPPED._prepare_loop_launchalso removes a leftover stand-down record before a resumed run is published. Without that, if settlement failed to remove the loop, the watchdog would settle the new run on the previous run's evidence. This mirrors the existing tombstone cleanup.Resume is unchanged: from
FAILEDit relaunches through_launch_loop, which arms a fresh loop with fresh streaks.Left to the re-arm on purpose.
approval_stalled,structural_terminalandmodel_refusedstill go through the watchdog's re-arm. None of them has a streak that the inactive→active flip zeroes, so reviving one does not erase its bound: the next cycle that meets the same condition stops the loop again. Their remedy is also a person (an approval, a corrected goal or context), not a worker that keeps failing, so settling the campaignFAILEDwould misreport them. This answers the First Principles CONCERNS on973753b3c1.Docs (second commit,
371df3cc25). The_stalled_campaign_verdictdocstring now has the stand-down bullet:FAILED, ranked below a verified finding and above the tombstone and the marker. The Research Lab watchdog section ofdocs/system-specs/modules/learn-cron-dashboard.mdnow covers the stand-down settlement, the stand-down records removed at launch, and why the three reasons above stay with the re-arm.Backwards compatibility
Compatible: no config key, API field, route or stored value changes shape. A campaign whose loop stood down now ends
FAILEDinstead of stayingrunning, and Resume keeps working.Removes nothing: the only record removed is a stood-down research loop of a campaign that has just been settled terminal, the same removal the stop tombstone already gets, and Resume re-arms a new one.
Tests
test/test_auto_research_watchdog.py::TestWatchdogFailureStandDown(the watchdog tests moved there when #21269 splittest_auto_research.py):test_stand_down_settles_campaign_instead_of_reviving[CONSECUTIVE_FAILURE_REASON|SESSION_START_FAILURE_REASON]uses a realAutoNudgeService: arm a research loop, set both streaks, stand it down with the reason, and run two watchdog polls. It asserts the campaign isFAILEDwith the stand-down message, the loop is removed rather than revived, and the streak was never reset.test_stand_down_with_done_marker_is_still_a_failure: a staleworker_done.jsonplus a readable finding does not turn a stand-down intoSTOPPED.test_launch_clears_a_stand_down_record: a resumed run removes the previous run's stand-down record.Negative control: with only
campaign/watchdog.pyandcampaign/agent_mode.pyreverted toorigin/main, all 4 tests fail. The two real-service cases fail withassert 'running' == 'failed': the loop was revived and the campaign kept running.Focused run (Windows, py3.13): the 4 new tests pass. Across
test_auto_research*.pyplus the in-packageauto_researchtests: 544 passed. The 2 failures and 2 errors also fail with this change reverted, from a localcp950locale decode and a missingevent_loopfixture in this environment, so they are unrelated. flake8 and isort are clean on the touched files.The watchdog tests pace the loop through one helper,
_drive_watchdog_polls. It is the single place in the file that replaces the loop'sasyncio.sleep, and the twoTestWatchdogStopTombstonetests use it too, sotest_flake_pattern_ratchet.pycounts no new shape (60 passed).On the rebased head (
973753b3c1, one commit onda762d16d6),test_auto_research_watchdog.py,test_auto_research_worker_loop.pyandtest_auto_research.pygive 147 passed and 1 error. The error istest_cancellation_waits_for_persisted_loop_removal, which fails identically on a clean main checkout here (fixture 'event_loop' not found, a local pytest-asyncio mismatch).On
371df3cc25(docs and a docstring only): black, flake8, comment-history and brand gates are clean, andtest_auto_research_watchdog.pygives 18 passed and the same local-onlyevent_looperror.Manual verification
N/A — unit coverage sufficient: the regression test drives the real
AutoNudgeServiceand the real watchdog loop over a real campaign DB, which is the whole path involved. No UI or external service is touched.Related Issues
Closes #21254
Part of the stand-down revival class addressed for Issue Radar in #20602, and requested by that PR's First Principles review.
Pattern harvest
Rule candidate: review-prompt
Pattern: a reconciler that re-arms "any inactive loop" with
update(active=True)must first exclude stop reasons it does not own. A revival resets failure streaks, so an unconditional re-arm erases another component's stand-down.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code