Repository navigation
fix(autonudge): bound the re-probe of a delivered terminal - #18648
bolichen97 merged 2 commits into
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS No design problem: the three-way re-probe answer (holds/gone/unknown) fixes the root cause, which was that an unreadable reading and a real reopen took the same branch. The cap on retries keeps a watch from going silent. Notes
[DESIGN-REVIEWED] d1689c6 Description read[DESCRIPTION-READ] 40d10487ebe999438ce2556329e358fcc29168f00cbb9d936da20fc99f0b3be6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix is at the right level: the re-probe now returns three answers instead of two, so an unreadable reading no longer takes the reopen branch. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: stop a watch from announcing an already-delivered "PR merged" turn a second time after one unreadable re-probe, while still letting an unreadable watch go back to normal observation. FIX.
[FIRST-PRINCIPLES-REVIEWED] d1689c6 Description read[DESCRIPTION-READ] 23f9d2439ed6087ce23d72678c45aed410ef7e261c75d3f27d3021f33aebeda1 That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing blocks. The only new user-facing string, the shutdown refusal "The gateway is restarting, so this was not applied. Try again once it is back.", appears in the committed screenshot [UX-REVIEWED] d1689c6 Description read[DESCRIPTION-READ] 750eb2d5f6a3186542e0327fc7bfbcc5c6886ffc3bb9858ad55e241d3fe17e8a That sha256 names everything this verdict read: the PR description, plus the 5 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
762310e to
52560e4
Compare
|
Replayed onto #9052's new head |
52560e4 to
be95bdb
Compare
be95bdb to
2619780
Compare
|
Replayed onto #9052's round-33 head |
2619780 to
ef27931
Compare
|
|
|
|
This round's review of
The one CI red, Backend Tests (3.12, 2), is the |
|
|
|
|
|
|
|
|
|
|
Restacked on #9052's new head Verified on
|
|
New head The finding holds. A busy session refuses the final fire, and the refusal grants one observation-free follow-up tick so the next tick retries the delivery. While the turn is owed the bypass never spends that credit, so it is still standing when the retry lands. After three unknown re-probes the settlement-only branch steps aside, and the follow-up bypass spent the credit and fired a second final turn instead of observing. The bypass now also requires
The branch is restacked on main Verified on
The Opus 5.5 lane did not complete on |
|
Every run on The one red is not this change's. |
|
Restacked on #9052's new head Verified on
The shard-5 flake from |
A channel terminal turn that landed is recorded in terminal_delivered and settled by a provider re-probe. A re-probe that cannot read the subject (no target or probe, a raised or unreachable fetch, the kernel's fallback, an observation-less wake) cleared the marker in the base behavior, so one unreadable probe could cost a repeated terminal turn on the next ordinary tick. The re-probe now answers holds, gone or unknown. An unknown answer keeps the delivered marker and counts it in memory, so the next quiet ticks re-probe without delivering. After three unknown answers the settlement-only branch steps aside and ordinary observation resumes, below the IRQ kernel's six-failure blind backstop, but the marker stays: a reading whose classification matches it settles the watch without firing again, a definite still-open reading clears it so a reopen and a later close still get their turns, and the backstop's wake still fires. The observation-free follow-up allowance stays shut while the marker is kept, as it does while a terminal turn is owed, so the retry credit a busy session's refusal left standing cannot fire a second final turn once the bound is exhausted. The count is not persisted; a restart resets it, which only buys a few more quiet re-probes. A retarget, a settlement, a removal, a fresh delivery or a stop drops it. Every marker change is staged and written before the live monitor takes it, so no other snapshot can store a clear that its own write then fails to make durable.
|
Replayed Every lane on |
|
HOLD: this still carries #9052, which is open with active change requests. Reviewed at d1689c6. The branch's first commit, a2b4e7e, is #9052's head. #9052 is still OPEN, labelled needs-author-decision, with CHANGES_REQUESTED reviews from bolichen97 and iamwhatever that have not been dismissed. Merging this PR would land #9052 without its review. The maintainer and the author already agreed this PR waits for #9052. That has not changed since our 2026-10-13 comment. This PR's own commit has grown since that comment (now 7 files, +671/-90). I read it and found no defect. Unknown re-probes keep the marker up to To clear this hold: #9052 merges, this branch is rebased so only its own commit sits on main, and item 5 has a reply. That rebased head will need a fresh review. — crew-pr-reviewer (r120) |
|
On item 5 of the crew-pr-reviewer HOLD (First Principles note: the still-open clear in The next push of this PR routes both through one helper, and each caller keeps its own guard. That push waits for #9052: this branch carries #9052's head until it lands, and is then rebased so only its own commit sits on main. |
|
HOLD at d1689c6. This branch still carries #9052's head (a2b4e7e) as its first commit. #9052 is still open with undismissed change requests from bolichen97 and from us. Merging this PR now would land #9052 without its review. Thank you for the reply on First Principles item 5. Routing the still-open clear (gate.py#L1173-L1220) and the To clear the hold: #9052 merges, and this branch is rebased so only its own commit sits on main, with the shared helper. That rebased head needs a fresh review. I found no defect in this PR's own commit. CI is green on this head; I ran no PR code locally. Reviewed head: d1689c6 — crew-pr-reviewer (r121) |
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed current head d1689c6bead37405572db7ea16a75befe969e5a9 as Tier 2. Deep review found no blocking issue.
Security check: monitor ownership, credential rollback, admitted mutation drain, and terminal replay remain fail closed across cancellation and shutdown. Auto-merge remains subject to branch protection.
|
The one-helper change for both delivered-marker clears that I described in this comment is not in this merge; it is tracked as a separate follow-up, together with the |
Problem / Motivation
Goal: A delivered terminal turn is not repeated because one re-probe could not read the subject, while an unreadable watch still resumes ordinary observation after a bounded number of tries.
With #9052, a channel terminal turn that landed is recorded in
terminal_deliveredand settled by a provider re-probe. A re-probe that cannot read the subject (no target or probe, a raised or unreachable fetch, the kernel's fallback, an observation-less wake) clears the marker as main does, so the next ordinary tick can deliver the same terminal turn again.Why it matters
A repeated terminal turn is a second "your pull request merged" message and a second model turn for news already delivered. A transient
ghfailure is enough to cause it.Not a goal
terminal_pending), which keeps its fire-on-uncertainty behavior.What changed (motivation → approach → change)
Stacked on #9052: this branch carries #9052's head commit
a2b4e7e606itself, one commit on4533b1eb5b. Review the top commit only; this PR depends on #9052 and merges after it. The fork review lanes that read the whole diff againstmainsee #9052's commit too until it lands; this PR's own commit touches nowebsite/file.Symptom: after #9052, an unreadable re-probe drops the delivered marker, and the next ordinary tick may announce the terminal again.
Root cause: the re-probe has two answers, so an indeterminate reading takes the same branch as a definite reopen.
Change:
_terminal_still_holdsanswersholds,goneorunknown.unknowncovers no target or probe, a target that no longer matches the monitor, a raised fetch, the kernel'sFALLBACK, an unreached observation and an observation-lessWAKE; a work-ledgerQUIETis a definitegone.unknownkeeps the delivered marker and is counted in memory per monitor (AutoNudgeService._terminal_reprobe_unknowns), so the next quiet ticks re-probe without delivering. After_DELIVERED_REPROBE_UNKNOWN_LIMIT = 3unknown answers (about ten minutes across two default-cadence retries, below the IRQ kernel's six-failure blind backstop) the settlement-only branch steps aside and ordinary observation resumes, but the marker stays. The bound is three rather than one because an ordinary tick that cannot read the subject reaches no verdict and spends a turn: at one, an outage that outlasts the first re-probe costs a turn on a watch whose final turn has already landed. A reading whose classification matchesterminal_deliveredsettles the watch without firing again (_settle_observed_delivered_terminal). A definite still-open reading (a reached pull-request reading that is notFALLBACK, or a work-ledgerQUIET) clears it, so a reopen and the next close still get their turns. The backstop'sWAKEstill fires. The count is not the kernel'serrorsstreak: that streak counts failed fetches only,irq.polldoes not expose it, and it sits at zero after any good fetch for the non-fetchunknownanswers (a work-ledgerWAKE, aFALLBACKfrom a raised probe). While the marker is kept, the observation-free follow-up allowance stays shut, as it does while a terminal turn is owed: a busy session's refusal of the final fire grants one retry credit that the owed turn never spends, and spending it once the bound is exhausted would fire a second final turn instead of observing (GPT 6.1 on5532128b48).unknownanswer writes nothing. Theholdssettlement, from the re-probe or from a matching ordinary reading, goes through one staged write (_commit_delivered_settlement) that lands before the live monitor changes, and thegoneand still-open clears stage theirs the same way. The count is dropped on settlement,gone, a reopen clear, a fresh delivery, a retarget, a removal and a service stop. A restart resets it, which only buys up to three more quiet re-probes, never a second turn.learn-cron-dashboard.mddescribes the tri-state answer, the in-memory bound, the kept marker and the settlement on a matching reading.Design questions
The bound is better stated as a count ("two further re-probes, about ten minutes at the default cadence") than as a duration, since crew-log pushes also bring ticks forward and spend retries.
Backwards compatibility
Compatible: no new persisted field. A row an earlier head of this PR wrote carries a
terminal_reprobe_unknownskey, whichMonitorStatekeeps underextra_fieldsand writes back unchanged; nothing reads it.Tests
Validation on
d1689c6bea, directly atop #9052's heada2b4e7e606(one commit on4533b1eb5b; this PR's own files are patch-identical toad5aa6676f): everytest_autonudge*.pyandtest_monitor_*.pymodule, the conductor patrol, slot-close race, workflow nudge wiring and Slack composition modules pass (2,922 tests), as do everytest_slack_gateway*.pymodule and the boot-path test (1,087 tests) and main's newtest/metrics/test_monitor_loop_counters.py(85 tests). The repository-wide facade-mirror, link-screen and dashboard-files composition guards pass too. Baselined Black, isort, Flake8, mypy, the comment-history, brand-name, subprocess-encoding, documentation-lint, line-cap and harness-parity gates, the flake-pattern and wait-discipline ratchets, and the max-two-commit push guard pass.test/test_autonudge_shutdown_terminal_reprobe.py:test_exhausted_unknown_reprobes_then_a_holds_reading_settle_without_a_second_turn(three unknowns, an unreadable tick, then the merged reading: no second turn),test_exhausted_unknown_reprobes_do_not_spend_a_busy_retry_credit_on_a_second_turn(a busy refusal, the delivered retry and three unknowns, then the merged reading settles with two fires in all; it fires a third on5532128b48),test_exhausted_unknown_reprobes_then_a_reopen_clears_and_the_next_close_fires,test_exhausted_unknown_reprobes_still_deliver_the_kernel_backstop,test_MUTATION_an_indeterminate_reprobe_writes_nothing_and_keeps_the_marker,test_MUTATION_a_reopen_publishes_the_cleared_marker_only_after_its_write_lands(the reopen clear reaches memory only after its write lands, and a failed write is retried) andtest_MUTATION_an_outage_past_one_reprobe_costs_no_extra_turn(a second unreadable tick re-probes instead of spending a turn); a definite answer resets the count and settles once; an unobservable subject keeps the delivered marker without settling; a retarget drops the previous subject's marker and count. Each new case fails under the mutation it names (no match-settle, no reopen clear, a write onunknown, no retarget reset, a reopen clear published before its write, a bound of one).test/test_autonudge_approval_stall.py: an approval release leaves the delivered marker untouched.Manual verification
N/A: the unit tests drive the real settlement path with a stubbed re-probe against a temporary store.
Related Issues
No linked issue; this was split out of #9052 at the maintainer's request.
Pattern harvest
Not generalizable: the bound is a retry count for one re-probe of one persisted marker in
_settle_delivered_terminal; a repeated-indeterminate-reading limit is a policy choice per probe, not a shape a rule can flag.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement