Repository navigation
feat(schedule): say when a cron job is in failure backoff - #22642
Conversation
bolichen97
left a comment
There was a problem hiding this comment.
Looks good overall, one should-fix below. Not approving yet: readiness is still checking on this head and the fork review lanes have not finished.
Head reviewed: 54f369b.
What I checked:
failure_backoff_until()on main returns null at or below the default limit, andcompute_next_run_tsalready applies the same epoch. The wire fields match what the scheduler uses.fmtTimereads a number below the seconds ceiling as epoch seconds, so the seconds value from the board renders the right time.- The plural forms match each locale's existing
retried_n_timesset, and the key sits inpluralKeys.jsonin sorted order. - The wire test calls the real
api_cronswith a realCronJoband fixed timestamps. The page test covers future, past, null and absent values.
Should-fix (1): a disabled job can show the backoff note next to an empty next run. Details are inline.
| Only a FUTURE `failure_backoff_until` shows the note; a | ||
| past one has elapsed and explains nothing. Wraps like the | ||
| retry note: the column is narrow. */} | ||
| {typeof j.failure_backoff_until === 'number' && j.failure_backoff_until > Date.now() / 1000 && ( |
There was a problem hiding this comment.
🟡 A disabled job can show "Backing off ... until" next to an empty next run
failure_backoff_until() does not look at enabled. compute_next_run_ts returns null for a disabled job, so the cell shows the empty placeholder. If an owner disables a failing job with auto-pause off soon after its 5th failure in a row, the cell shows the empty placeholder and "Backing off after 5 failures in a row, until HH:MM" for up to one hour. The note says the job will run later, and the empty next run says it will not run.
{typeof j.failure_backoff_until === 'number' && j.failure_backoff_until > Date.now() / 1000 && (Why it matters: The note exists to explain a next run time. With no next run, it gives a wrong reason for a state the owner created.
Suggestion: Show the note only when a next run exists, for example add j.next_run_ts != null && to this condition. Another option is to send null from api_crons when j.enabled is false. Add one case to the page test for a disabled job. If the page already hides this cell for disabled jobs, I will withdraw this.
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this only adds things. Two read-only fields go on the wire, the page shows a note when they are present, and screenshots of this revision are attached. Notes
[DESIGN-REVIEWED] 055e4cb Description read[DESCRIPTION-READ] 51328495015c51f59c18ea7bc192552de7c938c3207a7af4ed1b91175a34b914 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS Both screenshots show the new "Delayed until 8:09 AM · 7 failures in a row" note, at desktop width and at 390 px. Nothing blocks. This fork lane has no blind read, so nobody has read the note without knowing the code; pushing the branch to this repository runs the lane that does. Notes
[UX-REVIEWED] 055e4cb Description read[DESCRIPTION-READ] 3c9e0824a3a14fe364ac94bc63c23ccac948f89400782763a2af6631e3c5c728 That sha256 names everything this verdict read: the PR description, plus the 5 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item comes from issue #22641, and the code reuses the existing What this change shipsInventory (8 items) — 7 justifiedIntent: let a cron job's owner see that its next run is late because of failure backoff, not a broken scheduler. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 055e4cb Description read[DESCRIPTION-READ] 3653b4a3f44f66d40a033f68c7fa1a7368742815b9449c607eca1e9bdf629714 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)Reviewed Opus 5.5 adjudication downgraded every blocking finding. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents. Review detailsBLOCKING -- website/src/pages/SchedulePage.tsx:1139 -- Backoff notes do not expire on an idle page
Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 — LOW. The note only goes stale on a page left idle. It is computed at render time (authentic.patch, SchedulePage.tsx:1139), and the page re-renders on every crons refresh pushed when a job starts or ends ( |
|
self-added: yes
|
|
self-added: yes
|
|
54f369b to
f194351
Compare
|
|
|
f194351 to
669e6d2
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
|
|
|
|
|
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 669e6d2, compared against the previous review at 54f369b.
The earlier should-fix is addressed. The board now sends failure_backoff_until from schedule.backoff_delaying_next_run(), which returns null for a disabled job, so a disabled row no longer shows a backoff note next to an empty next run. The same helper also returns null when the next scheduled slot already falls after the backoff end, and compute_next_run_ts now shares that test through _backoff_shift, so the note and the shifted time cannot disagree. New backend tests cover the disabled, slot past backoff and elapsed backoff cases on a pinned clock, and the frontend test now pins Date and checks the tooltip carries the note.
No new findings on this head.
Maintainer checks that still remain before this can merge:
- The readiness label is still
checkingand PR Readiness is pending on this head. - This is a
featchange, so it needs a human maintainer approval rather than an automated one.
|
chore (non-blocking): rebased on main, no behaviour change. |
669e6d2 to
8de044b
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES. The code looks sound, but the deferred UX work is tracked only on your fork, and the PR description does not link it.
Reviewed head: 8de044b. It is a no-change rebase of 669e6d2; range-diff shows the same patch.
The UX dispositions defer five items to Pearcekieser#48: the reword away from 'Backing off', the two-line target, the 'a successful run ends the delay' tooltip, a speaker review of the 11 translations, and a first-time-reader check. #48 covers all of them. Maintainers of this repository cannot track or close it, though, and the description does not link it. Please open it as an issue in kirodotdev/KiroCrew and link it from the description, or do the reword in this PR.
The code: backoff_delaying_next_run returns null for a disabled job and shares the shift test with compute_next_run_ts (
KiroCrew/src/kiro_crew/cron_service/schedule.py
Lines 371 to 396 in 8de044b
This head is not ready to approve yet, whatever happens with the issue. CI has not started, PR Readiness is pending, and no AI lane has posted for this head. A later review has to cover those results.
— crew-pr-reviewer (r116)
A job whose auto-pause limit is 0 or above the default backs off from its 5th failure in a row, and compute_next_run_ts already shifts its next run, but the Schedule page showed the later time with no reason. The cron board now serializes consecutive_failures and failure_backoff_until, and the Next Run cell shows 'Backing off after N failures in a row, until HH:MM' while the backoff is in the future.
8de044b to
055e4cb
Compare
|
@iamwhatever todo (non-blocking): the reword is now in this PR, so nothing is left on the fork tracker. On
The two remaining items on Pearcekieser#48 were a speaker review of the translations and a first-time-reader check. This repository has no speaker-review step for catalog strings, and the i18n check is the gate other UI PRs ship on, so I have not kept a tracker for them; #48 is closed. The description's Related Issues section now carries |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at 055e4cb against the previous head 669e6d2. Besides the rebase onto main, the change since then is copy only:
- The note now reads "Delayed until HH:MM · N failures in a row" in all 13 locale files, and the spec paragraph in learn-cron-dashboard.md matches it.
- A new non-plural key,
backoff_ends_on_success, is added to every locale and appended to the Next run cell tooltip. The claim holds: a successful run resetsconsecutive_failures, which ends the backoff. - The Russian
oneform is now nominative ("1 сбой"), which fixes the previous wording, andfew/manyagree with their counts. - The tests follow the new copy and also assert that the tooltip carries the success line.
No new findings. Approval is still not mine to give: this is a feat (Tier 2) and needs a human maintainer, and on this head the readiness label is still readiness: checking, PR Readiness is pending and most CI jobs are still running.
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE.
Reviewed head: 055e4cb.
This clears my earlier change request. The deferred UX copy is now done in this PR: the note reads "Delayed until HH:MM · N failures in a row" in all 13 locale files, the tooltip adds "A successful run ends the delay.", and the description no longer points at a fork tracker. I accept your reason for dropping the speaker review and the first-time-reader check: this repository has no such step for catalog strings.
The code: backoff_delaying_next_run returns null for a disabled job and uses the same _backoff_shift test that compute_next_run_ts shifts by (
KiroCrew/src/kiro_crew/cron_service/schedule.py
Lines 371 to 397 in 055e4cb
KiroCrew/website/src/pages/SchedulePage.tsx
Lines 1138 to 1159 in 055e4cb
The current-head AI notes repeat points you already answered on 669e6d2. For a cron-expression job the "until" time is the backoff end and Next Run is the first slot after it. A note left on an idle page clears on the next crons refresh. All checks on this head are green, including PR Readiness. The one open review thread is outdated and was fixed in 669e6d2.
— crew-pr-reviewer (r117)
Problem / Motivation
Goal: The Schedule page says when a cron job is in failure backoff, so an owner can see why its next run is later than the schedule.
A job whose auto-pause limit is 0 or above the default of 5 does not pause at its 5th failure in a row. It backs off instead: the next run waits 5 minutes, doubling per failure up to one hour, until a run succeeds.
compute_next_run_tsalready returns the shifted time, so the Next Run column shows "in 47m" for a job scheduled every minute, with no reason given.Why it matters
An owner who sees a next run far later than the schedule cannot tell a backoff from a broken scheduler, and has to read the store or the logs to find out. The backoff exists to bound the spend of a job that keeps failing, so the owner most likely to be looking at the row is the one trying to fix that job.
Not a goal
What changed (motivation → approach → change)
Symptom: a job in backoff shows a later next run with no explanation.
Approach: the page needs the time the backoff ends and the failure count behind it.
CronJob.failure_backoff_until()already computes the first, so the board sends it as a raw epoch rather than a "backing off" flag. The page compares it with its own clock, so the note clears once the time passes, without waiting for a refresh from the server.Change:
api_cronsinsrc/kiro_crew/dashboard/handlers/cron_board.pyaddsconsecutive_failuresandfailure_backoff_until(epoch seconds ornull) to each job.failure_backoff_untilcomes from the newschedule.backoff_delaying_next_run(), which returns the backoff end only when the job is enabled and its next scheduled slot falls before it, the same comparisoncompute_next_run_tsmakes to shift the time. A disabled job, and a daily job whose next slot is later than the backoff anyway, sendnull.website/src/types/index.jobs.tstypes both as optional, so an older gateway that omits them shows no note.website/src/pages/SchedulePage.tsxshows "Delayed until HH:MM · N failures in a row" under the Next Run value whilefailure_backoff_untilis in the future. The cell's tooltip repeats it and adds "A successful run ends the delay." It uses the existingtext-warntoken and wraps like the "Retried N times" note. The crons list reloads when a run starts or ends, which is when an elapsed backoff stops showing.backoff_ends_on_successsentence are inen.manual.jsonand the 11 translated catalogs,en-XA.jsonis regenerated, and the plural key is inpluralKeys.json. The note puts the time first and the count second, so it wraps to four short lines in the narrow column instead of five.docs/system-specs/modules/learn-cron-dashboard.mddescribes the two fields and the note.Backwards compatibility
Compatible: the board response gains two fields and drops none, and the page shows the note only when both are present.
Removes nothing: the change adds two read-only fields and a note.
Tests
test/test_cron_backoff_wire.pycalls the realapi_cronswith a realCronJoband the handler'stime.timepinned. A limit-0 job at 5 failures serializesconsecutive_failures == 5andfailure_backoff_until == last_run_ts + 300, equal to itsnext_run_ts. A disabled job, a daily job and an elapsed backoff each serializenull; those three fail on the first revision of this PR. The same job at 4 failures serializes4andnull, and a healthy job serializes0andnull.website/src/test/SchedulePage.backoffNote.test.tsxpinsDatewithvi.useFakeTimers({ toFake: ['Date'] })andvi.setSystemTime, and builds every fixture from that instant. It checks that a backoff one hour ahead shows the note with its count, that the cell's tooltip carries the note and "A successful run ends the delay.", and thatnull, a past time and a missing field each show nothing.test/test_dashboard_cron_approval.pysets the two attributes on its mock job.Determinism: both new test files pin their clock and read fixed timestamps, and neither starts a thread or timer.
3,258 cron and dashboard-cron tests, 93 Schedule page tests and 1,022 i18n tests pass, and
tsc -p tsconfig.app.json,I18N_BASE_REF=origin/main npm run i18n:check, black, flake8 and mypy are clean.Manual verification
The sandbox these were built in cannot open a listening socket, so the screenshots load the built
website/distin Playwright with request interception. The/api/cronsresponse comes from the realapi_cronsserializer.Screenshots / video
The "PR review sweep" row at desktop width, with the note under the next-run time:
At 390 px, where the note wraps under the next-run time:
Related Issues
Closes #22641. The backoff note deferred from #15524. The wording review the UX lane asked for is done in this PR, so no follow-up is left open.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.