Skip to content

fix(cron): keep only each job's 20 newest run transcripts - #31757

Merged
iamwhatever merged 1 commit into
mainfrom
fix/cron-run-transcript-keep-per-job
Nov 8, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/cron-run-transcript-keep-per-job

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A stateless cron job's run files stay near 20 per job, however often it fires, so busy jobs can no longer bury the gateway in session files.

A cron job with persistent_session: false writes one cron_<job>_<run>.jsonl file per run into ~/.kiro/crew/sessions/. The cleanup deletes a run file only after 7 days, and at most 200 per hourly pass. A job that fires every minute leaves about 10,000 files. A user had about 50,000, mostly from ops-mission-control's dispatch and rotation-check crons. After upgrading to 0.12.0.8 their gateway booted but never answered and kirocrew token kept saying port 5476 was up but busy. Deleting the files fixed it.

Why it matters

Anyone running a frequent stateless cron (an app's poller, a dispatch loop) slowly fills the sessions folder. Every full session listing then pays one file read per run, until the dashboard stops loading. Users can't see why, and reinstalling doesn't help because the files stay.

Not a goal

What changed (motivation → approach → change)

The 7-day age limit can't bound the pile, because the count depends on how often a job fires. So each job now keeps only its 20 newest run files, plus the age limit. Recent runs stay for debugging, and each run's result and trace stay in cron history either way.

Part Before After
Kept per job every file under 7 days the 20 newest, none over 7 days
Fire refused at the gate or admission leaves a tagged empty file leaves no file
Files per pass 200 2,000
First pass 1 hour after start 5 minutes after start
Next pass after a full one 1 hour 5 minutes

A keep count only holds if refused fires don't fill it. So the cron_run="ephemeral" tag is written when the run starts, in slack/gateway_runtime/cron_turn.py _cron_run_single, right after its key is marked active, instead of in cron_fire.py _cron_capture before the gate. A refused fire writes no file, and a tagged file is held by a live key until its run ends, so the prune can't remove it mid-run.

The rules that already protected a file still hold: it must be tagged, not held by a live run, and not pinned. The prune is in cron_service/run_residue.py (ranking and counts) and cron_service/service/run_teardown.py (the reaper's timing). The cron spec (learn-cron-dashboard.md) names the new rules.

Backwards compatibility

Compatible: no setting, API or file format changes. Old installs just lose more of their old run files.

Reader: src/kiro_crew/cron_history.py:CronHistoryStore -- cron -- run summary, error and trace live in their own store, untouched by this prune
Reader: src/kiro_crew/cron_service/service/run_kill.py:_run_session_keys -- cron -- reads only a live run's key, which active_session_keys keeps out of the prune
Reader: src/kiro_crew/dashboard/chat_utils_slots.py -- chat -- maps a live run's key to the job's tab; live runs are never pruned
Reader: src/kiro_crew/history.py:ConversationLog -- chat -- History lists what is on disk; a pruned run drops out, as a 7-day-old one already does
Reader: src/kiro_crew/slack/gateway_runtime/cron_delivery.py -- cron -- re-tags a run that opened a tab to tab; delivery runs after the start-of-run tag, so tab still wins (test_cron_new_chat_per_run.py)

Tests

test/test_cron_run_residue.py adds:

  • a busy job keeps its 20 newest runs and loses the rest, even when all are minutes old;
  • two jobs each keep their own 20;
  • a live run or an untagged file past the 20 is kept;
  • the reaper reports a backlog only when a pass fills its cap, and not when the pass raises.

test/test_cron_ephemeral_session.py covers the tag site:

  • a fire capture writes no transcript, for every persistent and sequence mix;
  • a stateless run is tagged only after its key is active;
  • a persistent run is never tagged;
  • a fire refused by the fire-time gate, or at admission, writes no transcript.

Proof: not provable -- other: prove.py reverts the new RUN_TRANSCRIPT_KEEP_PER_JOB constant the test imports, so the run errors at collection. By hand on origin/main with the constant inlined in the test: 4 failed (the busy-job test and the three reaper tests), 12 passed. With cron_fire.py and cron_turn.py restored from the parent, 5 of the tag-site tests fail (both refusal tests, the active-before-tag test, and the two stateless capture cases).

Determinism: 10 repeats all passed, and a run in reversed test order passed.

Manual verification

N/A — unit coverage sufficient: the change is file selection and a timer, both exercised directly with real files in a temp sessions folder.

Related Issues

Closes #31734

Sibling sweep

Swept: git grep -nE "retention_days|_TTL_SECS|PRUNE_CAP" -- src/kiro_crew
Sibling: src/kiro_crew/crew_log/store_retention.py -- not affected -- one log per closed session, not one per timer fire
Sibling: src/kiro_crew/history_files.py -- not affected -- archive files are one per closed session, at the rate a person closes chats

Pattern harvest

Not generalizable: the only file written once per timer fire is this per-run transcript; other retention paths store one file per session.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 6, 2026 20:10
@iamwhatever
iamwhatever requested a review from Zedmor November 6, 2026 20:10
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b5085b46ab7dea948c0ad9c0ea86aa86a0197258 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (6 items) — 6 justified

Intent: stop frequent stateless cron jobs from piling up run files until the gateway stops answering. This is a FIX (#31734).

  1. Each job keeps only its 20 newest run transcripts, and the 7-day limit still applies — justified
  2. A fire refused at the gate or at admission no longer leaves an empty tagged transcript — justified
  3. A run's transcript is tagged only once its key is active, so the prune cannot remove it mid-run — justified
  4. One cleanup pass now removes up to 2,000 files instead of 200 — justified
  5. The first cleanup pass runs 5 minutes after start instead of 1 hour — justified
  6. A pass that hits its cap schedules the next pass 5 minutes later instead of 1 hour — justified

(The pr-policy.yml, platform-compat.md and kiro_prerequisite spec hunks in the patch come from #32320, which is already on the base. They are not part of this branch.)

[FIRST-PRINCIPLES-REVIEWED] b5085b4

Description read

[DESCRIPTION-READ] e29fd6a7ac4fe2b84972669f1e666ead6ad8e41a9db48f11a83c0f44f1766bf8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b5085b46ab7dea948c0ad9c0ea86aa86a0197258 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b5085b46ab7dea948c0ad9c0ea86aa86a0197258 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. _cron_run_single is the only stateless agent path, so moving the tag there loses no transcript the prune used to cover, and the per-job keep count holds however often a job fires.

Notes

  • The ranking counts every cron_<job>_<run>.jsonl file, including untagged legacy files and tab-tagged ones. Those still count toward the 20 but are never deleted, so a job with old files keeps fewer than 20 ephemeral runs. The cleanup still stays bounded, and that is acceptable.

[DESIGN-REVIEWED] b5085b4

Description read

[DESCRIPTION-READ] 42472f31e1bbddd31c2edfa7b8140d15f290ce9c591295ed60960df6f093b0f3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b5085b46ab7dea948c0ad9c0ea86aa86a0197258 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b5085b4

Verdict parsed from the review's SHA-scoped output markers for commit b5085b46ab7dea948c0ad9c0ea86aa86a0197258.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b5085b46ab7dea948c0ad9c0ea86aa86a0197258 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b5085b46ab7dea948c0ad9c0ea86aa86a0197258 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b5085b4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b5085b46ab7dea948c0ad9c0ea86aa86a0197258 false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 7) and (Windows) (7) fail on test/test_pull_request_target_allowlist.py::test_only_the_allowed_workflows_use_pull_request_target. That is main's red, tracked in #31771 (from #31396); this PR touches no workflow. The other reds on this head (Backend Tests (3.12, 4), E2E, Sensitive Change Review) failed on GitHub API rate limits (HTTP 429). This PR will rebase once the #31771 fix lands on main.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head went green (39717a0 -> 7c364bc); CI re-running. The new base carries the #31771 fix for the pull_request_target allowlist red. The same push drops the unused keep_per_job parameter (First Principles note); behavior is unchanged.

@iamwhatever
iamwhatever force-pushed the fix/cron-run-transcript-keep-per-job branch 2 times, most recently from a4fd6cc to c753e29 Compare November 7, 2026 00:49
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/learn-cron-dashboard.md since this head went green (7c364bc -> f11c19b); CI re-running. The previous head's CI run sat queued for over 2 hours with no runner, and its other reds were GitHub API 429s.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@iamwhatever

iamwhatever commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=23a1a256310a refused fires leave metadata-only stubs that take keep slots

fixed in af0fd83: a transcript holding only its metadata line takes no keep slot and stays a deletion candidate, still behind the ephemeral-tag, live-key and pinned checks.
Regression test test_metadata_only_stubs_take_no_keep_slot: 25 stubs newer than 3 real runs are removed and the real runs kept; it fails with the stub handling reverted (8 removed instead of 25).
test_a_held_metadata_only_stub_is_kept pins that a stub held by a live run survives.

@iamwhatever
iamwhatever force-pushed the fix/cron-run-transcript-keep-per-job branch from c753e29 to af0fd83 Compare November 7, 2026 06:56
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 7, 2026
@iamwhatever
iamwhatever force-pushed the fix/cron-run-transcript-keep-per-job branch from af0fd83 to c8ca52b Compare November 7, 2026 07:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@iamwhatever

iamwhatever commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=7eeda5878368 immediate stub pruning races with cron startup

fixed in c8ca52b: the cron_run="ephemeral" tag moved from _cron_capture (before the fire-time gate and admission) into _cron_run_single, right after register_active_session_key, so a tagged file is held by a live key until its run ends.
A refused fire now writes no file at all, and the stub-deletion path (_has_message_line) is removed, so the prune never deletes a file whose run has not started.
Tests: test_a_stateless_run_is_tagged_only_after_its_key_is_active, plus the gate- and admission-refusal tests that assert no transcript is written; all fail with cron_fire.py and cron_turn.py restored from the parent.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=23a1a256310a the prune deletes a tagged transcript while its run waits to start

fixed in c8ca52b: the tag is written in _cron_run_single after register_active_session_key, so the key is active before the file exists and active_session_keys keeps it out of every pass.
A fire refused at the gate or admission writes no file, so the metadata-only stub branch is removed and ranking is back to keep-count plus 7-day age.
Tests: test_a_stateless_run_is_tagged_only_after_its_key_is_active and the two refusal tests in test_cron_ephemeral_session.py; all fail with the tag at its old site.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 2), Windows (2) and the Coverage Gate fail on test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget (34,855 > 34,845). That is main's red: it fails the same way on origin/main 5f7f8d1, and this PR touches nothing under src/kiro_crew/security. The fix is #32183; this PR will rebase onto it once it lands.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 7, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 7, 2026
@iamwhatever
iamwhatever force-pushed the fix/cron-run-transcript-keep-per-job branch from c8ca52b to b5085b4 Compare November 7, 2026 19:43
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/slack/gateway_runtime/cron_turn.py, cron_fire.py, cron_service/service/run_teardown.py and docs/system-specs/modules/learn-cron-dashboard.md (base b65af1a -> 20764b5). The only conflict was the TYPE_CHECKING import list in cron_turn.py: main added auto_job_approver and this PR adds agent_sequence_dispatches, so both stay. Re-ran test_cron_run_residue.py, test_cron_ephemeral_session.py, test_security_gate_liveness.py (47 passed) and the cron_turn/cron_fire/run_teardown tests (96 passed). test_app_script_cron_sibling_import.py and test_cron_run_scratch.py fail at collection on main 20764b5 too. New head b5085b4.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 7, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 8, 2026 08:16
@iamwhatever
iamwhatever disabled auto-merge November 8, 2026 09:12
@iamwhatever
iamwhatever merged commit 072b98f into main Nov 8, 2026
110 of 111 checks passed
@iamwhatever
iamwhatever deleted the fix/cron-run-transcript-keep-per-job branch November 8, 2026 09:12
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 8, 2026
bolichen97 pushed a commit that referenced this pull request Nov 9, 2026
Co-authored-by: Joe Guo <zejiangg@amazon.com>
(cherry picked from commit 072b98f)

Backport note: cron_turn.py's TYPE_CHECKING list gains only
agent_sequence_dispatches (auto_job_approver is #30692, not on release).
learn-cron-dashboard.md takes the prune wording without the
new_chat_per_run clause and Retention bullet, which release does not have.
bolichen97 pushed a commit that referenced this pull request Nov 9, 2026
Co-authored-by: Joe Guo <zejiangg@amazon.com>
(cherry picked from commit 072b98f)

Backport note: cron_turn.py's TYPE_CHECKING list gains only
agent_sequence_dispatches (auto_job_approver is #30692, not on release).
learn-cron-dashboard.md takes the prune wording without the
new_chat_per_run clause and Retention bullet, which release does not have.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stateless cron run files pile up faster than the cleanup removes them, wedging the gateway

3 participants