Skip to content

feat(apps): cap how many finished job records a kind keeps - #33196

Merged
bolichen97 merged 1 commit into
mainfrom
fix/7588-job-sdk-retention-cap
Nov 9, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/7588-job-sdk-retention-cap

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let an app bound how many finished job records each kind keeps, so a store that gains a run every night stops growing forever, starting with the backup jobs.

The Job SDK writes one record file per run and removes records only when the app is disabled. list_recent's limit caps the response, not the store. So every kind's file count grows with every run, and every read that needs the whole set (list_active, list_recent, reconcile) opens every file the app ever wrote. The backup jobs already run on the Job SDK: the nightly loop adds a record per due kind every night, and the Backup row reads the whole store four times on each status read.

Why it matters

Without a bound, a long-lived install pays a little more on every backup status read, forever, and an app that moves onto the Job SDK inherits the same growth. The cap lands with a real caller, as the maintainer decision on #7588 asks, so its shape is tested by a consumer before other apps depend on it.

Not a goal

What changed (motivation → approach → change)

Records pile up because nothing removes them, so register gains an opt-in count cap and the backup jobs use it.

Piece What it does
register(kind, fn, retention=N) After each terminal write of that kind, removes that kind's oldest finished records until at most N remain. 0, the default, keeps everything.
The sweep (job_sdk_retention.py, a new owner behind the job_sdk facade) Never removes a non-terminal record, a run still executing, or a run whose own sweep has not run yet, so two runs finishing at once cannot remove each other. Ranks the rest by the existing finished_at, updated_at, created_at; there is no new field. Never raises, runs one at a time per app (a run is let go before the next sweep scans), and removes nothing once the app is disabled.
Reading jobs/ Each record is opened with the existing jsonl_util.open_regular_nofollow, so a planted link, a non-regular or oversized file, or a body naming another run is skipped. Not the agent-facing read: its path screen refuses aws-control's own data dir.
A capped kind's dedupe_key Refused past 512 characters (ValueError; the start route answers 400 invalid_dedupe_key). It is the one record field a caller sets with no length bound, and a record too big for the scan would never be removed.
Evidence Each removal is audited as jobs.job_evicted, and reconcile will not write back a record that is gone. The start route answers with the run as it stood at the start when a fast run's record is gone before it reads it back.
aws_control backups Both kinds register with retention=100, at or above every reader's window (the Backup row reads 20 of a kind, _jobs/recent at most 100).

The App Kit API reference documents the option, including when not to use it, the key limit, and that an older host raises TypeError. The app-kit-platform spec gains §26.1, and docs/decisions/ records the maintainer decision.

The app repository carries the same registration in KiroCrewCaptain/kirocrew-app-aws-control#1 (commit d9243e312c): it falls back to an uncapped kind on a host without the option.

Builds on #22711 by @dodgeblaster. It keeps that PR's sweep, store delete, no-follow scan, audit event and reconcile guard, and drops live, on_evict and the seq counter.

Backwards compatibility

Compatible: retention is keyword-only with a default that keeps every record, so an app that does not pass it behaves exactly as on main, long dedupe_key values included.

The backup jobs now remove their oldest finished records past 100 per kind, and their start route refuses a dedupe_key over 512 characters. No client sends one: the backup page starts runs through its own backup/{account}/run route with the 12-digit account as the key, and the nightly loop does the same. Readers of those records:

Reader: src/kiro_crew/apps/builtins/aws_control/backend/routes_parts/backups.py:_account_jobs -- app -- test_the_backup_row_still_reports_the_newest_failure_after_the_cap
Reader: src/kiro_crew/apps/job_routes.py:_handle_recent -- app -- test_a_run_the_retention_cap_removed_reads_as_unknown
Reader: src/kiro_crew/apps/job_routes.py:_handle_get -- app -- test_a_run_the_retention_cap_removed_reads_as_unknown (a removed run answers 404 job_not_found)
Reader: src/kiro_crew/apps/job_routes.py:_handle_cancel -- app -- test_a_run_the_retention_cap_removed_reads_as_unknown (a removed run answers 404 job_not_found instead of 409; no page holds such an id)
Reader: src/kiro_crew/apps/job_routes.py:_handle_start -- app -- when the run it reads back is already gone it answers 200 with the run as it stood at the start, instead of 500 run_unreadable: test_a_start_whose_run_the_cap_removed_still_answers_the_run; a capped kind's overlong key answers 400: test_a_capped_kind_answers_400_to_an_overlong_key
Reader: src/kiro_crew/apps/job_routes.py:_handle_active -- app -- only finished records are removed: test_an_executing_run_is_never_removed
Reader: src/kiro_crew/apps/builtins/aws_control/backend/backup.py:_run -- app -- its sdk.get reads its own executing run, which is never removed: test_an_executing_run_is_never_removed
Reader: src/kiro_crew/apps/job_sdk.py:JobSDK.reconcile -- app -- test_reconcile_does_not_bring_back_a_removed_record
Reader: src/kiro_crew/apps/builtins/aws_control/hooks.py:_register_job_runners -- app -- test_both_backup_kinds_register_with_the_cap, test_the_cap_is_at_least_every_readers_window

Tests

test/test_job_sdk_retention.py (new):

  • a kind without retention keeps every record; a capped kind settles at the cap and keeps the newest; a cap on one kind leaves another alone;
  • the run that just finished is kept even when older records sort newer;
  • an executing run and another process's unresolved record are never removed; two runs finishing at once do not remove each other; a sweep lets its run go before the next sweep scans; a run finishing during a scan is not removed;
  • each removal is audited; a negative cap is refused;
  • a capped kind refuses a key past the limit and still removes a record whose key is at it; an uncapped kind still takes a long key; the largest record a capped kind can write fits the scan;
  • a failing sweep leaves the run terminal and released; a disabled SDK's late worker removes nothing; reconcile does not bring a removed record back;
  • the scan skips a body naming another run, a planted link, an oversized record and an unparseable one;
  • the cap applies in aws-control's own data dir; both backup kinds register with the cap; the cap is at least every reader's window; and the Backup row still reports the newest failure after the cap removed older runs.

test/test_job_routes.py: a run the cap removed is gone from recent and answers 404 to get and cancel, while the kept run is served; a start whose run the cap removed before the read-back still answers 200 with the run; a capped kind answers 400 invalid_dedupe_key to an overlong key without quoting it. test/test_job_sdk_facade.py lists the new owner.

Determinism: the two files' 59 tests passed at -n0 in three shuffled orders (seeds 4242, 31337, 777), and the four key-limit tests passed 20/20 at -n0. The refusal tests fail with the length check removed and pass with it. They wait on record file names, the live table and each run's own sweep, never on a sleep, and never patch the clock.

Manual verification

N/A — unit coverage is sufficient: the change is a backend module and one registration call with no screen of its own, and the sweep, its order and every reader are exercised by the tests above.

Screenshots / video

N/A — no frontend file is touched.

Related Issues

Closes #7588

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 8, 2026 13:40
@bolichen97
bolichen97 requested a review from Zedmor November 8, 2026 13:40
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. It's an opt-in, keyword-only cap with a default that keeps every record, and its first caller is the backup jobs. Every reader of the removed records is listed in the description and backed by a test.

Notes

  • _handle_start now calls the private sdk._start_run instead of going through start_async. A later change to the public start path could then skip the route. Consider having start_async return the start-time copy, or adding a public variant that does.

[DESIGN-REVIEWED] 3a9b1c7

Description read

[DESCRIPTION-READ] 001d379cddd3d301a5b417ee68157c6f41092d7f7443237b2b1955ea7db3c4d2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The cap has a real caller and a maintainer decision (#7588), and the backup readers' windows (20 and 100) sit at or below the cap of 100. What's left is one second-spelling scan that pulls in a new key limit.

Notes

  • Item 6 — duplicate of JobStore.iter_runs (src/kiro_crew/apps/job_sdk.py:391): _iter_runs_nolink is a second full-store scan. Its max_bytes=_MAX_RECORD_BYTES bound is the only reason item 3 exists ("a record too big for the scan would never be removed"). Its link refusal guards a read that iter_runs already does unguarded for list_recent, and remove unlinks a link rather than its target. Subtraction: have the sweep use iter_runs and keep only the run.run_id != path.stem skip. Then delete _MAX_DEDUPE_KEY, the ValueError branch in _start_run and the invalid_dedupe_key 400.
  • Item 4 — undeclared for uncapped kinds: run or started turns any None read-back into a 200, including a corrupt record on a kind with no cap, where it used to be 500 run_unreadable. Subtraction: use the start snapshot only when runner.retention > _NO_RETENTION.

What this change ships

Inventory (7 items) — 5 justified

Intent: stop a Job SDK kind's finished-run records from growing forever, starting with the nightly backups. This is an ADDITION.

  1. Apps can pass register(..., retention=N) to keep only the newest N finished records of a kind — justified
  2. The backup kinds now delete their finished records past 100 per kind — justified
  3. A capped kind refuses a dedupe_key over 512 characters, and the start route answers 400 invalid_dedupe_key — oversized; exists only because of item 6's size bound
  4. Start answers 200 with the start snapshot instead of 500 when the read-back is gone — undeclared for uncapped kinds
  5. reconcile no longer writes back a record the cap removed — justified
  6. A second no-follow, size-bounded scan of the jobs/ folder — duplicate of JobStore.iter_runs
  7. A new jobs.job_evicted audit event for each removal — justified

[FIRST-PRINCIPLES-REVIEWED] 3a9b1c7

Description read

[DESCRIPTION-READ] aef617c4080ff58fba7704b0a91d670721778d76ff4a530e5a51c4d1e93f9033

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@bolichen97
bolichen97 enabled auto-merge (squash) November 8, 2026 13:44
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3a9b1c7

Verdict parsed from the review's SHA-scoped output markers for commit 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/apps/job_sdk.py:1023 -- Terminal writes outside workers bypass retention

self._sweep_retention(runner, run)
Trigger: When the host refuses new worker threads, repeated starts reach _start_run’s terminal FAILED write without entering _execute. Startup reconciliation similarly writes INTERRUPTED records without reaching this sole sweep call.
Outcome: With retention=1, three refused starts retain three failed records; reconciling three stale runs retains three interrupted records. Repeated launch failures grow the disk store beyond the configured cap until another worker completes.
Anchor: defect
Fix: Apply the existing retention sweep after successful launch-failure and reconciliation terminal writes, preserving the held-run and disable protections.

[BLOCK-MERGE-DOWNGRADED] 3a9b1c7
[GPT-REVIEWED] 3a9b1c7

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — harm rung: LOW. Both conditions are confirmed. A refused thread start persists a FAILED record without sweeping (src/kiro_crew/apps/job_sdk.py:834, :860-863). reconcile writes INTERRUPTED records without sweeping (src/kiro_crew/apps/job_sdk.py:1220-1230). The first needs the OS to refuse new threads, which only happens when the host is already out of threads. The second runs once per gateway start, so the extra records number at most the runs that were in flight when the previous process died.
Recovery: the next run of that kind to finish calls _sweep_retention (src/kiro_crew/apps/job_sdk.py:1023), and that sweep looks at every terminal record of the kind and evicts down to the cap (src/kiro_crew/apps/job_sdk_retention.py:73-95). So the overshoot is limited to a handful of records, causes no corruption, and corrects itself on the next completion.
Fix cost: two more sweep call sites. Each has to add the run to _unswept, take _sweep_lock, and follow the _closed/discard ordering on paths that don't follow it today (the start-failure unwind and the reconcile loop). That is new lock-ordering obligations and tests on rare paths, all to stop a temporary, bounded excess of record files. The cost is out of proportion to that harm.

[ADJUDICATION] 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/apps/job_sdk.py:1023 reason=disproportionate-remedy
[GPT-ADJUDICATED] 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3a9b1c74e9a15e1d3a64a7d27b03dd609618b511 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@bolichen97
bolichen97 force-pushed the fix/7588-job-sdk-retention-cap branch from ebab280 to 1183bb0 Compare November 8, 2026 18:59
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97]

  • Sweep completion is published after releasing the sweep lock span=dcc29bf3e1fa

fixed in 1183bb0: sweep_retention now holds _sweep_lock across the sweep and its finally that takes the run out of _unswept, so the next sweep cannot read the finished run as held. Lock order stays _sweep_lock then _lock. A kind with no cap skips the lock and only leaves _unswept.
test_a_sweep_lets_its_run_go_before_the_next_sweep_scans finishes a second run right after the first sweep prunes; it left two records under a cap of one on the parent head and leaves one on the fix.

AI-assisted, posted on behalf of bolichen97

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97]

  • Eviction races the successful start response span=dcc29bf3e1fa

fixed in 1183bb0: the start route calls JobSDK._start_run, which returns a copy of the record taken before the worker exists (or, on adoption, a copy of the live run under the SDK lock). When the read-back finds the record gone, the route answers 200 with that copy instead of 500 run_unreadable. JobSDK.start still returns the run id.
test_a_start_whose_run_the_cap_removed_still_answers_the_run lets a newer run's sweep remove the started run before the read-back; it got 500 on the parent head and 200 with the run on the fix.

AI-assisted, posted on behalf of bolichen97

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97]

  • Second copy of an existing no-follow capped read span=00210a65df9d

fixed in 1183bb0: _read_record_nolink is deleted. JobStore._iter_runs_nolink reads each record through jsonl_util.open_regular_nofollow(path, max_bytes=_MAX_RECORD_BYTES) and skips the file on OSError, so the link, FIFO and size refusal has one copy, with the open-time size check and the per-read charge.
The scan's link, oversized and unparseable record tests pass unchanged on the shared reader.

AI-assisted, posted on behalf of bolichen97

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@bolichen97
bolichen97 force-pushed the fix/7588-job-sdk-retention-cap branch from 1183bb0 to 75dc34c Compare November 8, 2026 19:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
The Job SDK writes one record file per run and removes them only when
the app is disabled, so a kind's store grows with every run.
register() takes an opt-in per-kind retention count: after each
terminal write the sweep removes that kind's oldest finished records
until at most that many remain. It never removes a run still running
or one whose own sweep has not run yet, and it reads each record
without following a link. aws_control's two backup kinds opt in at
100, at or above every reader's window.

The liveness predicate and eviction callback are left to #33058.
Builds on #22711.

Closes #7588
@bolichen97
bolichen97 force-pushed the fix/7588-job-sdk-retention-cap branch from 75dc34c to 3a9b1c7 Compare November 8, 2026 23:56
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97]

  • Accepted SDK records permanently escape the retention cap span=93fe5f5d0d5b

fixed in 3a9b1c7: JobSDK._start_run refuses a dedupe_key longer than _MAX_DEDUPE_KEY (512) characters for a kind registered with retention, before any record is written; the start route maps it to 400 invalid_dedupe_key without quoting the key. An uncapped kind keeps today's behaviour.
Class swept: dedupe_key was the one record field a caller sets with no length bound (params are bounded by _validated_params, error is cut to 2000 chars at both writers). test_the_largest_capped_record_fits_the_scan builds a record with every field at its bound in 12-byte JSON escapes and asserts it stays under a quarter of _MAX_RECORD_BYTES.
test_a_capped_kind_refuses_a_key_its_scan_could_not_read and test_a_capped_kind_answers_400_to_an_overlong_key fail with the check removed and pass with it.

AI-assisted, posted on behalf of bolichen97

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The shard 6 red on 75dc34c (Linux and Windows, plus Coverage Gate) was test/test_dashboard_server_composition_contract.py::test_the_lazy_imports_stay_inside_the_functions_that_need_them (assert 140 == 139), main's, fixed by #33377; 3a9b1c7 is rebased onto main past it and the test passes locally. The aws-control app repository carries the same registration in KiroCrewCaptain/kirocrew-app-aws-control#1.

AI-assisted, posted on behalf of bolichen97

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
KiroCrewCaptain added a commit to KiroCrewCaptain/kirocrew-app-aws-control that referenced this pull request Nov 9, 2026
Kiro Crew's Job SDK gains an opt-in per-kind retention cap
(kirodotdev/KiroCrew#33196). Both backup kinds register with
retention=100, at or above every reader's window, so the nightly
loop's records stop growing forever. A host older than the option
raises TypeError on the keyword, and the kind is then registered
without it, keeping every record as before.

Co-authored-by: KiroCrewCaptain <312312717+KiroCrewCaptain@users.noreply.github.com>
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 9, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head 3a9b1c7: PR Readiness is green, every AI review lane on this head passes with no blocking findings, and no change request stands.

@bolichen97
bolichen97 merged commit ec4cc69 into main Nov 9, 2026
99 checks passed
@bolichen97
bolichen97 deleted the fix/7588-job-sdk-retention-cap branch November 9, 2026 00:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 9, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Decision 6 of #26967 (comment) (an agent-evidenced docs/decisions/ entry is dropped from its PR, its rationale moved to the owning spec) cannot be applied here: this PR merged at 2026-11-09T00:54:07Z, six minutes before that decision was posted, so docs/decisions/2026-11-08-the-job-sdk-retention-cap-ships-with-the-backup-jobs.md is already on main.

Removing a shipped entry is refused by Fast Gate's decision-ledger-history job (scripts/check_decisions_history.py, no exemptions; checked locally: "is GONE ... exit 1"), and docs/decisions/README.md rule 3 says a correction is a superseding entry written by, or quoting, a maintainer. Neither is an agent's call, so nothing was pushed and the entry stays as it is.

Needs a decision from a maintainer: leave the entry as shipped, or supersede it with an entry quoting you. If it goes, the rationale to keep is one bullet in app-kit-platform.md §26.1: the cap ships with its first caller, so a consumer tests its shape; the backup kinds do no work after a run finishes, so a status-only cap is safe for them; live and on_evict wait for #33058.

The frozen Goal / Why it matters / Not a goal sections do not cite the entry (they cite the #7588 comment), so nothing in them changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Job SDK has no retention, and status-only liveness makes eviction unsafe

2 participants