Skip to content

fix(apps): prewarm store art for cold rows before stale ones - #22300

Merged
iamwhatever merged 2 commits into
mainfrom
fix/22090-prewarm-cold-first
Oct 25, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/22090-prewarm-cold-first

Conversation

@cruisercohen

@cruisercohen cruisercohen commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Repeated registry refreshes give every app in a long owner-tier catalog its store art, instead of re-fetching the same apps at the top each time.

For an owner-tier registry that lists apps in private repositories, a not-installed app's icon and screenshots come only from the owner-tier store-art prewarm. An explicit refresh (refresh_registries) expires every row's manifest cache and then prewarms. Right after that expiry _store_assets_warm reads every row as cold, including rows whose art is all in the blob cache, so the batch clones from the top of the index until its 20 s budget runs out. The next refresh does the same. Apps past what one batch reaches never get their art and keep the generic icon in Discover (#22090, with a measured reproduction).

Why it matters

Anyone running an owner-tier registry with more apps than one batch can clone sees a growing tail of apps without icons, and refreshing more often does not help: each refresh spends its budget re-cloning the apps that already have art.

Not a goal

  • Changing what counts as warm, the rewarm age, the refresh's manifest expiry, or which rows get an owner-credentialed clone.
  • The examination cap (_PREWARM_MAX_ROWS): an index longer than the cap still leaves its tail past the cap cold, as documented.
  • The Discover list card's installed-art fallback the reporter mentions; that is a separate issue.

What changed (motivation → approach → change)

  • Symptom: after each refresh the batch clones the same head rows; with 30 rows and a batch that reaches about 12, 18 rows are never warmed (the reporter's table).
  • Cause: the batch queues every not-warm row in index order, and a refresh makes every row not warm, so "has art, manifest only aged out" and "never had art" were queued alike.
  • Approach: the reporter's first direction. Rank the kept rows cold-first: a row whose declared art is all present (or recorded unobtainable) and that is not warm only because its manifest cache aged out goes after every row whose art is missing. Index order is kept inside each group, and stale rows still re-warm when the budget allows, so the refresh's fresh display info still arrives. The second direction (let the stale manifest count as warm) was not taken: for a private repository the prewarm is the only way the renewed manifest arrives, so skipping those rows would leave their display info stale.
  • Change:
    • store_art._declared_art_present: the art check _store_assets_warm already ran, split out without the age bound. _store_assets_warm behaves as before.
    • store_art._stale_row_art_present: reads the row's manifest cache file past its TTL through the same no-follow, size-bounded descriptor the sidecars use (_read_pinned_sidecar) and answers whether its art is all present. It is used only to rank rows; nothing is served or merged from it. Anything missing, refused or malformed, including a body under the size cap that nests deeply enough to raise RecursionError in the JSON decoder, ranks the row with the cold rows. _read_unobtainable_art, which the same check reads for the .unobtainable.json record beside the manifest, catches RecursionError the same way and reads such a record as absent (second commit).
    • _prewarm_owner_tier_store_assets queues cold rows first, then stale rows.
    • Both helpers are declared on the registry facade; the manifest GC docstring in caches.py and the app-kit-platform spec describe the ranking read.

Security: no credential decision changes. The eligible set, the provenance gate, the backoff, the transport allowlist and the SEL grant per clone are untouched; only the order of already-eligible rows changes. The ranking read opens an agent-writable cache file, so it uses the pinned no-follow, size-bounded read; a forged file can at most move an eligible row between the two groups.

Backwards compatibility

Compatible: the batch clones the same rows with the same credentials and writes the same caches; only the order inside one batch changes.

Removes nothing: no row, cache, key or function is removed; the change adds a ranking step.

Tests

test/test_registry_store_art_prewarm_order.py (new):

  • three rows warmed and then expired (as a refresh leaves them) plus two never-warmed rows further down: clone order is app-03, app-04, app-00, app-01, app-02, and every row ends with art;
  • a stale row whose blob-cache art was removed ranks as cold and keeps its index place among the cold rows;
  • a stale manifest the ranking cannot use (nested past the decoder's stack, malformed JSON, not UTF-8, not an object) ranks its row cold and the batch still clones both rows;
  • a nested .unobtainable.json record beside an expired manifest reads as no record, and the batch still runs;
  • the reported case through refresh_registries: six rows, each batch lands two clones and the rest of the batch fails (standing in for the budget cutting it off, with the clone-failure backoff held open so a failed row is retried like a cancelled one; no timing involved). Three refreshes now clone [00, 01], [02, 03], [04, 05] and all six get art.

Failing first: on origin/main 1f520f3 seven of the eight cases fail (the ordering cases and the unreadable-manifest cases, which main does not rank); the nested-manifest case fails with RecursionError on the first commit alone (acf595d), and the nested-record case on the earlier second commit (cddf98b); all eight pass on this head. The refresh case reproduces the report exactly: [00, 01], [00, 01], [00, 01], four rows never warmed.

Also run: the store-art prewarm suites, the registry composition-contract suites and every test_*registry* / test_apps_* / test_*blob* file (3,154 passed, 5 skipped); black, isort and flake8 on the touched files; mypy on store_art.py and the facade; scripts/docs-lint.sh; scripts/check_comment_history.py with COMMENT_HISTORY_BASE_REF=origin/main.

Manual verification

N/A — unit coverage sufficient: the change is the order of one in-process queue, and the refresh test drives the real refresh_registries, _expire_cache_file, warm check and batch budget with only the clone and the index fetch faked.

Related Issues

Fixes #22090

Follow-up: #22359 (the two remaining cache readers without a RecursionError guard).

Pattern harvest

Rule candidate: semgrep
Pattern: json.loads over bytes read from an agent-writable file, inside an except that lists ValueError but not RecursionError (a body under a size cap can still nest past the decoder's stack). _read_clone_failure in store_art.py and _read_manifest_cache in caches.py have the same shape on main; they are tracked in #22359.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license, per CONTRIBUTING.md.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (5 items) — 4 justified

Intent: let repeated registry refreshes give every app in a long owner-tier catalog its store art, instead of re-cloning the same apps at the top of the index each time. This is a FIX (#22090, a test that fails on base).

  1. The prewarm batch now clones apps whose art is missing before apps that are only stale. Order within each group is unchanged. — justified
  2. Each kept row now gets one extra read of its expired manifest cache, used only to rank it. — justified
  3. A deeply nested .unobtainable.json record now reads as no record instead of raising RecursionError. — rides along; the other two unguarded readers are deferred to apps: two registry cache readers let a deeply nested JSON body raise RecursionError instead of reading as absent #22359
  4. The art check is split out of _store_assets_warm into _declared_art_present, which has 2 consumers, and both helpers are declared on the registry facade. — justified
  5. The app-kit-platform spec and the caches.py GC docstring now describe the ranking read past the TTL. — justified

[FIRST-PRINCIPLES-REVIEWED] 9a4aa57

Description read

[DESCRIPTION-READ] cb04ac40a7935dc06723daee7d4b4608ef8b1bd60ab9ecee6400014a90f34f1b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9a4aa57

Verdict parsed from the review's SHA-scoped output markers for commit 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: putting cold rows first fixes the cause in #22090, keeps the eligible set and the credentials the same, and if the ranking read fails, the row keeps the order it had before.

Notes

  • Each examined row now costs one extra file read off the event loop (asyncio.to_thread), up to _PREWARM_MAX_ROWS reads. If that pre-queue cost shows up in the listing path's cache-miss latency, batch it into a single thread call.

[DESIGN-REVIEWED] 9a4aa57

Description read

[DESCRIPTION-READ] d4776f982fe9187d72820ade04a10d3c5e134a170fbbb5a80fa0ac718909c57d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a4aa57

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9a4aa572ad89780f6fe65fb0bf3e9886a052e06b false-positive: <reason, 20 to 500 characters>

@cruisercohen

Copy link
Copy Markdown
Contributor Author

GPT 6.1 finding (store_art.py:716, RecursionError on a deeply nested stale manifest): fixed in cddf98b. _stale_row_art_present catches RecursionError with the other malformed-file errors, so such a row ranks as cold and the batch and the refresh carry on. New case test_an_unreadable_stale_manifest_ranks_the_row_cold_and_the_batch_runs[nested-past-the-decoder-stack] writes 100,000 nested arrays (200 KB, under the cap) into an expired manifest cache; it fails with RecursionError on acf595d and passes on cddf98b.

@cruisercohen
cruisercohen force-pushed the fix/22090-prewarm-cold-first branch from cddf98b to 97a06c3 Compare October 19, 2026 22:31
@cruisercohen

Copy link
Copy Markdown
Contributor Author

GPT 6.1 review of cddf98b, by finding:

  1. store_art.py:723, RecursionError from a nested .unobtainable.json record (blocking, fenced): fixed in 97a06c3. _read_unobtainable_art catches RecursionError with its other malformed-record errors and reads such a record as absent. New case test_an_unreadable_unobtainable_record_does_not_fail_the_ranking writes 100,000 nested arrays into the record beside an expired manifest; it fails with RecursionError on cddf98b and passes on 97a06c3.

  2. test_registry_store_art_prewarm_order.py:136, refresh test racing a 300 ms budget (blocking): fixed in 97a06c3. The test no longer depends on time: each batch lands two clones and the rest of the batch fails, with the clone-failure backoff held open, so the batch order is fixed by the fake alone. It still fails on main with [00, 01] three times.

  3. StartupVideoModal.tsx:394, unrelated frontend label change (non-blocking): this PR touches no file under website/. Its diff against main is five files: the app-kit-platform spec, the registry facade, caches.py, store_art.py and the new test.

@cruisercohen

Copy link
Copy Markdown
Contributor Author

Advisory notes on 97a06c3, by finding. Nothing changed for either.

First Principles, the two other unguarded json.loads reads (_read_manifest_cache in caches.py, _read_clone_failure in store_art.py): both have that shape on main, and this PR does not touch them. The description's Pattern harvest section records them under the semgrep rule candidate for json.loads over an agent-writable file without RecursionError. No open issue tracks them yet.

Design Review, cost of the ranking before the batch budget: measured on 97a06c3 with a throwaway test (not committed) that warms 200 rows, expires every manifest as a refresh does, and runs the real prewarm with clones faked. The 200 ranking reads took 77 to 95 ms in total, and the slowest single read 0.64 ms, against the 20 s batch budget. Rows that are already warm, or backing off after a failed clone, skip the ranking, so on the listing path's cache miss it runs only for rows that will be queued anyway.

@cruisercohen
cruisercohen marked this pull request as ready for review October 19, 2026 23:24
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 19, 2026 23:24
@cruisercohen
cruisercohen enabled auto-merge (squash) October 19, 2026 23:24
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 20, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES (tracking issue needed) at 97a06c3.

The fix looks sound. Ranking runs only after the warm, provenance, backoff and field gates, so the same rows get the same credentials and only the queue order changes (store_art.py L1607-L1619). The ranking read goes through the pinned, size-bounded sidecar read and falls back to the old order on any unreadable file (L698-L725). The GPT, Opus and Design items are answered, and CI is green.

One deferred item has no issue. First Principles asked that the two remaining json.loads reads without a RecursionError guard be tracked: _read_clone_failure, which runs before the ranking on every non-warm row, and _read_manifest_cache. Your reply says no open issue tracks them, and the Pattern harvest rule candidate is not an issue. Please either add RecursionError to those two except clauses here, or open one issue for them and link it in the PR description.

— crew-pr-reviewer (r113)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 20, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Thanks. Opened #22359 for the two remaining readers, _read_clone_failure in store_art.py and _read_manifest_cache in caches.py, and linked it in the description (Related Issues and Pattern harvest). This PR keeps its scope; no code change, and the head is still 97a06c3.

An explicit registry refresh expires every manifest cache before the owner-tier store-art prewarm runs, so every row read as not warm and the batch walked the index from the top. Each refresh re-cloned the same head rows until the budget ran out, and rows further down never got their art. The batch now queues rows whose declared art is missing ahead of rows whose art is cached and whose manifest only aged out, keeping index order inside each group.
… as cold

The prewarm's ranking read opens the agent-writable manifest cache and its .unobtainable.json record. A body under the size cap can nest deeply enough to exhaust the JSON decoder's stack, and the RecursionError escaped the batch and failed the refresh. Both reads catch it with the other malformed-file errors, so the row ranks as cold (manifest) or reads as having no record. The refresh-order test lands two clones per batch and fails the rest instead of racing a 300 ms budget.
@iamwhatever
iamwhatever force-pushed the fix/22090-prewarm-cold-first branch from 97a06c3 to 9a4aa57 Compare October 25, 2026 02:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 25, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 9a4aa57.

The r113 item is cleared: the two remaining readers without a RecursionError guard are tracked in #22359, which names both functions and a done-when, and it is linked in the description. This head is a rebase of 97a06c3 with an identical patch.

The fix is sound. Ranking runs only after the warm, provenance, backoff and field gates, so the batch clones the same rows with the same credentials and only the queue order changes (store_art.py L1649-L1661). The ranking read uses the pinned, size-bounded sidecar read and ranks any unreadable or deeply nested file as cold (L698-L726). Every AI lane on this head is clean or answered, and all checks are green.

— crew-pr-reviewer (r125)

@iamwhatever
iamwhatever merged commit bac0a33 into main Oct 25, 2026
106 checks passed
@iamwhatever
iamwhatever deleted the fix/22090-prewarm-cold-first branch October 25, 2026 07:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 25, 2026
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359

(cherry picked from commit 94de517)
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Three readers of the agent-writable registry cache directory (the
manifest cache, the external registry cache and the clone-failure
record) let RecursionError from json.loads escape. A body nested deeply
enough to exhaust the decoder's stack then dropped a registry from the
App Store listing or ended the store-art prewarm batch, instead of
reading as "no record" as every other malformed body does. Each
reader's except now lists RecursionError, matching the readers #22300
already guarded.

Closes #22359

(cherry picked from commit 94de517)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apps: an explicit registry refresh restarts the owner-tier store-art prewarm from the top, so apps near the end of a long catalog never get their icons

2 participants