Repository navigation
fix(apps): prewarm store art for cold rows before stale ones - #22300
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (5 items) — 4 justifiedIntent: let repeated registry refreshes give every app in a long owner-tier catalog its store art, instead of re-cloning the same apps at the top of the index each time. This is a FIX (#22090, a test that fails on base).
[FIRST-PRINCIPLES-REVIEWED] 9a4aa57 Description read[DESCRIPTION-READ] cb04ac40a7935dc06723daee7d4b4608ef8b1bd60ab9ecee6400014a90f34f1b That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: putting cold rows first fixes the cause in #22090, keeps the eligible set and the credentials the same, and if the ranking read fails, the row keeps the order it had before. Notes
[DESIGN-REVIEWED] 9a4aa57 Description read[DESCRIPTION-READ] d4776f982fe9187d72820ade04a10d3c5e134a170fbbb5a80fa0ac718909c57d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
GPT 6.1 finding (store_art.py:716, RecursionError on a deeply nested stale manifest): fixed in cddf98b. |
cddf98b to
97a06c3
Compare
|
GPT 6.1 review of cddf98b, by finding:
|
|
Advisory notes on 97a06c3, by finding. Nothing changed for either. First Principles, the two other unguarded Design Review, cost of the ranking before the batch budget: measured on 97a06c3 with a throwaway test (not committed) that warms 200 rows, expires every manifest as a refresh does, and runs the real prewarm with clones faked. The 200 ranking reads took 77 to 95 ms in total, and the slowest single read 0.64 ms, against the 20 s batch budget. Rows that are already warm, or backing off after a failed clone, skip the ranking, so on the listing path's cache miss it runs only for rows that will be queued anyway. |
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES (tracking issue needed) at 97a06c3.
The fix looks sound. Ranking runs only after the warm, provenance, backoff and field gates, so the same rows get the same credentials and only the queue order changes (store_art.py L1607-L1619). The ranking read goes through the pinned, size-bounded sidecar read and falls back to the old order on any unreadable file (L698-L725). The GPT, Opus and Design items are answered, and CI is green.
One deferred item has no issue. First Principles asked that the two remaining json.loads reads without a RecursionError guard be tracked: _read_clone_failure, which runs before the ranking on every non-warm row, and _read_manifest_cache. Your reply says no open issue tracks them, and the Pattern harvest rule candidate is not an issue. Please either add RecursionError to those two except clauses here, or open one issue for them and link it in the PR description.
— crew-pr-reviewer (r113)
An explicit registry refresh expires every manifest cache before the owner-tier store-art prewarm runs, so every row read as not warm and the batch walked the index from the top. Each refresh re-cloned the same head rows until the budget ran out, and rows further down never got their art. The batch now queues rows whose declared art is missing ahead of rows whose art is cached and whose manifest only aged out, keeping index order inside each group.
… as cold The prewarm's ranking read opens the agent-writable manifest cache and its .unobtainable.json record. A body under the size cap can nest deeply enough to exhaust the JSON decoder's stack, and the RecursionError escaped the batch and failed the refresh. Both reads catch it with the other malformed-file errors, so the row ranks as cold (manifest) or reads as having no record. The refresh-order test lands two clones per batch and fails the rest instead of racing a 300 ms budget.
97a06c3 to
9a4aa57
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at 9a4aa57.
The r113 item is cleared: the two remaining readers without a RecursionError guard are tracked in #22359, which names both functions and a done-when, and it is linked in the description. This head is a rebase of 97a06c3 with an identical patch.
The fix is sound. Ranking runs only after the warm, provenance, backoff and field gates, so the batch clones the same rows with the same credentials and only the queue order changes (store_art.py L1649-L1661). The ranking read uses the pinned, size-bounded sidecar read and ranks any unreadable or deeply nested file as cold (L698-L726). Every AI lane on this head is clean or answered, and all checks are green.
— crew-pr-reviewer (r125)
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359 (cherry picked from commit 94de517)
Three readers of the agent-writable registry cache directory (the manifest cache, the external registry cache and the clone-failure record) let RecursionError from json.loads escape. A body nested deeply enough to exhaust the decoder's stack then dropped a registry from the App Store listing or ended the store-art prewarm batch, instead of reading as "no record" as every other malformed body does. Each reader's except now lists RecursionError, matching the readers #22300 already guarded. Closes #22359 (cherry picked from commit 94de517)
Problem / Motivation
Goal: Repeated registry refreshes give every app in a long owner-tier catalog its store art, instead of re-fetching the same apps at the top each time.
For an owner-tier registry that lists apps in private repositories, a not-installed app's icon and screenshots come only from the owner-tier store-art prewarm. An explicit refresh (
refresh_registries) expires every row's manifest cache and then prewarms. Right after that expiry_store_assets_warmreads every row as cold, including rows whose art is all in the blob cache, so the batch clones from the top of the index until its 20 s budget runs out. The next refresh does the same. Apps past what one batch reaches never get their art and keep the generic icon in Discover (#22090, with a measured reproduction).Why it matters
Anyone running an owner-tier registry with more apps than one batch can clone sees a growing tail of apps without icons, and refreshing more often does not help: each refresh spends its budget re-cloning the apps that already have art.
Not a goal
_PREWARM_MAX_ROWS): an index longer than the cap still leaves its tail past the cap cold, as documented.What changed (motivation → approach → change)
store_art._declared_art_present: the art check_store_assets_warmalready ran, split out without the age bound._store_assets_warmbehaves as before.store_art._stale_row_art_present: reads the row's manifest cache file past its TTL through the same no-follow, size-bounded descriptor the sidecars use (_read_pinned_sidecar) and answers whether its art is all present. It is used only to rank rows; nothing is served or merged from it. Anything missing, refused or malformed, including a body under the size cap that nests deeply enough to raiseRecursionErrorin the JSON decoder, ranks the row with the cold rows._read_unobtainable_art, which the same check reads for the.unobtainable.jsonrecord beside the manifest, catchesRecursionErrorthe same way and reads such a record as absent (second commit)._prewarm_owner_tier_store_assetsqueues cold rows first, then stale rows.caches.pyand the app-kit-platform spec describe the ranking read.Security: no credential decision changes. The eligible set, the provenance gate, the backoff, the transport allowlist and the SEL grant per clone are untouched; only the order of already-eligible rows changes. The ranking read opens an agent-writable cache file, so it uses the pinned no-follow, size-bounded read; a forged file can at most move an eligible row between the two groups.
Backwards compatibility
Compatible: the batch clones the same rows with the same credentials and writes the same caches; only the order inside one batch changes.
Removes nothing: no row, cache, key or function is removed; the change adds a ranking step.
Tests
test/test_registry_store_art_prewarm_order.py(new):app-03, app-04, app-00, app-01, app-02, and every row ends with art;.unobtainable.jsonrecord beside an expired manifest reads as no record, and the batch still runs;refresh_registries: six rows, each batch lands two clones and the rest of the batch fails (standing in for the budget cutting it off, with the clone-failure backoff held open so a failed row is retried like a cancelled one; no timing involved). Three refreshes now clone[00, 01], [02, 03], [04, 05]and all six get art.Failing first: on origin/main 1f520f3 seven of the eight cases fail (the ordering cases and the unreadable-manifest cases, which main does not rank); the nested-manifest case fails with
RecursionErroron the first commit alone (acf595d), and the nested-record case on the earlier second commit (cddf98b); all eight pass on this head. The refresh case reproduces the report exactly:[00, 01], [00, 01], [00, 01], four rows never warmed.Also run: the store-art prewarm suites, the registry composition-contract suites and every
test_*registry*/test_apps_*/test_*blob*file (3,154 passed, 5 skipped); black, isort and flake8 on the touched files; mypy onstore_art.pyand the facade;scripts/docs-lint.sh;scripts/check_comment_history.pywithCOMMENT_HISTORY_BASE_REF=origin/main.Manual verification
N/A — unit coverage sufficient: the change is the order of one in-process queue, and the refresh test drives the real
refresh_registries,_expire_cache_file, warm check and batch budget with only the clone and the index fetch faked.Related Issues
Fixes #22090
Follow-up: #22359 (the two remaining cache readers without a
RecursionErrorguard).Pattern harvest
Rule candidate: semgrep
Pattern:
json.loadsover bytes read from an agent-writable file, inside anexceptthat listsValueErrorbut notRecursionError(a body under a size cap can still nest past the decoder's stack)._read_clone_failureinstore_art.pyand_read_manifest_cacheincaches.pyhave the same shape on main; they are tracked in #22359.Checklist
Contribution License Agreement
Contributed under the project's Apache-2.0 license, per CONTRIBUTING.md.