Repository navigation
Chunk D1 IN lists to stay under the 100-parameter cap - #9
Merged
Merged
Conversation
The hourly sweep and refresh jobs failed for Kiln-AI/Kiln#1959 with "D1_ERROR: too many SQL variables". After each refresh, store.deleteEvents removes the PR's consumed webhook events by binding repo, number and one id per event. A push to a large CI matrix records hundreds of check_run events within one debounce window, which goes over D1's 100-parameter limit. The events were never consumed, so every later sweep and refresh of the PR failed the same way until the 7-day prune. - core/db: add MAX_BOUND_PARAMS and inListChunks, a shared helper that splits an IN list into chunks that fit alongside a statement's other parameters. - deleteEvents: delete in chunks, all in one batch, so a PR's events are consumed together or not at all. - openRecords: query the repo IN list in chunks and merge the rows in the order ORDER BY repo, number gave. No other query builds a variable-length list. Regression tests (refresh job, sweep, store) use 250 values, and a db test checks that local D1 enforces the same cap as production. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HdZSuxk92ytcUGW4mSYZyR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production alerts in Nathan-admin kept repeating every hour:
At the end of every refresh,
store.deleteEventsdeletes the PR's processed webhook events. It bindsrepo,numberand oneid IN (?, …)value per event. Cloudflare D1 allows 100 bound parameters per query, so a PR with 99 or more pending events fails. Offset 357 is exactly where the 101st parameter sits in that query.Kiln-AI/Kiln#1959 hit this because every
check_runandstatusevent is stored. A push to a large CI matrix records hundreds of them within one debounce window. When the delete failed, the events were never removed, so every later sweep and refresh of that PR failed the same way until the 7-day prune.Fix
src/core/db.ts: addsMAX_BOUND_PARAMS = 100andinListChunks(values, otherParams), a shared helper. It splits an IN list into chunks that fit alongside the statement's other parameters.deleteEvents: deletes in chunks, all in onedb.batch, so a PR's events are still consumed together or not at all.openRecords(repos): had the same unboundedrepo IN (…). It's used by the sweep, the daily report and the personal queue. It now queries in chunks and merges the rows in the samerepo, numberorder. This one couldn't fail yet because the tracked repo list is short.Tests
test/core/db.test.tschecks that local D1 accepts 100 parameters and rejects 101, like production. If that ever changed, the regression tests would stop testing anything.npm run checkpasses: lint, typecheck, config, imports, and 810 tests.After deploy
The first hourly sweep should clear #1959's backlog of events. Nothing needs to be done by hand.
Generated by Claude Code