Skip to content

Chunk D1 IN lists to stay under the 100-parameter cap - #9

Merged
scosman merged 1 commit into
mainfrom
claude/sharp-cori-trsuh6
Oct 27, 2026
Merged

scosman merged 1 commit into
mainfrom
claude/sharp-cori-trsuh6

Conversation

@scosman

@scosman scosman commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Problem

Production alerts in Nathan-admin kept repeating every hour:

pr_management.sweep failed: D1_ERROR: too many SQL variables at offset 357: SQLITE_ERROR {"repo":"Kiln-AI/Kiln","number":1959}
pr_management.refresh failed: D1_ERROR: too many SQL variables at offset 357: SQLITE_ERROR {"attempts":3}

At the end of every refresh, store.deleteEvents deletes the PR's processed webhook events. It binds repo, number and one id IN (?, …) value per event. Cloudflare D1 allows 100 bound parameters per query, so a PR with 99 or more pending events fails. Offset 357 is exactly where the 101st parameter sits in that query.

Kiln-AI/Kiln#1959 hit this because every check_run and status event is stored. A push to a large CI matrix records hundreds of them within one debounce window. When the delete failed, the events were never removed, so every later sweep and refresh of that PR failed the same way until the 7-day prune.

Fix

  • src/core/db.ts: adds MAX_BOUND_PARAMS = 100 and inListChunks(values, otherParams), a shared helper. It splits an IN list into chunks that fit alongside the statement's other parameters.
  • deleteEvents: deletes in chunks, all in one db.batch, so a PR's events are still consumed together or not at all.
  • openRecords(repos): had the same unbounded repo IN (…). It's used by the sweep, the daily report and the personal queue. It now queries in chunks and merges the rows in the same repo, number order. This one couldn't fail yet because the tracked repo list is short.
  • Rest of the codebase: no other query builds a variable-length list.

Tests

  • New regression tests for refresh, the sweep and the store use 250 events or repos. All four failed before the fix with the exact production error.
  • A test in test/core/db.test.ts checks that local D1 accepts 100 parameters and rejects 101, like production. If that ever changed, the regression tests would stop testing anything.
  • npm run check passes: lint, typecheck, config, imports, and 810 tests.

After deploy

The first hourly sweep should clear #1959's backlog of events. Nothing needs to be done by hand.


Generated by Claude Code

The hourly sweep and refresh jobs failed for Kiln-AI/Kiln#1959 with
"D1_ERROR: too many SQL variables". After each refresh,
store.deleteEvents removes the PR's consumed webhook events by binding
repo, number and one id per event. A push to a large CI matrix records
hundreds of check_run events within one debounce window, which goes over
D1's 100-parameter limit. The events were never consumed, so every later
sweep and refresh of the PR failed the same way until the 7-day prune.

- core/db: add MAX_BOUND_PARAMS and inListChunks, a shared helper that
  splits an IN list into chunks that fit alongside a statement's other
  parameters.
- deleteEvents: delete in chunks, all in one batch, so a PR's events are
  consumed together or not at all.
- openRecords: query the repo IN list in chunks and merge the rows in
  the order ORDER BY repo, number gave.

No other query builds a variable-length list. Regression tests (refresh
job, sweep, store) use 250 values, and a db test checks that local D1
enforces the same cap as production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HdZSuxk92ytcUGW4mSYZyR
@scosman
scosman merged commit 8567018 into main Oct 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant