Repository navigation
Corpus-b flake: large-command-batches overflow polls exhaust PHP reference lock-pressure retries #1194
Description
What fails
Shared Server fixtures → execution-and-codec (corpus-b) intermittently fails before comparison:
RuntimeException: Large command batches: no workflow task for select_member_overflow {"http_status":503,…,"poll_retries":15,"poll_status":"backend_lock_pressure","reason":"backend_lock_pressure",…}
The poll that large-command-batches-probe.php makes against the published PHP SQLite reference keeps getting 503 backend_lock_pressure until lockPressurePollRetryDelay (scripts/conformance/server-parity/lock-pressure-retry.php, 15 retries / 30 s) gives up. The PHP reference shares its SQLite file with its maintenance loop, and the large batches (up to 1,000 commands) appear to hold the file long enough that the budget runs out.
Evidence (merge queue, 2026-11-13; 3 of the last ~10 groups)
| Group | Run | Case |
|---|---|---|
| pr-1187 on 4c5f226 | 43345755536 job 131626581193 | select_member_overflow |
| pr-1189 on 4c5f226 | 43349737218 job 131639640905 | select_member_overflow |
| pr-1186 | 43349740361 job 131639561736 | select_base_overflow |
The PRs in those groups don't touch the corpus, the probe or the server code it exercises (#1187 is a Python benchmark script, #1189 adds a Rust test, #1186 bounds command names). Every earlier group and every main push in the same window passed. All three were re-queued unchanged.
Acceptance
- Find what holds the reference's SQLite file during the overflow cases (maintenance pass vs. the probe's own batch writes) and remove the contention at its source. For example, finish or quiesce the reference maintenance loop around the large-batch cases, or poll each case's task after its namespace's writes settle. Don't just raise the retry budget blindly.
- If the budget does change, keep it bounded and reported (
lock_pressure_poll_retrieson stderr), and keep recorders seeing only the final reply. - Native recordings are unaffected and must stay unchanged.
- Evidence: 20 consecutive green corpus-b runs, or a deterministic reproduction that the fix closes.
Activity
rmcdaniel commentedon Nov 13, 2026
Evidence on #1195 (head 5b3ff71, which terminates each large-command-batches run as soon as it's recorded): run 43353355060's
execution-and-codec (corpus-b)passed on attempts 1–5 (jobs 131651317384, 131658917257, 131662725363, 131666891832, 131670037451). Attempt 6 is running. Five greens against the observed ~3-in-10 group failure rate is suggestive, not proof (~17% chance unfixed). After merge, every merge-queue group runs corpus-b, so I'll keep counting there and close this at 20 consecutive green corpus-b runs, or reopen with the quarantine fallback on the first lock-pressure failure.rmcdaniel commentedon Nov 13, 2026
Attempt 6 of run 43353355060 also passed
execution-and-codec (corpus-b)(job 131675499385): 6 of 6 green on #1195's head 5b3ff71. That's about 12% likely by chance at the unfixed ~3-in-10 rate. I'll keep counting in merge-queue groups after #1195 merges.rmcdaniel commentedon Nov 13, 2026
#1195 merged 2026-11-13 04:57Z (f294214). From here I count
execution-and-codec (corpus-b)in merge-queue groups and close this at 20 consecutive greens, counting the 6 on #1195's head. The first lock-pressure failure reopens this with the quarantine fallback.rmcdaniel commentedon Nov 13, 2026
I audited actual corpus-b jobs after #1195. The verified gate is 8/20, not 20:
execution-and-codec; their green result summaries are not corpus-b executions and do not count.Owner: Server. Next action: continue the existing exact-head #1195 corpus-b qualification, one job at a time, until the remaining 12 qualifying successes are recorded. Any original lock-pressure recurrence stops the count and reopens the source investigation. I am keeping unrelated #1202 retained-receipt comparator failures separate; those failed jobs are not counted as green evidence. This issue stays open until its gate is met.
rmcdaniel commentedon Nov 13, 2026
The existing exact-head qualification has advanced to10/20: attempt7 of43353355060 passed job131831498524, and attempt8 passed job131836811910. These add to the six original attempts and two audited actual merge-group corpus-b jobs. Attempt9 is starting; one job at a time, same head5b3ff717, no changed retry budget. The20-success gate remains open, and any original lock-pressure recurrence stops qualification.
rmcdaniel commentedon Nov 13, 2026
The existing exact-head qualification is now12/20. Attempt9 passed corpus-b job131841336506 and attempt10 passed job131846481354; these add to the prior10 recorded successes. Same5b3ff717 head, one job at a time, unchanged retry budget. Attempt11 starts next. No skipped merge-group summary, unrelated failed comparator or incomplete local control is counted. The20-success gate remains open.
rmcdaniel commentedon Nov 13, 2026
Qualification is14/20: attempt11 passed corpus-b job131851723330 and attempt12 passed job131858531842, on the same5b3ff717 head. The two audited actual merge-group jobs are included; skipped queue summaries and incomplete controls are excluded. Attempt13 starts next, one job at a time with unchanged retry budget. No original lock-pressure recurrence observed; gate stays open until20.
rmcdaniel commentedon Nov 13, 2026
The exact-head corpus-b gate is 16/20, with no original lock-pressure recurrence in the counted passes. Newly completed exact-head jobs: attempt 13 / job 131864339951 and attempt 14 / job 131871474079 of run 43353355060. Both actual corpus-b jobs concluded success; the source remains
5b3ff717a27e301f6b6067bf5efa555333093b67.The count is 14 successful original/extra attempts plus the two previously audited actual merge-group corpus-b executions. Skipped execution lanes, incomplete controls and infrastructure-blocked runs are excluded. I started attempt 15 by rerunning only the completed corpus-b job; no retry budget, source or coverage changed. Owner server continues the same gate to 20 and will stop/investigate if the original failure recurs. #1204 remains a separate open MariaDB reliability investigation; no pool-timeout fix is inferred.
rmcdaniel commentedon Nov 13, 2026
The exact-head corpus-b gate is 18/20. Attempts 15 / job 131875845316 and 16 / job 131884069614 of 43353355060 both completed successfully, with source still
5b3ff717a27e301f6b6067bf5efa555333093b67. No original lock-pressure recurrence appeared in the counted executions.The count is 16 successful original/extra attempts plus the two already audited actual merge-group corpus-b jobs. Skipped, incomplete and infrastructure-blocked controls stay excluded. I started attempt 17 by rerunning only the successful corpus-b job; retry limits, source and coverage are unchanged. Owner server retains the final two passes and will stop/investigate on recurrence. Separate #1204 remains open; this is not a pool-timeout fix.
rmcdaniel commentedon Nov 13, 2026
20/20 execution gate complete
Fresh individual job/step reads confirm all 20 counted corpus-b executions succeeded, including their actual execute/compare steps. The source fix is merged through #1195 (
f294214a). Eighteen attempts of run 43353355060 remain on exact qualified head5b3ff717a27e301f6b6067bf5efa555333093b67; the other two are the already audited actual merge-group executions. No original overflow-poll lock-pressure recurrence appeared in these 20 executions.The green summaries of merge groups 43378323976 and 43381127748 skipped execution and are excluded. Incomplete/infrastructure-blocked controls remain excluded, and retry limits/source/coverage were unchanged. The bulk historical attempt/all-jobs API returned HTTP 502 during final revalidation; the successful individual job reads above provide the explicit evidence instead. No additional execution was launched because of that read failure.
This completes this issue’s requested gate. It does not fix or close separate #1204, and the installed #216/#228 client cohorts remain separate. Reopen this record on recurrence.