Repository navigation
[Bug]: responses-state-spill/ grows unbounded on a busy hub — 830 files / 270MB in ~2 days, no eviction, restart does not drain (2.58.0) #5369
Description
Description
On a long-running Linux hub (systemd user service, ocx start --port 10100, moderate multi-client traffic), the durable spill directory ~/.opencodex/responses-state-spill/ grows without any apparent retention or eviction. The main responses-state.json stays roughly stable (~24MB) while the spill directory accumulates one *.spill.json file per response and is never cleaned up. A full service restart does not drain or prune it.
Environment
- opencodex 2.58.0 (npm
@bitkyc08/opencodex), Linux x64, bun runtime - Hub deployment: systemd user unit, data-plane port + management plane, 4–6 chained clients, mix of Codex/CLI traffic
- Upgrade path: 2.42.0 → 2.58.0 on 2026-09-18 (the fleet ran 2.42.0 before with no spill growth: the directory did not exist at any noticeable size before the upgrade)
Observed timeline (from daily ~/.opencodex archives)
| Date | spill dir size | file count |
|---|---|---|
| 2026-09-18 | ~0 (not present in top-10 du) | — |
| 2026-09-19 | 49MB | ~150 |
| 2026-09-20 | 270MB | 830 |
File naming pattern: resp_<hash>.<hash>.<hash>.<int>.<bytes>.spill.json, one per response, avg ~325KB. responses-state.json meanwhile: 17MB → 24MB → 24MB (stable — growth moved to the spill dir).
Reproduction / evidence
- Let the hub serve normal traffic for ~2 days on 2.58.0.
du -sh ~/.opencodex/responses-state-spill→ 270MB,ls | wc -l→ 830.systemctl --user restart opencodex-proxy, wait 60s.- Re-measure: still 270MB / 830 files. Zero
spill/drainlines inservice.logbefore or after restart.
Expected behavior
v2.39.0 shipped "Bound the durable spill directory with an aggregate byte cap" (#3097), and v2.58.0 shipped "enforce the shutdown drain reserve on the injected spill clock" (#4833). Either an aggregate cap or a shutdown/periodic drain should bound this directory. Neither appears to apply to these files on a hub that stays up and keeps serving traffic.
Impact
- Daily config archives of
~/.opencodexballooned from ~1MB to 107MB (compressed) in three days; storage cost compounds with no ceiling. ~/.opencodexreached 314MB total on day 3. Left alone this is a slow disk leak on any always-on deployment.
Notes
- Related history: [Bug] responses-state.json full-snapshot rewrite on a 2s debounce causes 34–137 GB/day of disk writes under agent traffic #1385 (responses-state write amplification) and Opencodex aggresive disk writtings on %USER%/.opencode/responses-state.json #3141 were about write storms on the main file; this is about unbounded retention of spilled per-response files.
- Possibly related: [Bug][Windows] Continuation spill failures accumulate behind healthy readiness after #3011 (observed on 2.39.0) #3522 (spill accumulation on Windows observed on 2.39.0).
- We have not manually pruned anything yet; happy to test a fix or a safe manual cleanup procedure on this hub.
Activity
Ingwannu commentedon Sep 21, 2026
I checked the shipped 2.58.0/current implementation rather than inferring from the directory size. The response-state store already has three independent retention bounds:
MAX_SPILLED_RESPONSE_BYTES = 1 GiBMAX_STORED_RESPONSES = 1000RESPONSE_TTL_MS = 24hThe spill budget is enforced on mutation, lazy restart load, and the periodic sweep. A restart intentionally preserves referenced spill files because they are the durable continuation state; “restart did not drain it” is therefore not itself a failure.
The reported 270 MiB / 830 files is below both hard ceilings, so the current evidence does not establish unbounded growth. I am leaving this open for one bounded follow-up rather than closing it as invalid: please re-measure after the oldest cohort has exceeded 24 hours and report whether the directory exceeds 1 GiB, exceeds 1000 owned entries, or retains an owned entry whose response
createdAtis older than the TTL. File mtime alone is not sufficient because a live response can be rewritten/referenced.If none of those boundaries is crossed, the behavior is bounded but operationally surprising; that would be a diagnostics/documentation request rather than a disk-leak bug. Please do not manually delete individual spill files while their entries remain referenced in
responses-state.json.tizerluo commentedon Sep 21, 2026
Re-measured on the same hub after the oldest cohort exceeded the 24h TTL. None of the three boundaries is crossed:
responses-state.json(cap 1000). The other 1045 files on disk are unreferenced.response createdAtages range 3.3–3.9 h — none over 24 h.The sweep is demonstrably working: every file from the initial 09-19 cohort (~49 MiB, observed before any of this reporting) is gone; the oldest surviving file's mtime is now ~38 h, and the age histogram is 845 files in 0–24 h / 254 files in 24–48 h / 0 files beyond 48 h. So the earlier 0 → 49 → 270 → 332 MiB curve was ramp-up toward a traffic-dependent plateau (~last 38 h of responses), not unbounded growth. I'll take the correction on "restart should drain" as well — understood that preserving referenced spill across restart is the durable continuation state.
The operationally surprising part, for whatever the diagnostics/docs follow-up is worth: ~95% of the directory's bytes are unreferenced files that linger up to ~38 h before the sweep removes them. On a busy always-on hub that's a ~330 MiB steady-state footprint of dead weight (and it dominates anything that archives
~/.opencodexwholesale). If unreferenced spill files have no continuation value, sweeping them on the same 24 h cadence as ownership (or on the mutation-path budget check) would shrink the footprint several-fold without touching any owned entry.No manual deletion performed, per your note. Happy to re-test if a candidate build changes the sweep cadence.
fix(responses): resume periodic spill sweep past the first scan window (
perf(management): cache the spill report on /api/system/memory (#5369)
devin-ai-integration commentedon Sep 25, 2026
New matched pull request: #5825 (would resolve this issue) — fix: P0 #5369 spill growth + P1 bug batch [
priority: P0]fix(responses): keep the periodic spill sweep's directory iterator op…
fix(doctor): do not report a truncated spill scan as clean (#5369)
fix: P0 #5369 spill growth + P1 bug batch (#5421, #5008, #4191, #3522, …
lidge-jun commentedon Sep 25, 2026
Fixed by #5825 (22b22ae): orphaned spill files are now reclaimed on the liveness tick with a bounded sweep, and the snapshot budget keeps spill stubs and tombstones ahead of resident entries, so their files are no longer orphaned after a restart.
ocx doctorand/api/system/memorynow report spill directory usage too.