[Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process

Client or integration

GitHub Actions — Cross-platform CI (ci.yml), Linux test N/4 shards

Area

Service lifecycle

Summary

Linux test shards intermittently stop producing output mid-run and are killed at the 15-minute job timeout. The run reports cancelled, not failure, so it reads as infrastructure noise — but it is reproducible enough across unrelated branches to be worth tracking rather than rerunning past.

Filing this because I hit it five times today while doing maintainer triage and kept treating it as flake. That was the wrong call: the hang is consistent in shape, and cancelled runs are easy to rerun into green without anyone noticing the underlying problem.

Observed pattern

Every occurrence looks the same:

the shard runs normally for ~30-90 seconds and passes hundreds of tests;
output stops abruptly right after a test that starts a proxy server;
nothing further is logged for ~14 minutes;
the runner cancels at the 15-minute timeout;
cleanup reports Terminate orphan process: pid (NNNN) (bun).

Across four runs on four unrelated branches, the affected shard varies, which argues against a single bad test:

Run	Branch	Shard	Result
31263738953	`codex/260808-1185-windows-shard-assertion`	`test 4/4`	cancelled (twice, incl. rerun)
31255199569	`fix/windows-powershell-popup`	`test 2/4`	cancelled, green on retry
31258815611	`codex/260808-1195-unbound-quota-unknown`	`test 3/4`	cancelled, green on retry
31152916419	`agent/test-windows-ci-shard-command`	`test 3/4`	failure — `EEXIST: epoll_ctl` in a Bun `WriteStream`

It also occurred on dev itself with no PR involved (runs 31259450263, 31259447622), so it is not contributor-branch specific.

The one non-cancelled instance is the useful clue

Run 31152916419 did not hang — it crashed outright while loading tests/autostart-health.test.ts:

# Unhandled error between tests
error: EEXIST: file already exists, epoll_ctl
      at new WriteStream (internal:fs/streams:244:58)
# collateral:
error: Cannot call describe() after the test run has completed

EEXIST from epoll_ctl means a file descriptor was registered with the event loop twice. That is the same class of resource-lifecycle problem a hang would produce if the registration deadlocked instead of erroring, which suggests the hang and the crash are one bug with two outcomes rather than two problems.

Last log line before the stall

In 31263738953 (test 4/4), the final output before 14 minutes of silence:

[web-search-loop] done — 1 search, 2 iterations, 13ms
[web-search-loop] cancelled — 1 real searches, 0 placeholders, 13ms
(pass) routed Claude requests give OpenAI sidecars main auth without leaking it to the routed provider

That test is in tests/claude-messages-endpoint.test.ts, and the stall is after it passes — so the suspect is teardown or the next file's setup, not the assertion. In the other runs the stall also follows a test that had just started a proxy listener.

Not reproducible locally

bun test tests/claude-messages-endpoint.test.ts → 38 pass / 0 fail in 2.7s on macOS. Full bun run test → 10009 pass / 0 fail. So this needs the sharded Linux runner environment; a single-file local run will not show it.

Why this matters beyond the noise

cancelled is not failure, and the natural response is to rerun. That works — three of the four went green on retry. The risk is that a genuine hang introduced by a future change is indistinguishable from this, and gets rerun into green by whoever is on triage that day.

Suggested starting points

The orphan is always bun, and the stall always follows a proxy-server test, so an un-awaited server close or an fd registered twice across test-file boundaries is the first place to look.
--isolate is already used; whether shard boundaries share any process-global listener state is worth checking.
Raising the timeout would hide it. A shard that stops emitting for 14 minutes has already failed.

Version

dev at 14e948525.

Operating system

ubuntu-latest GitHub-hosted runners.

_{Automated translation bookkeeping — detected language: unknown.}

Two more occurrences today, and one of them is the first case where two shards hung in the same run.

Run 31249781400 on PR #1256:

test 4/4  cancelled  18:44:24Z → 18:59:39Z   (15m15s)
test 3/4  cancelled  18:42:18Z → 18:57:33Z   (15m15s)

Both hit the 15-minute job timeout to the second, which is the same signature as every prior instance. Also #1301's run 31263738953, test 4/4, 15:28:02Z → 15:43:17Z — likewise 15m15s.

Updated tally: eight occurrences across six branches plus dev itself, with the affected shard varying between 2/4, 3/4 and 4/4. That two shards can hang in a single run makes a single bad test even less likely as the explanation.

The 15m15s figure is worth noting on its own: the job timeout is 15 minutes, so these are not tests that run long and finish late. The shard stops producing output and the runner kills it at the deadline — consistent with the orphaned bun process reported in cleanup rather than with slow work.

Practical effect on triage today: I approved and reran a batch of contributor PRs, and this is now the main source of non-green results among them. Six came back green; #1256 needed a rerun purely for this. Anyone doing PR triage will keep hitting it, and the risk stays what it was — a genuine hang from a real change is currently indistinguishable from this one, and reruns make both go away.

A repeat at the same file, which narrows this considerably.

Run 31274685166 on PR #1301, test 3/4, 19:32:43Z → 19:47:59Z (15m16s). The last output before the silence:

##[group]tests/cli-native-profile.test.ts:
...
(pass) ocx account main > a stalled heartbeat is aborted before lease expiry, kills login, and cancels staging
##[endgroup]

##[group]tests/cli-restart-health.test.ts:
killed 1 dangling process          ← 19:33:18Z
                                    ← 14 minutes of nothing
##[error]The operation was canceled. ← 19:47:55Z
Terminate orphan process: pid (2482) (bun)

That is the second time the stall lands at tests/cli-restart-health.test.ts, immediately after cli-native-profile and immediately after that file logs killed 1 dangling process. The first was run 31152916419 (test 3/4, PR #1185's original head), which showed the identical sequence — including the same killed 1 dangling process line as the last thing emitted.

Earlier occurrences stalled after other proxy-starting tests, which is why I described the shard as varying. But the file repeating twice with the same preceding line is a stronger signal than the shard number varying, and it suggests a more specific place to look than "some proxy test":

cli-native-profile spawns login children and cancels staging; cli-restart-health then kills a dangling process at startup.
The orphan reported in cleanup is always bun.
EEXIST: epoll_ctl in run 31152916419 was a descriptor registered twice — plausibly the same lifecycle fault when it errors rather than deadlocks.

Suggested next step for whoever picks this up: run bun test --isolate tests/cli-native-profile.test.ts tests/cli-restart-health.test.ts in that order under Linux, repeatedly, and watch whether a child from the former survives into the latter's dangling-process kill.

Tally is now nine occurrences. Not rerunning #1301 past it — I would rather this one stay red as evidence than be retried into green.

Retracting the file-pair narrowing in my previous comment. It was wrong.

I claimed run 31152916419 showed the same cli-native-profile → cli-restart-health → killed 1 dangling process sequence as 31274685166. It does not. Checking its log again:

31152916419 hit EEXIST: ... epoll_ctl while loading tests/autostart-health.test.ts at 07:05:53, continued running, and finished as 1 fail / 2 errors across 2143 tests in 85 seconds. It contains no killed 1 dangling process line and was never cancelled. It is a failure, not a hang.
31274685166 (test(ci): bind the Windows shard assertion to an executable command (#1185) #1301) reports the same Bun epoll_ctl error while loading tests/api-storage.test.ts, then reaches the CLI pair and stalls there.
My earlier hang capture (31263738953, also test(ci): bind the Windows shard assertion to an executable command (#1185) #1301) stalled after tests/claude-messages-endpoint.test.ts.

So three runs, three different first-affected files. The pair I pointed at is one observation, not a repeated signature, and I built a hypothesis on a comparison I had not actually made.

What survives, stated at the strength the evidence supports:

a shared symptom — Bun EEXIST on epoll_ctl, a descriptor registered with the event loop twice — appearing across unrelated files;
two distinct outcomes from it: the process errors and continues (a normal test failure), or it deadlocks and the runner kills it at 15 minutes with an orphan bun;
no file-level culprit identified.

That is weaker than what I posted, and the earlier comment should be read as retracted rather than refined. My "the shard varies, so it is not one bad test" framing was also imprecise for a different reason — sharding distributes files differently per run, so a varying shard number never argued against a single file.

Apologies for the noise; better to correct it here than leave a wrong lead in the issue for whoever picks it up.

Correcting my retraction — I named the wrong file for one run while retracting a claim about files, and overstated causation while correcting an overstatement.

Two different questions, which I had collapsed into one column. Where the EEXIST first appears is not where the output stops:

Run	First logged `EEXIST` under	Last output before silence	Outcome
`31152916419`	`tests/autostart-health.test.ts`	— (no silence)	kept running, finished 1 fail / 2 errors in 85s
`31263738953`	`tests/baseten-provider.test.ts`	`tests/claude-messages-endpoint.test.ts`	hung, cancelled at 15m
`31274685166`	`tests/api-storage.test.ts`	`tests/cli-restart-health.test.ts`	hung, cancelled at 15m

My retraction above said claude-messages-endpoint was the first affected file in 31263738953. It is the last file before the silence; the first EEXIST in that run is under baseten-provider. Neither column repeats across the three runs.

And the causal claim was still too strong. I wrote that the EEXIST has "two outcomes — error and continue, or deadlock and get killed". The logs show co-occurrence, not causation: the error appears in all three runs, one of which completed normally as a failure. Whether it causes the hang, shares a cause with it, or is incidental is not established.

The honest residue, third attempt:

Bun EEXIST on epoll_ctl — a descriptor registered with the event loop twice — appears in all three runs, under a different file each time.
Two of the three then stop emitting and are killed at the 15-minute job timeout, leaving an orphan bun.
No file-level culprit, and no established causal link between the two facts.

Three passes to state three sentences accurately. Leaving the whole chain visible rather than editing it down, since the corrections are the useful part of the record.

docs(devlog): record WP18, and retract a false narrowing on the CI ha…

docs(devlog): record WP19, the merges that arrived after the campaign…

docs(devlog): plan the seven paired bug-fix landings

test(catalog): give the convergence suite the timeout its runtime needs

Merge pull request #1461 from lidge-jun/agent/fix-1453-sync-transaction

Backlog triage update: keeping this open, but treating it as CI/service-lifecycle work rather than something we can mark fixed from adjacent changes.

The correction chain above matters: the evidence establishes repeated Linux shard hangs and repeated Bun EEXIST: epoll_ctl observations, but it does not establish a single test-file culprit or prove that the epoll_ctl error causes the hang.

Before closing this issue, re-run the sharded Linux suite on current dev and check whether the 15-minute silent-stall/orphan-bun signature still occurs. If it no longer reproduces across repeated runs, close with the exact fixing change or runtime update. If it still occurs, keep the investigation at the process/fd lifecycle level rather than targeting whichever test happened to be last in one run.

Live reproduction on a Linux host, with the hang state captured before the kill

While running campaign CI on a Linux box (Bun 1.3.14, Ubuntu, 8 cores) I ended up with two orphaned bun test processes from earlier shard runs, still alive at 4h34m and 2h56m of accumulated CPU. That is the same failure this issue describes, except nothing killed them at 15 minutes, so I could inspect them instead of reading a truncated log.

What the stuck process actually looks like

PID 2553709   elapsed 16494s   CPU 04:34:35   State: R (running)
PID 2656721   elapsed 10777s   CPU 02:56:42   State: R (running)

Both are spinning, not blocked. wchan is 0 and the state is R, so the "hang" is not a process waiting on a syscall that never returns — it is a busy loop burning a full core. That distinction matters for the diagnosis in this thread: a deadlock on a lock or a never-resolving promise would park the thread in S/D with a nonzero wchan.

Per-thread breakdown is identical in both:

 1 State: R (running)     <- the main bun thread
19 State: S (sleeping)

So exactly one thread spins and the whole 20-thread runtime is otherwise idle.

File descriptors at the moment of the hang

PID 2553709 (221 fds)          PID 2656721 (294 fds)
  6 anon_inode:timerfd          16 anon_inode:timerfd
  4 anon_inode:eventpoll        14 anon_inode:eventpoll
  4 anon_inode:eventfd          14 anon_inode:eventfd
  2 /tmp/ocx-xai-401-codex-R3ShgC/.opencodex-native-main.owner.sqlite
  2 /tmp/ocx-xai-401-codex-R3ShgC/.opencodex-native-main.owner.sqlite-journal

Two things worth naming.

The eventpoll/eventfd/timerfd counts grow with elapsed time — 4/4/6 on the younger-by-CPU process versus 14/14/16 on the other. Epoll instances accumulating rather than being reclaimed is consistent with the EEXIST: epoll_ctl observations reported earlier in this thread, and it points at the loop being inside the event-loop/timer machinery rather than inside any one test's assertion.

The still-open SQLite owner file and its journal in a per-test temp dir (ocx-xai-401-codex-*, from tests/server-xai-oauth-401-replay.test.ts) means the process died mid-transaction from the database's point of view. An open -journal is an uncommitted transaction. This is a native-main owner lock, which lines up with the cli-native-profile → cli-restart-health neighbourhood named earlier — but note it is an open fd, not proof that this test is the culprit. A later test can inherit a leaked handle from an earlier one.

Why the CI signal is misleading

The correction chain above is right that no single test file has been proven at fault, and I am not claiming one. What this evidence does establish:

the hang is a CPU-bound spin, so a wall-clock timeout is the only thing that will ever stop it;
it survives the parent shell — both orphans outlived the command that spawned them, which is why the run shows up as cancelled rather than failed and why a rerun quietly passes;
it poisons neighbouring runs. My third bun test was competing with two spinning orphans for cores, and the load average sat above 3 on an otherwise idle box. Any timing-sensitive test running in that window is measuring contention, not correctness. That alone would explain a chunk of the intermittency in this thread.

Suggested next step, without guessing at a culprit

Rather than keep bisecting test files, make the runner report instead of vanish: wrap the shard in a hard wall-clock timeout that, on expiry, dumps /proc/<pid>/task/*/stack, the fd table, and the per-thread states before SIGKILL, and exits non-zero so the shard fails rather than cancels. The evidence above took about thirty seconds to collect once the process was still around to inspect; the reason this issue has been hard to close is that CI has been throwing that evidence away.

I did not kill these with SIGQUIT first, which would have been the better move — a Bun crash dump would have named the spinning frame. Next time one appears I will try that before reclaiming the host.

docs(devlog): record WP11 security review of #1540 and #1541

Same hang on macOS, this time with a call graph

Follow-up to my previous comment. The Linux orphans there were State: R with wchan 0, which told us the process was spinning rather than blocked but could not say where. I have now hit the same thing on macOS during a release preflight — bun test --isolate tests at 55 minutes of CPU, one process, nothing else competing — and macOS sample gives the frame the Linux /proc route could not.

The call graph

The whole 55 minutes sits on the main thread, in one stack:

Thread ... DispatchQueue_1: com.apple.main-thread  (serial)
  start (dyld)
    ... bun frames ...
      bun + 0x328b324
        bun + 0x328b324
          bun + 0x328b324      <- same frame, 7 deep
            bun + 0x328b324
              ...
                bun + 0xdfe014
                  bun + 0x10948a0
                    bun + 0xe96d60
                      bun + 0xf709bc
                        kevent64  (libsystem_kernel.dylib)

Two things this adds to the thread.

The repeated identical frame. 0x328b324 appears seven times consecutively in a single stack. That is a recursive or trampolining call, and it is the same address every time — not a deep-but-varied stack. Whatever loop this is, it is re-entering itself.

The bottom is kevent64, not a lock. Sampling for 3 seconds and counting syscall leaves:

 41 read
  5 pthread_rwlock
  4 kevent64

So it is not parked in a futex or a rwlock — it is cycling the kqueue event loop and doing reads, over and over, making no progress. That is consistent with the Linux side, where the eventpoll/eventfd/timerfd descriptor counts grew with elapsed time. Same shape on both platforms: the event loop is being driven forever rather than draining.

This also rules out one plausible theory: it is not a test awaiting a promise that never settles. That would idle the process. This burns a full core.

Practical notes for whoever picks this up

--isolate does not avoid it. My run was bun test --isolate tests, so the per-file isolation the release helper relies on is not a workaround.

It is not load-related. The Linux occurrence had three competing processes and I wondered if contention was involved; this macOS one had the machine to itself.

The same run completes in about 7 minutes on the Linux CI host. So this is intermittent rather than a straightforward "the suite got slow" — same commit, same command, 7 minutes there versus 55+ minutes and counting here.

sample <pid> 3 -mayDie is the cheap way to capture this on macOS and does not disturb the process. On Linux, SIGQUIT should produce a Bun dump; that remains untried because I killed the earlier orphans before thinking of it, and this macOS process I sampled instead.

Still not proposing a culprit file. But the target is now narrower than "a test hangs": something re-enters bun + 0x328b324 and spins the kqueue loop, on both platforms, intermittently.

Resolved at the CI level on dev: 9ef7df8 (2026-08-11) added scripts/ci/run-bun-test-batches.sh, wired at ci.yml:301 — Linux shards now run fresh-process batches bounded by GNU timeout (120s, kill-after 15s), and is_bun_runtime_crash matches the exact EEXIST epoll_ctl / WriteStream internal:fs/streams crash signature from this report for bounded retry. The tracked failure shape (14-minute silent hang, shard cancelled, orphan bun process) is structurally eliminated: a stuck batch dies within ~135s and retries. Follow-up watchdog scaling in 053a034 / 12c4fe6. The Bun-runtime root cause remains upstream, but the CI defect this issue tracks is gone. (260818 campaign resolved-issue sweep, verified against origin/dev e446607.)

docs(devlog): record WP18, and retract a false narrowing on the CI ha…

docs(devlog): record WP19, the merges that arrived after the campaign…

docs(devlog): plan the seven paired bug-fix landings

test(catalog): give the convergence suite the timeout its runtime needs

Merge pull request lidge-jun#1461 from lidge-jun/agent/fix-1453-sync-…

docs(devlog): record WP11 security review of lidge-jun#1540 and lidge…

docs(devlog): archive the 48 units whose work landed on dev

Navigation Menu

Repository navigation

[Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process #1302

Client or integration

Area

Summary

Observed pattern

The one non-cancelled instance is the useful clue

Last log line before the stall

Not reproducible locally

Why this matters beyond the noise

Suggested starting points

Version

Operating system

Live reproduction on a Linux host, with the hang state captured before the kill

What the stuck process actually looks like

File descriptors at the moment of the hang

Why the CI signal is misleading

Suggested next step, without guessing at a culprit

Same hang on macOS, this time with a call graph

The call graph

Practical notes for whoever picks this up

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process #1302

Description

Client or integration

Area

Summary

Observed pattern

The one non-cancelled instance is the useful clue

Last log line before the stall

Not reproducible locally

Why this matters beyond the noise

Suggested starting points

Version

Operating system

Activity

github-actions commented on Aug 8, 2026

lidge-jun commented on Aug 8, 2026

lidge-jun commented on Aug 8, 2026

lidge-jun commented on Aug 8, 2026

lidge-jun commented on Aug 8, 2026

Wibias commented on Aug 12, 2026

lidge-jun commented on Aug 12, 2026

Live reproduction on a Linux host, with the hang state captured before the kill

What the stuck process actually looks like

File descriptors at the moment of the hang

Why the CI signal is misleading

Suggested next step, without guessing at a culprit

lidge-jun commented on Aug 12, 2026

Same hang on macOS, this time with a call graph

The call graph

Practical notes for whoever picks this up

lidge-jun commented on Aug 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions