Repository navigation
fix(io): park the JS File and Process effects so io_wait flushes stdout - #1899
Conversation
Lorenzobattistela
left a comment
There was a problem hiding this comment.
Merged onto main 757180e, all gates green: repo 54/54, test 1754/1754, ping 81/81, perf 124/124, safe 0 false rejects. The approach looks right to me. One must-fix from the merge with #1887, plus a few small things.
Must fix: File.write_at neither parks nor flushes after the merge. #1887 added file_write_at, which shares file_write_buffer. This PR removes the io_sync() from file_write_buffer and parks only File.write and File.write_bytes, so on JS write_at now runs before the text IO.print holds. Repro on Bun, stdout redirected to a file: open /dev/fd/1 with "w", then IO.print("hello"), File.write_at(f, 0, [88, 89]), IO.print("bye"). On main this prints XYllo then bye. With this PR it prints hello then bye: the positioned write lands first, and the held text then overwrites it. C runs write_at through io_work (file_write.c), so it should park like its siblings, keeping the Fail(22) early return for bad bytes and append files.
Small things:
- The new comment in
file_open.jssays every effect C runs throughio_workparks here. That becomes true oncewrite_atparks. "As in C" is also only exact for effects that block: C'sio_workhands the job to a helper thread right away and flushes later inio_wait, so C does not flush before the effect runs. process_run(...a)witha.pop()reads differently from the siblings. Naming the parameters (program, args, input, max, ms, k) costs about the same.tests/io/file_open_fifo_peer.c:peer_armignoresmkfifo,pipeandforkfailures. Ifpipe()fails,dup2(p[1], 1)uses uninitialised fds, and the test hangs instead of failing at setup.
Non-blocking notes:
- Every effect now has to remember the
io_park_on(undefined, 0, k, …, 0)wrapper (11 copies so far), andwrite_atalready slipped through. Before,file_write_bufferflushed for every caller. A shared helper next toio_park_on, or a flag onio_eff, would make it hard to forget, ifcomp.tsever has the room. - Cost on JS (not measured; no bench uses File): every File or Process op now ends the fiber's run and goes through a full
io_wait,io_syncincluded. A read-a-line, print-a-line loop on a pipe makes onewrite(2)per line instead of one per 64 KiB. That's what C does too, so I'm noting it, not asking for a change.
I closed #1895 in favour of this PR. Its file_read_prompt test (a prompt written with IO.write, no newline, before a read_line) passes here and fails on main. It could be added alongside file_read_stdout_order if you like, but that's optional.
On JS, File.open/read*/size/write* and Process.run ran synchronously inside the effect, so the loop never reached io_wait and never wrote what IO.print holds before blocking: a program that prints and then reads from a peer waiting for that output deadlocked. C runs these through io_work, parks the fiber, and flushes in io_wait. Each of these effects now parks itself with io_park_on, due at once, so it runs at the next io_wait, after its io_sync, as in C. Other fibers run before its result, as in C. io_wait skips select when it waits on no fd with a zero timeout, so parked files need no bun:ffi on node. File.write's own io_sync goes away. Supersedes #1893; its test file_read_stdout_order is kept. Co-authored-by: MattCozendey <MattCozendey@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
file_read_stdout_order reads a regular file, which never blocks, so it pins the order of the output but not that the flush comes before the effect blocks. Here stdout is moved onto a pipe read by a shell that opens the FIFO for writing only after it sees "ready", so a late flush hangs the run. On main the JS lane hangs; native passes on both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1887 added File.write_at on file_write_buffer, which no longer flushes held stdout itself, so on JS the positioned write ran before the text IO.print held. It now parks like File.write and File.write_bytes, keeping its Fail(22) for bad bytes and append files. Also: process_run names its parameters, the comment above file_open goes, and the FIFO test fails at setup if mkfifo, pipe or fork fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b25fd63 to
b8f361b
Compare
|
Thanks for the review! Rebased onto main and pushed b8f361b:
Tested on macOS: all 33 I left the shared helper for now: 🤖 Generated with Claude Code |
From #1895: IO.write("prompt> ") with no newline, then File.read_line. The JS twin checks that the prompt was written before the read ran. Passes with this PR; on main the read runs first. Co-authored-by: toprameen <11931132+toprameen@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bug: on JS,
IO.printholds stdout when it is not a terminal.File.open,File.read*,File.size,File.write*andProcess.runrun synchronously inside the effect (openSync,readvSync,Atomics.wait), so the loop never reachesio_waitand never writes the held text before blocking. A program that prints and then reads from a peer waiting for that output deadlocks. C runs these effects throughio_work, parks the fiber, and flushes inio_waitbeforeselect.Fix: each of these effects parks itself with
io_park_on(undefined, 0, k, () => …, 0), due at once, so it runs at the nextio_wait, after itsio_sync, as in C. This follows the ruleio_effalready enforces ("an effect that waits parks itself"), so nothing new goes into the loop:io_run: unchanged.io_wait: skipsselect(andio_sys, which needsbun:ffi) when it waits on no fd with a zero timeout, so parked files still work on Node.file_write.js: its ownio_syncis gone; the wait flushes first.comp.ts: +11 ttok (65,985 / 66,000).As in C, the fiber now parks, and other fibers run before its result comes back. The remaining difference is inherent to JS: the syscall runs on the main thread, so it blocks every fiber while it runs.
Supersedes #1893 (by @MattCozendey), which added an
io_synctoFile.readonly. Its testtests/io/file_read_stdout_orderis kept.Tested (macOS):
bun gates/repo.ts: PASS 54 / 54.All 30
tests/iotests that use File,Process.runorrandom_u32, plusfile_read_stdout_order, pass on Bun.file_read_stdout_orderandstdout_file_orderpass on Node.Repro (
IO.print("ready"), thenFile.read_lineon/dev/stdin, fed by(sleep 3; echo hi) |): "ready" arrives at 0 s on Bun and Node.New
tests/io/file_open_fifo_peer: stdout is moved onto a pipe read by a shell that opens a FIFO for writing only after it sees "ready", so a flush that comes after the effect blocks hangs the run. With this PR it passes on native, Bun and through--checkup(stdout is put back for the next test). Onmain, the Bun lane hangs (killed at 10 s) and native passes.tests/io/file_read_prompt, from fix(io): flush stdout before JS file reads #1895 (by @toprameen):IO.write("prompt> ")with no newline, thenFile.read_line. Passes on Bun, Node and through--checkup; onmainthe read runs before the prompt is written.Not run: native, the cluster gates.
🤖 Generated with Claude Code