server: wait up to 10 s for a started worker; steady three macOS timing tests

[novelKR]

Requested by Seongjae

The gateway now waits up to 10 s for the worker it starts, so its first start after the worker binary is built or installed on macOS no longer fails with worker socket not ready; three test timings that failed intermittently under load on macOS are made steady. Product code changes in one function only (5787a92). No Codex or DevGuard pin change.

Issue

Follow-up to #90 (CSRG-U4), whose "Out of scope" named these as candidates for a separate change: the macOS first launch of a newly placed worker binary against the ~2 s socket waits of wait_for_socket and runtime_binary.rs, and the timing of prepare_stdout_flood_is_too_large. It does not close an issue. Tracking: #76.

Item	Value
Base	main at 3e5e3e8 (#90's merge)
Head	f91380a9cfe91f874bd05e700ed00992113be10b
Commits	cc1caf6 (runner test: flood with the system's yes; test code), 58ecec4 (codex-runtime tests: 10 s socket deadline; test code), 5787a92 (server: the gateway waits up to 10 s for a started worker's socket; product code), 7ea7fdb (runner tests: run each launch stand-in once when it is written; test code), f91380a (runner tests: that warm run on macOS only; test code)
Pins	Codex gitlink 6b9826e3 and DevGuard source 6e7e065, unchanged

Summary

• Gateway start (product code, 5787a92). wait_for_socket (crates/server/src/runtime.rs) polled the socket of the worker the gateway starts with --runtime-bin 100 times at 20 ms, about 2 s. macOS checks a new executable file on its first exec: on the test Mac a worker binary that has never run took 1.9-2.0 s to accept in this round (2.2-2.8 s in U4's local runs), against 9 ms once it has run, and a copy of an already-run binary at a new path still took 2.1 s on its first run. So the gateway's first start after the worker binary is built or installed could fail with worker socket not ready. It now waits up to 10 s (WORKER_READY_BOUND), and a worker that exits before accepting still fails the start at once.
• Worker binary tests (test code, 58ecec4). wait_connect in crates/codex-runtime/tests/runtime_binary.rs had the same ~2 s loop; cargo places the worker binary anew for each test command. It now waits up to 10 s.
• Launch stand-ins (test code, 7ea7fdb). The runner's launch tests (CSRG-U4) start a stand-in helper, a bash script written to a new path for every test, and CodeSpace's launch bounds run from its spawn. Its first exec pays the same check, and while the runner's tests start many new files at once that took longer than a_governed_process_times_out_like_any_other's 2 s timeout in a local macos-core run (the launch was reported as not started, HelperEnded). Each stand-in now exits at once when called with --warm, and on macOS is run so once when it is written, outside every launch's bounds. Not on Linux (f91380a): there the check does not exist, and running a file the test process has just written can fail with ETXTBSY when another thread forked meanwhile (the hazard linux_sandbox's write_script documents).
• Sandbox prepare flood test (test code, cc1caf6). prepare_stdout_flood_is_too_large expects output too large before the 2 s prepare deadline. Its stand-in was a script written anew for each run (which also pays the first-exec check) that wrote one byte per printf, so under load the deadline could come first. The test now spawns /usr/bin/yes, present on macOS and Linux, which floods in large writes; the assertions are unchanged.

Contract changes

None: no tool schema, error code, status value, ID or wire protocol change (still 7). The only behavior change: for a worker that is alive but not yet accepting, the gateway's start now reports worker socket not ready after up to 10 s instead of about 2 s. A worker that exits first fails the start at once, as before.

Tests

Verified on the test Mac (macOS, 8 CPUs) with CPU-bound processes running:

Test	Before	After
prepare_stdout_flood_is_too_large, 16 CPU-bound processes	timed out in 7 of 50 runs (a script writing large blocks, exec yes x, still timed out in 3 of 50)	passed 50 of 50
runtime_binary.rs, worker placed anew, 8 CPU-bound processes	2 of 10 runs failed, each at the socket wait	passed 10 of 10
the gateway's start test, worker placed anew, 8 CPU-bound processes	1 of 10 runs failed with worker socket not ready	passed 10 of 10
a_governed_process_times_out_like_any_other, 16 CPU-bound processes	5 of 20 runs failed, the launch not started (HelperEnded)	passed 20 of 20

New test a_worker_that_exits_before_its_socket_fails_at_once (crates/server/src/runtime.rs): with /usr/bin/false as the worker, it asserts that the start fails with worker exited before the runner socket was ready in under 5 s.

Local validation on the owner's Mac (macOS 27.0, Apple M1, 8 CPUs; scripts/validate-upstream.py):

Run	Head	Stages	Result
tt-local-1	5787a92	all	every stage passed but linux-isolation (Linux only, run by CI); integration and unit-codex-runtime passed starting from a worker each stage had placed anew (they had failed that way in #90's local runs)
tt-local-2	5787a92	macos-core, macos-fork-race	fork race 400 trials, 0 failed; macos-core failed once in the adapter crate (observation below)
tt-local-3	5787a92	macos-core, macos-fork-race	fork race passed; macos-core failed on a_governed_process_times_out_like_any_other (HelperEnded), which led to 7ea7fdb
tt-local-4	7ea7fdb	all	every stage passed but linux-isolation: gateway 31 tests without features (its start test included), runner 105 and, with --features devguard, 140, domain 32, store 47; runtime_binary 6
tt-local-5	7ea7fdb	macos-core, macos-fork-race	fork race 400 trials, 0 failed; macos-core failed only in the adapter crate (observation below)
tt-repro-1 to 5	f91380a	macos-core, five times	all passed

f91380a changes Linux only, so macOS behaves as on 7ea7fdb; on f91380a the runner's tests (140 with --features devguard, 105 without) and clippy (both) passed before the push.

Observation, outside this PR: the adapter test registration::tests::native::each_process_registers_only_its_own_identity failed twice, in provision before doing anything else, with "another authority owns this state directory" from DevGuard's TestAuthority bootstrap: in tt-local-2 and tt-local-5, each a macos-core run that came right after a full all run. It did not recur in 30 runs of the adapter's tests with 8 CPU-bound processes, nor in five consecutive macos-core runs. The authority takes flock(LOCK_EX | LOCK_NB) on its state directory's lock file, and TestAuthority takes it, drops it and takes it again while it starts. The hypothesis, unconfirmed: a child that another adapter test forks with pre_exec (handoff.rs, launch.rs) during that first hold keeps the locked descriptor until its exec, which is slower while macOS handles files a full run has just placed. This is test code from CS-RG U2/U4, unchanged here, and a candidate for a separate change.

Security scenarios

• No change to credential handling, the DevGuard carrier, descriptor handoff, workspace isolation or the private runner directory. The gateway only polls, for longer, the socket path it chose in its own private runner directory.
• A crashed worker is not hidden by the longer bound: the wait still checks the worker first and fails at once when it has exited (covered by the new test).
• Rollback: revert this PR; nothing stored, no wire or configuration change.

Out of scope

• Removing the DevGuard#24-era past_the_credential session retries in the devguard and server crate tests: left as is here. The pinned DevGuard 6e7e065 (DevGuard#27) still reads a secret already buffered when its 250 ms deadline has passed, but a sender that stalls past the deadline is still refused by design, so whether the retries are still needed has to be measured first.
• CSRG-U5 and U6, any pin change, a DevGuard release, or a change to the installed DevGuard or CodeSpace services.