Repository navigation
Conversation
A worker that accepts a connection and then never answers blocks the caller
forever. std.http.Client has no timeout: FetchOptions carries none, and the
only `timeout` field in the http client (ConnectTcpOptions.timeout) is never
read.
Both callers sit on long-lived loops. probeWorker runs on the engine's health
pass and dispatchStep runs on the engine and tracker ticks, so one silent
worker stalls the loop that is supposed to notice it. The circuit breaker
cannot trip, because the health check is the thing that is stuck. With the
tracker enabled the tick holds state.mutex across the call, so the whole HTTP
API stops answering.
Each request now runs on a dedicated threaded Io via Io.concurrent and is
cancelled once its deadline passes. Io.concurrent is used rather than
Io.async because async may run the call inline when the pool is saturated
(on a 1-CPU host its default limit is 0), which would silently disable the
deadline. The process-wide Io is left alone.
Two EngineConfig knobs, 0 disables either:
worker_probe_timeout_ms 5000 a worker that cannot answer a bare GET
promptly is not healthy
default_timeout_ms 1800000 already declared but never read; wired
up here. The default is raised from
300000 because agent turns legitimately
run for minutes, and a deadline shorter
than a real turn would break working
dispatches.
Tested against a worker that accepts and never responds. Before: failures
stay at 0 and the worker stays active indefinitely, and with the tracker on,
/health, /workers and /tracker/status stop responding. After: the worker
reaches 'dead' after three probe failures and the API keeps answering. Adds
a unit test covering the probe path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
|
Hi @impuls42 — sorry this sat without a review; the queue has been slow. This fix addresses #43 (dispatch has no timeout), which is still open and wanted. Would you be willing to reopen it? We'll review and merge promptly — and if you closed it due to the changes needing rework after the recent dispatch refactors (#46–#49 landed this week), let us know and we'll help rebase or take it from here with your approach credited. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds deadlines to the two outbound HTTP calls to workers, which today have none:
probeWorker(engine health pass): newengine.worker_probe_timeout_ms, default5000.dispatchStep(engine and tracker ticks): wires up the existingengine.default_timeout_ms, which was declared but never read. The default goes from300000to1800000.Setting either to
0disables it.Why
std.http.Client.fetchhas no timeout (FetchOptionscarries none, andConnectTcpOptions.timeoutis never read). A worker that accepts the TCP connection and then goes silent blocks the caller forever:state.mutexacross the dispatch, so/health,/workersand/tracker/*stop answering as well.How
Each request runs on a dedicated
std.Io.Threadedinstance, started withIo.concurrent. The caller polls a completion flag until the deadline, then cancels.Io.concurrentis used rather thanIo.asyncbecauseasyncmay run the call inline when the pool's async limit is reached. On a 1-CPU host that limit defaults to 0, which would silently disable the deadline. If no thread can be started, the request falls back to running inline and unbounded, as it does today. The process-wide Io is unchanged.The dispatch default is raised because agent turns legitimately take minutes. A deadline shorter than a real turn would break working dispatches, so it's meant as a backstop against a wedged socket, not as a latency budget.
Relation to #43
This partially addresses #43 (a hung worker hangs the step forever):
default_timeout_msinstead of hanging, and the probe gives up after 5 s.timeout_msis not enforced on the synchronous path yet.Both points could follow up separately. I've kept this patch to the "never block forever" part.
Note for existing configs
config.example.json,docker/nullboiler.config.jsonandexamples/multi-agent-slack/config.jsonsetdefault_timeout_msexplicitly (300000 or 600000). Until now that value was inert. After this change it's enforced, so those setups would cut off a dispatch after 5 or 10 minutes. I've left the example files alone; happy to bump them or drop the key if you prefer.Validation
zig buildis clean.zig build test: 356/356 pass. That includes a new test:probeWorkeragainst a listening socket that never accepts returnsfalsewithin the configured deadline.bash tests/test_e2e.sh: 57/57 pass.activeindefinitely; with the tracker on, the HTTP API stopped responding. After, the worker reachesdeadafter three failed probes and the API keeps answering.🤖 Generated with Claude Code