Skip to content

fix(dispatch): add timeouts to worker dispatch and health probes - #51

Closed
impuls42 wants to merge 1 commit into
nullclaw:mainfrom
sengine-cloud:fix/worker-http-timeouts
Closed

impuls42 wants to merge 1 commit into
nullclaw:mainfrom
sengine-cloud:fix/worker-http-timeouts

Conversation

@impuls42

Copy link
Copy Markdown

What

Adds deadlines to the two outbound HTTP calls to workers, which today have none:

  • probeWorker (engine health pass): new engine.worker_probe_timeout_ms, default 5000.
  • dispatchStep (engine and tracker ticks): wires up the existing engine.default_timeout_ms, which was declared but never read. The default goes from 300000 to 1800000.

Setting either to 0 disables it.

Why

std.http.Client.fetch has no timeout (FetchOptions carries none, and ConnectTcpOptions.timeout is never read). A worker that accepts the TCP connection and then goes silent blocks the caller forever:

  • The health probe hangs, so the failure is never counted and the circuit breaker can't trip. The check that should detect the worker is the thing that's stuck.
  • With the tracker enabled, the tick holds state.mutex across the dispatch, so /health, /workers and /tracker/* stop answering as well.

How

Each request runs on a dedicated std.Io.Threaded instance, started with Io.concurrent. The caller polls a completion flag until the deadline, then cancels. Io.concurrent is used rather than Io.async because async may run the call inline when the pool's async limit is reached. On a 1-CPU host that limit defaults to 0, which would silently disable the deadline. If no thread can be started, the request falls back to running inline and unbounded, as it does today. The process-wide Io is unchanged.

The dispatch default is raised because agent turns legitimately take minutes. A deadline shorter than a real turn would break working dispatches, so it's meant as a backstop against a wedged socket, not as a latency budget.

Relation to #43

This partially addresses #43 (a hung worker hangs the step forever):

  • A wedged synchronous dispatch now fails after default_timeout_ms instead of hanging, and the probe gives up after 5 s.
  • Per-step timeout_ms is not enforced on the synchronous path yet.
  • A dispatch timeout still goes through the generic dispatch-failure path, which counts toward the worker's circuit breaker. With the long default, that only happens for workers that are genuinely unresponsive.

Both points could follow up separately. I've kept this patch to the "never block forever" part.

Note for existing configs

config.example.json, docker/nullboiler.config.json and examples/multi-agent-slack/config.json set default_timeout_ms explicitly (300000 or 600000). Until now that value was inert. After this change it's enforced, so those setups would cut off a dispatch after 5 or 10 minutes. I've left the example files alone; happy to bump them or drop the key if you prefer.

Validation

  • zig build is clean.
  • zig build test: 356/356 pass. That includes a new test: probeWorker against a listening socket that never accepts returns false within the configured deadline.
  • bash tests/test_e2e.sh: 57/57 pass.
  • Manually tested against a worker that accepts and never responds. Before, the failure count stayed at 0 and the worker stayed active indefinitely; with the tracker on, the HTTP API stopped responding. After, the worker reaches dead after three failed probes and the API keeps answering.

🤖 Generated with Claude Code

A worker that accepts a connection and then never answers blocks the caller
forever. std.http.Client has no timeout: FetchOptions carries none, and the
only `timeout` field in the http client (ConnectTcpOptions.timeout) is never
read.

Both callers sit on long-lived loops. probeWorker runs on the engine's health
pass and dispatchStep runs on the engine and tracker ticks, so one silent
worker stalls the loop that is supposed to notice it. The circuit breaker
cannot trip, because the health check is the thing that is stuck. With the
tracker enabled the tick holds state.mutex across the call, so the whole HTTP
API stops answering.

Each request now runs on a dedicated threaded Io via Io.concurrent and is
cancelled once its deadline passes. Io.concurrent is used rather than
Io.async because async may run the call inline when the pool is saturated
(on a 1-CPU host its default limit is 0), which would silently disable the
deadline. The process-wide Io is left alone.

Two EngineConfig knobs, 0 disables either:

  worker_probe_timeout_ms  5000      a worker that cannot answer a bare GET
                                     promptly is not healthy
  default_timeout_ms       1800000   already declared but never read; wired
                                     up here. The default is raised from
                                     300000 because agent turns legitimately
                                     run for minutes, and a deadline shorter
                                     than a real turn would break working
                                     dispatches.

Tested against a worker that accepts and never responds. Before: failures
stay at 0 and the worker stays active indefinitely, and with the tracker on,
/health, /workers and /tracker/status stop responding. After: the worker
reaches 'dead' after three probe failures and the API keeps answering. Adds
a unit test covering the probe path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@impuls42 impuls42 closed this Sep 27, 2026
@impuls42
impuls42 deleted the fix/worker-http-timeouts branch September 27, 2026 06:19
@vernonstinebaker

Copy link
Copy Markdown
Contributor

Hi @impuls42 — sorry this sat without a review; the queue has been slow. This fix addresses #43 (dispatch has no timeout), which is still open and wanted. Would you be willing to reopen it? We'll review and merge promptly — and if you closed it due to the changes needing rework after the recent dispatch refactors (#46–#49 landed this week), let us know and we'll help rebase or take it from here with your approach credited.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants