Repository navigation
BUG: tracker dispatch mode fails before any worker HTTP — claims task, sets up workspace, never contacts tagged workers, retries exhaust silently #53
Copy link
Copy link
Closed
Description
Environment
- nullboiler v2026.5.29 (released binary), nulltickets v2026.3.2, nullhub v2026.5.29
- Single alpine container, one boiler process, tracker pull mode
- Workers (config.json
workersarray, per multi-bot-integration.md):nullclaw-1→http://127.0.0.1:3000/webhook, protocolwebhook, tags["coder"]zeroclaw-1→http://zeroclaw-test:42617/webhook, protocolwebhook, tags["reviewer"](ZeroClaw v0.8.5, reachable — direct POST returns a valid LLM response)
tracker:{ "url": "http://127.0.0.1:7700", "agent_id": "nullboiler", "poll_interval_ms": 5000 }
Setup
- Pipeline (nulltickets
POST /pipelines): FSMtodo → done,states.todo.agent_role = "reviewer",states.done.terminal = true - Workflow file in
workflows/(loaded and listed byGET /workflows):
{
"id": "hetero-demo",
"pipeline_id": "<uuid of pipeline above>",
"claim_roles": ["reviewer"],
"execution": "dispatch",
"dispatch": { "worker_tags": ["reviewer"], "protocol": "webhook" },
"prompt_template": "Task: {{task.title}}\nDescription: {{task.description}}",
"on_success": { "transition_to": "done" },
"on_failure": { "transition_to": "failed" }
}Observed
GET /tracker/status→{"mode":"pull", ...}— tracker connected, polling every 5s- Tasks created in the pipeline are claimed:
latest_run.status = "running",agent_id = "nullboiler",agent_role = "reviewer"✓ /tracker/statusshows the run in"workspace_setup", then"failed"(current_turn: 1, max_turns: 20),failed_countincrements on each retry cycle- The tagged worker's gateway logs show zero inbound requests —
POST /webhookcount unchanged across the entire experiment;tcpdump-level confirmation unnecessary (worker-side request log is quiet) - Task stage never transitions (stays
todo);on_failure.transition_to("failed") never applied — possibly the same root cause as fix(tracker): apply the workflow's on_failure transition when a dispatch fails #52 - Cycle repeats until retries exhaust; no log line names the dispatch failure reason
Direct evidence the failure is boiler-side
POST http://127.0.0.1:7700/leases/claimwith{"agent_id":"nullboiler","agent_role":"reviewer","lease_ttl_ms":60000}returns the task immediately — the tracker's claim-by-role works- Direct
POSTto the ZeroClaw worker's/webhookreturns a valid model response — the worker works - The failure is strictly inside nullboiler between claim and worker HTTP: no request is ever emitted
Expected
execution: "dispatch"withdispatch.worker_tagsmatching a configured worker'stagsshould select the worker, POST the renderedprompt_template, and applyon_success/on_failuretransitions — or at minimum log a named, actionable failure (e.g. "no eligible worker for tags [...]") instead of failing silently.
Related
- BUG: run with no eligible worker waits silently forever (no fail-fast on permanent dead-end) #44 (no eligible worker waits silently forever — same silent-failure family)
- fix(tracker): apply the workflow's on_failure transition when a dispatch fails #52 (on_failure transition not applied — observed here too)
- BUG: /tracker/* endpoints block for the duration of tick() network I/O #41/fix(tracker): serve /tracker/* from tick-independent snapshot #50 (/tracker/* snapshot staleness made diagnosis harder)
Note on ZeroClaw A2A
ZeroClaw v0.8.5 serves A2A (agent cards at /.well-known/agent-card.json, per-alias /a2a/{alias} JSON-RPC — confirmed from compiled binary and git history), so once dispatch-to-worker works, an A2A dispatch protocol option would let boiler drive ZeroClaw's native task lifecycle, not just its webhook. Happy to test.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
vernonstinebaker commentedon Sep 28, 2026
Resolved — the reproduction turned out to be three stacked issues, all now identified. End-state first: after fixes, the full loop works — task claimed →
workspace_setup→ A2Amessage/send→ nullclaw v0.3.0 agent turn (13.3s) → reply parsed by #48's parser →on_successtransition → stage=done, completed_count=1, failed_count=0.1. Zombie processes (environmental — tracked as nullhub#86). Hub restarts left overlapping boiler processes contending for the DB; claims silently starved. With exactly one clean process, claims flowed immediately.
2. Worker URL convention (the 404 mystery). The A2A dispatcher appends
/a2ato the configured worker URL. A worker URL ofhttp://host:3000/a2aproducesPOST /a2a/a2a→ 404. The worker URL must be the base URL. Captured via netcat on the exact request bytes:Body was correct per #48 the whole time. Docs PR incoming for the URL convention.
3. Transition convention.
on_success.transition_tois applied as the pipeline FSM trigger name. With an FSM transition{from: todo, to: done, trigger: "complete"}, the success path logstransition rejected … with trigger doneand the task stays intodo(retry loop). Naming the FSM trigger after the target state (trigger: "done") resolves it. Also worth documenting.#48 itself worked exactly as merged — no code issue found in the dispatch path beyond the conventions above.