Skip to content

BUG: tracker dispatch mode fails before any worker HTTP — claims task, sets up workspace, never contacts tagged workers, retries exhaust silently #53

Closed

Description

@vernonstinebaker

Environment

  • nullboiler v2026.5.29 (released binary), nulltickets v2026.3.2, nullhub v2026.5.29
  • Single alpine container, one boiler process, tracker pull mode
  • Workers (config.json workers array, per multi-bot-integration.md):
    • nullclaw-1 → http://127.0.0.1:3000/webhook, protocol webhook, tags ["coder"]
    • zeroclaw-1 → http://zeroclaw-test:42617/webhook, protocol webhook, tags ["reviewer"] (ZeroClaw v0.8.5, reachable — direct POST returns a valid LLM response)
  • tracker: { "url": "http://127.0.0.1:7700", "agent_id": "nullboiler", "poll_interval_ms": 5000 }

Setup

  • Pipeline (nulltickets POST /pipelines): FSM todo → done, states.todo.agent_role = "reviewer", states.done.terminal = true
  • Workflow file in workflows/ (loaded and listed by GET /workflows):
{
  "id": "hetero-demo",
  "pipeline_id": "<uuid of pipeline above>",
  "claim_roles": ["reviewer"],
  "execution": "dispatch",
  "dispatch": { "worker_tags": ["reviewer"], "protocol": "webhook" },
  "prompt_template": "Task: {{task.title}}\nDescription: {{task.description}}",
  "on_success": { "transition_to": "done" },
  "on_failure": { "transition_to": "failed" }
}

Observed

  1. GET /tracker/status → {"mode":"pull", ...} — tracker connected, polling every 5s
  2. Tasks created in the pipeline are claimed: latest_run.status = "running", agent_id = "nullboiler", agent_role = "reviewer" ✓
  3. /tracker/status shows the run in "workspace_setup", then "failed" (current_turn: 1, max_turns: 20), failed_count increments on each retry cycle
  4. The tagged worker's gateway logs show zero inbound requests — POST /webhook count unchanged across the entire experiment; tcpdump-level confirmation unnecessary (worker-side request log is quiet)
  5. Task stage never transitions (stays todo); on_failure.transition_to ("failed") never applied — possibly the same root cause as fix(tracker): apply the workflow's on_failure transition when a dispatch fails #52
  6. Cycle repeats until retries exhaust; no log line names the dispatch failure reason

Direct evidence the failure is boiler-side

  • POST http://127.0.0.1:7700/leases/claim with {"agent_id":"nullboiler","agent_role":"reviewer","lease_ttl_ms":60000} returns the task immediately — the tracker's claim-by-role works
  • Direct POST to the ZeroClaw worker's /webhook returns a valid model response — the worker works
  • The failure is strictly inside nullboiler between claim and worker HTTP: no request is ever emitted

Expected

  • execution: "dispatch" with dispatch.worker_tags matching a configured worker's tags should select the worker, POST the rendered prompt_template, and apply on_success/on_failure transitions — or at minimum log a named, actionable failure (e.g. "no eligible worker for tags [...]") instead of failing silently.

Related

Note on ZeroClaw A2A

ZeroClaw v0.8.5 serves A2A (agent cards at /.well-known/agent-card.json, per-alias /a2a/{alias} JSON-RPC — confirmed from compiled binary and git history), so once dispatch-to-worker works, an A2A dispatch protocol option would let boiler drive ZeroClaw's native task lifecycle, not just its webhook. Happy to test.

Activity

  1. vernonstinebaker commented on Sep 28, 2026

    @vernonstinebaker
    ContributorAuthor

    Resolved — the reproduction turned out to be three stacked issues, all now identified. End-state first: after fixes, the full loop works — task claimed → workspace_setup → A2A message/send → nullclaw v0.3.0 agent turn (13.3s) → reply parsed by #48's parser → on_success transition → stage=done, completed_count=1, failed_count=0.

    1. Zombie processes (environmental — tracked as nullhub#86). Hub restarts left overlapping boiler processes contending for the DB; claims silently starved. With exactly one clean process, claims flowed immediately.

    2. Worker URL convention (the 404 mystery). The A2A dispatcher appends /a2a to the configured worker URL. A worker URL of http://host:3000/a2a produces POST /a2a/a2a → 404. The worker URL must be the base URL. Captured via netcat on the exact request bytes:

    POST /capture/a2a HTTP/1.1        ← worker url was http://127.0.0.1:9999/capture
    content-type: application/json
    Authorization: Bearer ...
    {"jsonrpc":"2.0","method":"message/send","params":{"message":{..."kind":"message","messageId":"msg-<ctx>"...},"contextId":"<ctx>"}}
    

    Body was correct per #48 the whole time. Docs PR incoming for the URL convention.

    3. Transition convention. on_success.transition_to is applied as the pipeline FSM trigger name. With an FSM transition {from: todo, to: done, trigger: "complete"}, the success path logs transition rejected … with trigger done and the task stays in todo (retry loop). Naming the FSM trigger after the target state (trigger: "done") resolves it. Also worth documenting.

    #48 itself worked exactly as merged — no code issue found in the dispatch path beyond the conventions above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions