A waited job_run can outlive the harness's tool-call timeout, and the toolkit logs it as a success

[shrimalmadhur]

What happens

job_run chooses between waiting for a job and detaching from it by comparing the
job's own deadline against the turn's clock (packages/core/src/job-server.ts:324):

if (jobDeadlineMs(job) > turnTimeoutMs) { /* start, don't wait */ }

The comment directly above states the intent — the choice comes from "the job's own
declaration against the turn's own clock … not a field in this call, and not a field
in the manifest that could disagree with either number."

There is a third clock, and it is not in that comparison. The harness hosting the
MCP client applies its own per-tool-call timeout. The toolkit neither sets nor reads
it: MCP_TOOL_TIMEOUT and MCP_TIMEOUT appear nowhere in this repo. Whenever that
bound is shorter than the job's real duration, it is the one that binds, and the
comparison above is made against a number that never applies.

Observed

A job declaring wallClockMs: 150000 + deadlineHeadroomMs: 45000 against
limits.turnTimeoutMs: 300000 — so 195000 <= 300000 and job_run waited, correctly
by its own arithmetic. The body's real duration was ~91s.

turn_start  …
turn_done   … sent=1 ms=87760                              ← the turn closed here
…job posts its own result through its report channel…
tool_call tool="mcp__jobs__job_run" outcome=ok ms=90834     ← logged after that

The harness abandoned the call at ~87.8s. The model, handed a failing tool, composed a
message telling humans the job had timed out and produced nothing — and named two of
them to go look. Three seconds later the job finished and posted its own, correct
result. A human saw both messages.

The logging half, which is the worse half

There is no timeout, abort, or error line anywhere in the toolkit's log for that run.
outcome=ok ms=90834 is the only record and it reads as a clean success. The toolkit
cannot currently tell that a call it served was abandoned, so the failure is invisible
to it and to anyone reading its logs afterwards. The only artifact is a wrong sentence
in a channel, while every log the toolkit keeps says the run was fine.

Suggestions

Roughly in order of how much each helps:

1. Make the two clocks unable to disagree, as the existing comment already intends:
   set the harness's tool-call timeout explicitly from limits.turnTimeoutMs when
   spawning it, so a job that fits inside a turn also fits inside the transport.
2. Failing that, fold the harness bound into the comparison at job-server.ts:324, so
   a job near the edge detaches rather than waits.
3. Independently of both: log when a waited call completes after its turn has ended.
   Even with (1) this is the signal that says the clocks have drifted apart again, and
   it is what turns this from "a wrong message appeared in a channel" into something
   greppable.

(3) stands on its own merits — a job host that cannot observe an abandoned call has no
way to report this class of failure at all.

[veriton-dev]

Host-side notes from running unattended agent loops with nested clocks

This matches a failure mode we keep hitting on long-lived agent hosts: three clocks that look hierarchical but aren't actually composed.

What bites in production

1. Turn clock ≠ tool-transport clock ≠ job wall clock.
   job_run correctly compares job deadline vs turn timeout, then waits. The MCP client / harness still has its own per-tools/call bound (often env default or SDK default). That third number never enters the comparison, so a "safe to wait" decision can still be cut mid-flight by the transport.

2. Abandonment is silent upstream.
   When the harness drops the call, the job host keeps working and later reports success. Logs say outcome=ok. The model already got a truncated/error tool result and may have messaged humans that the job failed. Dual narratives, zero greppable error on the job side.

3. Late success is worse than a clean fail.
   A job that finishes 3s after the turn closed and posts a correct result into a channel that already contains "timed out / nothing produced" creates operator thrash. Prefer detach-or-fail over wait-and-lie.

Practical contract that has held up

• Compose clocks explicitly at spawn:
  toolCallTimeoutMs = min(harnessDefault, turnRemainingMs - margin) and set it on the MCP client / request options so the transport bound is never tighter than the wait decision without the job host knowing.

• Detach threshold uses the effective wait budget, not just turnTimeoutMs:

  effectiveWait = min(turnRemaining, toolCallTimeout, any_parent_deadline) - headroom
  if jobDeadline > effectiveWait → start + detach (don't wait)
  

• Mark waited calls with a generation / turn id. On completion after turn end: log late_completion after_abandon (job id, declared deadline, turn id, elapsed, outcome). Even with fixed composition this is the canary when clocks drift again.

• Never map harness abandon → model-facing "job failed empty".
  Prefer structured tool_call_abandoned / detached_in_progress so the model does not invent a human escalation, and so humans don't see contradictory success+failure.

• Cancel path: if the harness supports notifications/cancelled (or equivalent), fire it when the turn ends while still waiting — so the job host can observe abandon instead of only learning via late completion.

Acceptance checks worth adding

• Job with wall < turn but wall > harness tool timeout → must detach (or raise tool timeout), never wait-and-log-ok after abandon.
• Forced harness cut mid-wait → toolkit log contains abandon/late signal; no lone outcome=ok.
• Model-visible tool result distinguishes in_progress/detached vs failed.

Happy to refine if useful — the logging half (your #3) is the highest leverage even before composition is perfect.