Skip to content

sdk/python: bound a claim by the caller's deadline - #212

Merged
CMGS merged 3 commits into
mainfrom
fix/claim-deadline
Sep 17, 2026
Merged

CMGS merged 3 commits into
mainfrom
fix/claim-deadline

Conversation

@CMGS

@CMGS CMGS commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Closes #182.

Client._request gave every control-plane request a fresh client.timeout, and the redirect walk on a warm miss is a loop of such requests. The LangChain toolkit promises a five-minute wall clock per sandbox_exec and, since #179, honours it for the run — but not for the claim: two black-holed redirect candidates cost 600 s with a 300 s client timeout, so the tool could exceed its promise on a cluster with unreachable peers.

Claims now take an absolute time.monotonic() deadline:

  • Client.new, Template.new and Checkpoint.new accept keyword-only deadline, threaded through _claim_from, _post_json and the redirect walk into _request
  • _request bounds each attempt by min(timeout, deadline - now) and raises TimeoutError once the deadline is spent, before the request is built — _try_each and _redirect_fallback only catch APIError, so the walk stops there instead of trying the next candidate
  • the deadline arithmetic is the one conn.remaining_timeout the dial path already used, now shared rather than duplicated (it gained the operation name for the error message)
  • CocoonToolkit._exec passes the deadline it already computes, through sandbox() into _claim

Measured against a node that redirects to two addresses which accept TCP and never answer, client timeout 2 s:

no deadline (old behavior): APIError after 4.01s
deadline +0.3s:            TimeoutError after 0.30s

Default behavior is unchanged: without a deadline every request still gets the full timeout.

Tests: test_claim_refuses_a_spent_deadline (the node sees no request at all), test_claim_deadline_bounds_the_redirect_walk (two black holes, fails in 0.3 s instead of 2 x 2 s), plus test_exec_claims_inside_the_call_budget and test_exec_reports_a_claim_that_outlives_the_budget on the toolkit. conftest gained a black_hole fixture — a listening socket that never accepts.

Left alone deliberately: Client.lookup scatters its probes concurrently under min(_PEERS_TIMEOUT, timeout) = 5 s, so it cannot run away the same way, and the OpenAI adapter makes no wall-clock promise over its claim (its timeout is caller-supplied and applies to run).

Gates: ruff format --check and ruff check clean in sdk/python, sdk/openai and sdk/langchain; pytest 195 passed in sdk/python and 8 in sdk/langchain. sdk/openai's suite does not collect on this machine (openai-agents is not installed); it is untouched by this change and CI runs it.

Every control-plane request took a fresh client timeout, redirect targets
included, so the LangChain toolkit's five-minute promise bought 300s per
black-holed candidate: two of them cost 600s before the tool's own budget
check ran. Claims now accept an absolute time.monotonic() deadline that
bounds each request by what is left and refuses once it is spent, and the
toolkit passes the deadline it already computes for the run.

Measured against a node that redirects to two addresses which accept TCP
and never answer, client timeout 2s: 4.01s and an APIError before, 0.30s
and a TimeoutError with a 0.3s deadline.
Keep early transport failures as APIError for redirect fallback, but surface TimeoutError once an absolute claim deadline is exhausted so toolkit calls retain their bounded cutoff result. Keep the deadline tests ahead of their helper.
recording_claim already records every claim body a fake node sees; the
deadline test rebuilt it inline.
@CMGS
CMGS merged commit 21dc4e7 into main Sep 17, 2026
3 checks passed
@CMGS
CMGS deleted the fix/claim-deadline branch September 17, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sdk/langchain: the claim leg's redirect walk escapes the tool's wall clock

1 participant