feat: enforce V3 provider errors, deadlines and bounded retries

docs/index.md

@@ -28,6 +28,7 @@ This index covers the current product guides, developer references, and dated en
 - [Modular refactor validation](modular-refactor-validation.md), [production audit](production-audit-2026-10-04.md), [shipping audit](production-shipping-audit-2026-10-04.md), and [self-learning audit](self-learning-audit-2026-10-03.md) preserve historical evidence.
 - [V3 architecture ownership plan](superpowers/plans/2026-10-07-v3-issue-227.md) records the scoped implementation and verification for issue #227.
 - [V3 provider contract plan](superpowers/plans/2026-10-08-v3-issue-228.md) records the structured response and compatibility migration for issue #228.
+- [V3 provider error plan](superpowers/plans/2026-10-09-v3-issue-231.md) records typed failures, cancellation and bounded retry for issue #231.
 - Machine-readable audit receipts and benchmark inputs remain beside their corresponding reports under `docs/` and `docs/benchmarks/`.
 
 The repository's [English overview](../README.md) links here. Technical guides are maintained in English; README translations provide localized entry points.

docs/providers.md

@@ -101,3 +101,40 @@ live provider interoperability.
 ## Costs and changing provider catalogs
 
 The cost ledger attributes usage to the provider and model returned by the integration, including streaming usage when provided. Displayed costs can be estimates when a provider does not report token usage or published pricing is missing. Do not use a README model label or price as evidence that your account can access that model. See [usage and cost](usage-and-cost.md) for price provenance and interpretation.
+
+## Provider failures and request limits
+
+Provider failures raise SDK-independent `ProviderError`, exported with
+`ProviderErrorKind`. Kinds distinguish transport, transport timeout, rate limit,
+server failure, context overflow, authentication, invalid request, cancellation
+and unknown failures. Errors retain provider/model identity, available HTTP
+status and provider code, retry eligibility and bounded Retry-After information.
+Displayed messages omit request bodies and credentials; the original exception
+remains chained for private debugging and must not be exposed as a public traceback.
+`ProviderContractError` remains a separate terminal response-validation failure.
+
+Each logical request shares one wall-clock deadline and at most four transport
+attempts across retries and fallback. Runtime defaults remain 90 seconds for
+foreground calls and 180 seconds for child calls. `KYROZEN_PROVIDER_TIMEOUT_SECONDS`
+accepts finite values clamped to 1–600 seconds; invalid/nonfinite values use the
+default. Native SDK retries are disabled and transport timeouts use remaining time.
+Standalone adapter calls use the same bounded worker with a 90-second deadline.
+Pausing a direct stream does not retain its request context in the caller.
+
+Transient transport, timeout, rate-limit and server failures use exponential
+backoff from one second with jitter, respecting every transient candidate's
+Retry-After. If the required wait cannot fit before the deadline, the classified
+failure returns immediately.
+Fallback tries configured providers in order before repeating transient failures.
+Authentication can advance to another provider but never repeats the failed
+credentials. Invalid requests, context overflow, unknown failures, contract
+failures, cancellation and deadline expiry stop immediately. This intentionally
+replaces fallback on arbitrary exceptions. After a streaming chunk or a successful
+response, failure never causes regeneration or provider switching.
+
+Structured runtime callers receive typed errors. Existing text callers retain
+`[LLM Error]` rendering and the existing single context-compaction recovery.
+Provider retry does not execute or retry tools. Child cancellation and foreground
+interruption stop waiting and prevent later transport attempts/output. Streams are
+closed when the SDK permits; an uncooperative legacy transport may remain in a
+daemon worker until it returns because Python cannot forcibly stop that thread.

docs/superpowers/plans/2026-10-09-v3-issue-231.md

@@ -0,0 +1,82 @@
+# V3 provider errors, timeouts, and retries implementation plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking.
+
+**Goal:** Complete R008, R009 and R167 in [issue #231](https://github.com/EvanProgramming/OpenKyrozen/issues/231).
+
+**Architecture:** Provider-owned errors normalize transport failures. A context-scoped deadline, cancellation event and four-attempt budget are shared by adapters, runtime workers and fallback. Successful response decoding stays outside retry.
+
+**Tech Stack:** Python 3.12/3.13, standard-library contextvars/threading, existing provider SDKs, unittest.
+
+**Spec:** The user-approved issue #231 plan and GitHub source requirements.
+
+## Global constraints
+
+- One issue and one PR; branch Evan/v3-231-provider-errors starts at current main. No merge.
+- No new dependency or paid calls. Preserve successful response contracts, model mapping and usage attribution.
+- Default deadlines: 90 seconds foreground, 180 child; finite override clamped 1–600 seconds.
+- Four transport attempts total, including fallback; disable SDK retries and pass remaining timeout.
+- Retry transport, transport timeout, rate limit and server failures only. Authentication may advance to another provider without repeating credentials. Other failures and overall expiry/cancellation are terminal.
+- Try fallback candidates in order before repeating transient failures. No replay after a streamed chunk or received response.
+- Legacy text boundary retains error rendering and one context-compaction recovery. Tool retry and full tool/process cancellation are out of scope.
+
+## Review focus
+
+- SDK retries must not multiply the shared attempt limit.
+- Cancelled/timed-out workers must not start another request or publish late output.
+- Structured context errors must still trigger exactly one existing compaction attempt.
+- Errors and chained fallback summaries must not display API keys or request bodies.
+- Partial streams and successful responses must never be replayed; successful usage remains charged once.
+
+## Task 1: Normalize provider failures
+
+**Files:** providers/errors.py, providers/__init__.py, tests/test_provider_errors.py; adapter transport boundaries.
+**Interfaces:** ProviderErrorKind; ProviderError(kind, provider, model, status_code, provider_code, retry_after, attempts); normalize_provider_error(exc, provider, model).
+
+- [x] Add regressions for status/code classification, safe diagnostics, typed errors and terminal contract errors; watch them fail.
+- [x] Implement provider-owned normalization using optional SDK exception types and structured status/code fields, with narrow context-overflow matching for legacy exceptions.
+- [x] Preserve original causes; expose only bounded safe diagnostic fields. Run focused tests.
+
+## Task 2: Bound retries and fallback
+
+**Files:** providers/retry.py, providers/fallback.py, adapter transports; tests/test_provider_errors.py, tests/test_providers.py.
+**Interfaces:** provider_request_scope(timeout=90, cancelled=None); remaining_timeout(); shared request state; existing _retry_with_backoff gains provider/model identity.
+
+- [x] Add failing tests for message-only false positives, four-attempt shared limit, candidate order, authentication exclusion and partial-stream replay.
+- [x] Share monotonic deadline/cancellation/attempt state through contextvars. Preserve standalone adapter calls by opening a default request scope when none exists.
+- [x] Disable SDK retries and set finite transport timeouts for every adapter. Use a one-attempt adapter mode inside fallback, retaining the same global counter.
+- [x] Back off from one second with bounded jitter; respect bounded Retry-After and interrupt waits. Stop terminal failures and never retry received-response decoding.
+- [x] Normalize iteration failures; close streams in finally. Run focused tests on both supported Python versions.
+
+## Task 3: Integrate runtime lifecycle and compatibility
+
+**Files:** providers/calls.py, agent/context.py; runtime and delegation tests.
+**Interfaces:** Existing runtime methods/signatures preserved; structured boundary raises ProviderError; explicit text boundary renders legacy errors.
+
+- [x] Add failing tests for cancellation/deadline during requests and backoff, stalled streams, late output, nonfinite timeout settings and compaction compatibility.
+- [x] Enter shared scope before copying context into workers. Poll cancellation/deadline, set local cancellation on every foreground exit, reuse child coordinator events without cancelling completed child runs.
+- [x] Keep usage ledger and context bookkeeping, ProviderUnavailableError and ContextOverflowError compatibility.
+- [x] Use local HTTP servers/real installed SDKs to prove timeout settings, disabled hidden retries and request counts; verify no tool execution.
+
+## Task 4: Verify and deliver
+
+- [x] Update provider documentation and index; include worker-termination limitations and retry/fallback compatibility changes.
+- [x] Run focused 3.12/3.13 tests; full make test/check/lint/docs-check and offline agent/subagent acceptance. Report blocked checks distinctly.
+- [x] Obtain fresh independent whole-branch review and reproduce/fix material findings with regressions.
+- [ ] GPG-sign task-scoped commit(s), verify signatures locally, push issue branch, create and attach one PR.
+- [ ] Verify GitHub signatures and wait for CI on the final commit. Leave PR unmerged and stop.
+
+## Verification record
+
+- RED→GREEN: message-only retry, typed status, fallback ordering/budget, partial-stream replay, HTTPX subtype classification, runtime/nested fallback entry, received metadata-only stream failure and malformed error metadata.
+- Real SDK local HTTP tests cover all nine adapter families and streaming initialization; no paid calls. Google Gen AI 1.0.0 compatibility was additionally exercised through an extracted public wheel.
+- Independent whole-branch review found three material defects; each was reproduced, fixed and independently rechecked. No remaining material implementation findings were reported.
+- Final command outcomes and signed commit/CI evidence are recorded in the pull request; live-provider interoperability is not claimed by these synthetic checks.
+
+Local final verification: 597 full Python 3.12 tests; 96 focused Python 3.13 tests; make check/lint/docs-check and offline agent/subagent acceptance PASS. No paid calls.
+
+Final follow-up regressions cover explicit retry ownership on dynamic callables, all-candidate Retry-After, and cancellation synchronized during actual backoff. Independent review and all local checks pass.
+
+## Automated review follow-up
+
+GitHub findings were reproduced before fixes. Runtime and standalone adapters now reuse provider-owned bounded workers, including continuous-trickle responses. Stream request context remains in its producer worker; completed borrowed scopes stay usable while aborted consumption cancels. Azure identity and truncated Requests responses classify correctly, mid-stream attempt counts persist, unread error JSON does not replace the original failure, exhausted budgets remain typed, and impossible Retry-After waits fail immediately with the classified error. Legacy context sentinels also become typed at the structured boundary. Fresh whole-branch independent review reports no material findings.

openkyrozen/agent/context.py

@@ -67,12 +67,8 @@ def _cache_reported_context_tokens(self, messages: list[dict], prompt_tokens: in
 
 
 def _is_context_overflow_error(self, exc: Exception) -> bool:
-    text = str(exc).lower()
-    return any(marker in text for marker in (
-        "context length", "context window", "maximum context", "max context",
-        "prompt is too long", "input is too long", "too many tokens", "token limit",
-        "exceeds the context", "exceeded context", "context_limit",
-    ))
+    from openkyrozen.providers.errors import legacy_context_overflow
+    return legacy_context_overflow(exc)
 
 
 def _summarize_context_with_chat_model(self, text: str, output_chars: int, model: str) -> str | None:

openkyrozen/providers/__init__.py

@@ -16,3 +16,5 @@
 from openkyrozen.security.credentials import (_get_encryption_key, _get_fernet, encrypt_api_key, decrypt_api_key, save_provider_config_encrypted)
 
 from openkyrozen.providers.models import ModelResponse, ToolCall, ProviderCapabilities, FinishReason, ProviderContractError
+
+from openkyrozen.providers.errors import ProviderError, ProviderErrorKind

openkyrozen/providers/anthropic.py

@@ -7,7 +7,7 @@
 from openkyrozen.providers.base import LLMProvider
 from openkyrozen.providers.models import received_response, ModelResponse, model_response
 from openkyrozen.providers.config import ProviderConfig
-from openkyrozen.providers.retry import _retry_with_backoff
+from openkyrozen.providers.retry import _retry_with_backoff, provider_call, provider_stream, remaining_timeout, close_stream, mark_response_received
 
 
 class AnthropicProvider(LLMProvider):
@@ -25,7 +25,7 @@ def __init__(self, config: ProviderConfig) -> None:
         kwargs: dict[str, Any] = {"api_key": config.api_key}
         if config.base_url:
             kwargs["base_url"] = config.base_url
-        self._client = anthropic.Anthropic(**kwargs)
+        self._client = anthropic.Anthropic(max_retries=0, timeout=90.0, **kwargs)
 
     def _prepare_messages(self, messages):
         system_prompts: list[str] = []
@@ -42,6 +42,7 @@ def _prepare_messages(self, messages):
     def chat(self, messages: list[dict[str, str]], model: str | None = None) -> tuple[str, dict | None]:
         return self.chat_response(messages, model).as_legacy_tuple()
 
+    @provider_call
     def chat_response(self, messages: list[dict[str, str]], model: str | None = None) -> ModelResponse:
         model = model or self.config.model_simple
         system_prompts, claude_messages = self._prepare_messages(messages)
@@ -56,7 +57,7 @@ def chat_response(self, messages: list[dict[str, str]], model: str | None = None
             kwargs["system"] = "\n\n".join(system_prompts)
 
         def _call():
-            return self._client.messages.create(**kwargs)
+            return self._client.messages.create(**kwargs, timeout=remaining_timeout())
 
         response = _retry_with_backoff(_call)
         with received_response():
@@ -79,6 +80,7 @@ def _call():
                                   calls=calls, response_id=getattr(response, "id", None),
                                   raw_finish_reason=getattr(response, "stop_reason", None))
 
+    @provider_stream
     def chat_stream(self, messages: list[dict[str, str]], model: str | None = None) -> Iterator[str]:
         model = model or self.config.model_simple
         system_prompts, claude_messages = self._prepare_messages(messages)
@@ -95,11 +97,12 @@ def chat_stream(self, messages: list[dict[str, str]], model: str | None = None)
         completed = False
 
         def _call():
-            return self._client.messages.stream(**kwargs)
+            return self._client.messages.stream(**kwargs, timeout=remaining_timeout())
 
         stream_context = _retry_with_backoff(_call)
         try:
             with stream_context as stream:
+                mark_response_received()
                 for delta in stream.text_stream:
                     collected.append(delta)
                     yield delta

openkyrozen/providers/azure.py

@@ -31,4 +31,4 @@ def __init__(self, config: ProviderConfig) -> None:
                 )
             except ImportError:
                 pass
-        self._client = OpenAI(**kwargs)
+        self._client = OpenAI(max_retries=0, timeout=90.0, **kwargs)

openkyrozen/providers/base.py

@@ -5,6 +5,8 @@
 from inspect import getattr_static
 from openkyrozen.providers.models import ModelResponse, ProviderCapabilities, ProviderContractError
 from openkyrozen.providers.config import ProviderConfig
+from openkyrozen.providers.retry import provider_call, provider_stream, provider_request_scope, _retry_with_backoff
+from openkyrozen.providers.errors import normalize_provider_error
 
 class LLMProvider(ABC):
     """Unified interface for all LLM backends."""
@@ -17,6 +19,7 @@ def chat(self, messages: list[dict[str, str]], model: str | None = None) -> tupl
         """Send messages to the LLM. Returns (content, usage_dict_or_None)."""
         ...
 
+    @provider_call
     def chat_response(self, messages: list[dict], model: str | None = None) -> ModelResponse:
         """Bridge providers implementing only the existing text contract."""
         text, usage = self.chat(messages, model)
@@ -27,6 +30,7 @@ def get_capabilities(self, model: str | None = None) -> ProviderCapabilities:
         """Only advertise features usable through the shipped adapter API."""
         return ProviderCapabilities(text_streaming=type(self).chat_stream is not LLMProvider.chat_stream)
 
+    @provider_stream
     def chat_stream(self, messages: list[dict[str, str]], model: str | None = None) -> Iterator[str]:
         """Stream response tokens. Default: fall back to non-streaming chat()."""
         text, _ = self.chat(messages, model)
@@ -37,7 +41,7 @@ def name(self) -> str:
         return self.config.provider
 
 
-def get_model_response(provider, messages, model=None) -> ModelResponse:
+def _get_model_response(provider, messages, model=None) -> ModelResponse:
     """Use an explicitly supplied structured API, or bridge a legacy duck provider."""
     if callable(getattr_static(provider, "chat_response", None)):
         response = provider.chat_response(messages, model)
@@ -46,3 +50,15 @@ def get_model_response(provider, messages, model=None) -> ModelResponse:
         return response
     text, usage = provider.chat(messages, model)
     return ModelResponse(text=text, usage=usage, metadata={"legacy": True})
+
+
+def get_model_response(provider, messages, model=None) -> ModelResponse:
+    with provider_request_scope():
+        method = getattr_static(provider, "chat_response", None)
+        if getattr(method, "_provider_retry_managed", False) is True:
+            try:
+                return _get_model_response(provider, messages, model)
+            except Exception as exc:
+                raise normalize_provider_error(exc, getattr(provider, "name", None), model)
+        return _retry_with_backoff(lambda: _get_model_response(provider, messages, model),
+                                   provider=getattr(provider, "name", None), model=model)