Repository navigation
feat(retry): Retry webhooks, LLM proxy, guardrails and provider SDK calls - #1275
Open
Prajna1999 wants to merge 6 commits into
Open
Prajna1999 wants to merge 6 commits into
Prajna1999 wants to merge 6 commits into
Conversation
Add a thin tenacity decorator, retry_thrice, and a shared is_transient predicate in app/core/retry.py. - send_callback retries any non-2xx response or request error and re-signs the payload on each attempt. - The provider call in execute_llm_call retries 5xx errors. - The client-LLM proxy and guardrails calls retry 5xx and connect failures. Already-retried failures are marked retryable=False so fast-eval retries don't stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
changed the title
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one): 1
Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
OpenAPI changes ⚪ No API surface changesNote This PR does not modify the API contract.
|
Drop the retry_thrice wrapper. Call sites now use tenacity's @Retry with stop_after_attempt(3) and exponential backoff. app/core/retry.py keeps only the shared is_transient predicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the provider-call retry from execute_llm_call. Provider SDK retries move to a follow-up PR stacked on the google-genai SDK update. Provider failures keep retryable=True as before, and is_transient now only classifies HTTP exceptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prajna1999
changed the title
Use each SDK's built-in retry: max_retries=3 on the OpenAI and Anthropic
clients, and request_options={"max_retries": 3} on Sarvam STT/TTS calls.
Verified against a stub server: 4 attempts on 503, and retried Sarvam
uploads still carry the file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prajna1999
changed the title
Match the provider SDKs' max_retries=3 so webhooks, the LLM proxy and guardrails also retry three times after the first attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The SDK's max_retries is unusable: its counter starts at 2, and streaming TTS never retries. Wrap STT convert and the TTS stream consumption in a tenacity @Retry (4 attempts, 5xx/connect only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
No linked issue.
Summary
Transient failures in synchronous outbound calls currently fail a job on the first attempt. This covers a provider 5xx, a blip reaching the guardrails service, the client LLM proxy returning a 502, and a webhook receiver timing out. This PR adds 3 retries (4 attempts in total) to each of these. Batch providers are out of scope.
Kaapi's own HTTP calls: tenacity
@retryEach of these uses
stop_after_attempt(4)(3 retries),wait_exponential(max=10)and one shared retry check,app/core/retry.py::is_transient. The check is true for an HTTP 5xx, or for a connection that never reached upstream.utils.send_callback(webhooks)requests.RequestException. The HMAC is re-signed on each attempt.jobs._post_to_proxy(client LLM proxy)ReadTimeout(the call may already have run and been billed), 4xxguardrails.run_guardrails_validation/list_validators_configguardrails.proxy_guardrails_request(management API)Provider calls: each SDK's built-in retry
I checked each setting exists in the installed SDK version, then ran it against a fake server that always returns 503:
OpenAI(..., max_retries=3)Anthropic(..., max_retries=3)request_options={"max_retries": 3}per call@retryineleven_ai.py(5xx / connect only)The SDKs retry 408/409/429/5xx with backoff, and OpenAI/Anthropic also retry connection errors and timeouts.
Behaviour change: a proxy failure that was already retried now comes back as
BlockResult.retryable=False. This keeps fast-evalretry_llm_callfrom stacking more attempts. Provider failures keepretryable=Truefor now.The wiki pages are updated:
llm-call.md,guardrails.md,cross-cutting/exceptions.md.Checklist
fastapi run --reload app/main.pyordocker compose upin the repository root and test.Notes
max_retriesdoesn't work as documented: its retry counter starts at 2, somax_retries=3gives only 1 retry, and streaming TTS never retries.HttpRetryOptions(attempts=3)at client level.-n 0). Backoff is zeroed with an opt-inno_retry_backofffixture.api/routes/threads.pyhas its own unvalidatedsend_callbackwith no retry. It's untouched.🤖 Generated with Claude Code