Skip to content

Commit f1d2700

Browse files
ericdalloericdallo
authored
Merge pull request #561 from rschmukler/rs/fix-sub-agent-completion-reporting
**Improve sub-agent failure reporting and make retry policy configurable**
2 parents 1210f9e + bc745f1 commit f1d2700

9 files changed

Lines changed: 413 additions & 41 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
## Unreleased
44

5+
- Return exhausted provider errors and partial output from sub-agents instead of reporting successful empty results.
6+
- Add provider `retry` config for retry counts, backoff, and chat recovery; no-output retries replay the original request.
57
- Fix OpenAI Responses SSE errors, incomplete responses, and premature stream closes so chats retry or finish instead of staying active.
68
- Return task status to LLM when changing. #556
79

‎docs/config.json‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -694,6 +694,56 @@
694694
"additionalProperties": false
695695
}
696696
},
697+
"retry": {
698+
"type": "object",
699+
"description": "Retry count and exponential backoff policy for transient provider errors. Applies to normal chats and sub-agents.",
700+
"markdownDescription": "Retry count and exponential backoff policy for transient provider errors. Applies to normal chats and sub-agents.",
701+
"properties": {
702+
"maxRetries": {
703+
"type": "integer",
704+
"minimum": 0,
705+
"default": 10,
706+
"description": "Maximum retries for rate limits, overloaded providers, and custom retry rules. The initial request is not counted.",
707+
"markdownDescription": "Maximum retries for rate limits, overloaded providers, and custom retry rules. The initial request is not counted."
708+
},
709+
"prematureStopMaxRetries": {
710+
"type": "integer",
711+
"minimum": 0,
712+
"default": 3,
713+
"description": "Maximum retries when a stream ends before a terminal completion event. The initial request is not counted.",
714+
"markdownDescription": "Maximum retries when a stream ends before a terminal completion event. The initial request is not counted."
715+
},
716+
"maxAutoContinues": {
717+
"type": "integer",
718+
"minimum": 0,
719+
"default": 3,
720+
"description": "Maximum chat-level recovery prompts after partial output or after request-level retries are exhausted. Applies to normal chats and sub-agents.",
721+
"markdownDescription": "Maximum chat-level recovery prompts after partial output or after request-level retries are exhausted. Applies to normal chats and sub-agents."
722+
},
723+
"baseDelayMs": {
724+
"type": "integer",
725+
"minimum": 0,
726+
"default": 2000,
727+
"description": "Initial exponential backoff delay in milliseconds. ECA applies jitter from 50% up to 150% of the capped delay.",
728+
"markdownDescription": "Initial exponential backoff delay in milliseconds. ECA applies jitter from 50% up to 150% of the capped delay."
729+
},
730+
"backoffMultiplier": {
731+
"type": "number",
732+
"minimum": 1,
733+
"default": 2,
734+
"description": "Multiplier applied to the backoff delay after each retry.",
735+
"markdownDescription": "Multiplier applied to the backoff delay after each retry."
736+
},
737+
"maxDelayMs": {
738+
"type": "integer",
739+
"minimum": 0,
740+
"default": 60000,
741+
"description": "Maximum exponential backoff delay before jitter, in milliseconds. Provider-supplied rate-limit reset delays use rateLimitMaxWaitSeconds instead.",
742+
"markdownDescription": "Maximum exponential backoff delay before jitter, in milliseconds. Provider-supplied rate-limit reset delays use `rateLimitMaxWaitSeconds` instead."
743+
}
744+
},
745+
"additionalProperties": false
746+
},
697747
"rateLimitMaxWaitSeconds": {
698748
"type": "integer",
699749
"minimum": 0,

‎docs/config/models.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,9 @@ Schema:
400400
| `thinkTagStart` | string | Optional override the think start tag tag for openai-chat (Default: "<think>") api | No |
401401
| `thinkTagEnd` | string | Optional override the think end tag for openai-chat (Default: "</think>") api | No |
402402
| `httpClient` | map | Allow customize the http-client for this provider requests, like changing http version | No |
403-
| `retryRules` | array | Custom retry rules that match by HTTP status and/or error pattern (see [Retry Rules](#retry-rules)) | No |
403+
| `retryRules` | array | Custom retry rules that match by HTTP status and/or error pattern (see [Retry Policy and Rules](#retry-policy-and-rules)) | No |
404+
| `retry` | map | Retry count and exponential backoff policy for transient errors; applies to normal chats and sub-agents | No |
405+
| `rateLimitMaxWaitSeconds` | integer | Maximum provider-supplied rate-limit reset wait, including ECA's one-second safety buffer (default: `60`) | No |
404406
| `extraHeaders` | map | Extra headers sent on all requests to this provider (completion and models list fetch). Model-level `extraHeaders` win on conflicts | No |
405407
| `models` | map | Key: model name, value: its config | Yes |
406408
| `models <model> extraPayload` | map | Extra payload sent in body to LLM | No |
@@ -611,9 +613,22 @@ Notes:
611613
- Authentication priority: a configured `key` (with dynamic string parse support, including `${env:OPENAI_API_KEY}`-style defaults that resolve from environment variables) takes precedence over `/login` (OAuth/subscription) auth, which in turn takes precedence over a bare `<PROVIDER>_API_KEY` env var. A configured/env key can therefore incur paid API usage even when you are logged in.
612614
- All providers with API key auth can use credential files.
613615

614-
### Retry Rules
616+
### Retry Policy and Rules
615617

616-
ECA automatically retries requests on common transient errors (429, 500, 502, 503, 529) with exponential backoff. You can define custom retry rules per provider using `retryRules` to handle additional status codes or error patterns.
618+
ECA automatically retries requests on common transient errors (429, 500, 502, 503, 529), provider overload errors, and premature stream termination. The provider-level `retry` object controls the retry budget and exponential backoff for both normal chats and sub-agents:
619+
620+
| Option | Default | Description |
621+
|--------|---------|-------------|
622+
| `maxRetries` | `10` | Retries for rate limits, overloaded providers, and custom retry rules; excludes the initial request |
623+
| `prematureStopMaxRetries` | `3` | Retries when a stream ends before a terminal completion event; excludes the initial request |
624+
| `maxAutoContinues` | `3` | Chat-level recovery prompts after partial output or exhausted request retries; applies to normal chats and sub-agents |
625+
| `baseDelayMs` | `2000` | Initial backoff delay; ECA applies jitter from 50% to 150% |
626+
| `backoffMultiplier` | `2` | Multiplier applied after each retry |
627+
| `maxDelayMs` | `60000` | Backoff cap before jitter; provider reset waits are controlled separately by `rateLimitMaxWaitSeconds` |
628+
629+
A value of `0` for either retry count disables request retries in that category; `maxAutoContinues: 0` disables chat-level recovery prompts. If a provider supplies an explicit rate-limit reset time, that delay takes precedence over exponential backoff and is accepted only when it does not exceed `rateLimitMaxWaitSeconds`.
630+
631+
You can define custom retry rules per provider using `retryRules` to handle additional status codes or error patterns.
617632

618633
Each rule can match by:
619634

@@ -630,6 +645,14 @@ At least one of `status` or `errorPattern` is required. When both are specified,
630645
"api": "openai-chat",
631646
"url": "${env:MY_COMPANY_API_URL}",
632647
"key": "${env:MY_COMPANY_API_KEY}",
648+
"retry": {
649+
"maxRetries": 15,
650+
"prematureStopMaxRetries": 5,
651+
"maxAutoContinues": 5,
652+
"baseDelayMs": 2000,
653+
"backoffMultiplier": 2,
654+
"maxDelayMs": 60000
655+
},
633656
"retryRules": [
634657
{"status": 418, "label": "Corporate proxy throttle"},
635658
{"errorPattern": "capacity.*exceeded", "label": "Capacity exceeded"},

‎src/eca/features/chat.clj‎

Lines changed: 50 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,26 @@
3434

3535
(def ^:private logger-tag "[CHAT]")
3636

37-
(def ^:private max-auto-continues
38-
"Max times a single prompt chain is auto-continued after interrupted or
39-
truncated responses (e.g. proxies dropping long streaming connections #547)."
40-
3)
37+
(def ^:private default-max-auto-continues 3)
38+
39+
(defn ^:private provider-max-auto-continues [config provider]
40+
(let [configured (get-in config [:providers provider :retry :maxAutoContinues])]
41+
(if (and (integer? configured) (not (neg? configured)))
42+
(long configured)
43+
default-max-auto-continues)))
44+
45+
(defn ^:private prompt-error-data
46+
"Returns serializable terminal prompt error details for chat state."
47+
[{:keys [message exception status code request-id response-id rate-limit-resets-at]} error-type]
48+
(assoc-some {:message (or (shared/not-blank message)
49+
(some-> exception ex-message shared/not-blank)
50+
"Unknown provider error")
51+
:error-type error-type}
52+
:status status
53+
:code code
54+
:request-id request-id
55+
:response-id response-id
56+
:rate-limit-resets-at rate-limit-resets-at))
4157

4258
(defn ^:private tool-output-text [msg]
4359
(let [contents (get-in msg [:content :output :contents])]
@@ -902,7 +918,7 @@
902918
(logger/info logger-tag "Superseding active prompt" {:chat-id chat-id
903919
:status (get-in @db* [:chats chat-id :status])}))
904920
(swap! db* assoc-in [:chats chat-id :status] :running)
905-
(swap! db* update-in [:chats chat-id] dissoc :prompt-finished?)
921+
(swap! db* update-in [:chats chat-id] dissoc :prompt-finished? :prompt-error)
906922
(swap! db* assoc-in [:chats chat-id :updated-at] (System/currentTimeMillis))
907923
(messenger/chat-status-changed messenger {:chat-id chat-id :status :running})
908924
(lifecycle/trigger-chat-status-hook! chat-ctx)
@@ -922,6 +938,7 @@
922938
model-capabilities (get-in db [:models full-model])
923939
provider-auth (get-in @db* [:auth provider])
924940
all-tools (f.tools/all-tools chat-id agent @db* config)
941+
auto-continue-limit (provider-max-auto-continues config provider)
925942
received-msgs* (atom "")
926943
reasonings* (atom {})
927944
server-tool-times* (atom {})
@@ -1141,7 +1158,7 @@
11411158
(not (string/blank? response-text))
11421159
(or (:premature? msg)
11431160
(truncated-response? response-text))
1144-
(< (:auto-continue-count chat-ctx 0) max-auto-continues)
1161+
(< (:auto-continue-count chat-ctx 0) auto-continue-limit)
11451162
(not (or (:on-finished-side-effect chat-ctx)
11461163
(:on-after-finish! chat-ctx))))
11471164
(do
@@ -1430,10 +1447,24 @@
14301447
(let [partial-text @received-msgs*
14311448
transient-error? (contains? #{:overloaded :premature-stop} error-type)
14321449
stopping? (identical? :stopping (get-in @db* [:chats chat-id :status]))
1450+
user-messages-recorded? (boolean
1451+
(when-let [user-content-id (:user-content-id chat-ctx)]
1452+
(some #(= user-content-id (:content-id %))
1453+
(get-in @db* [:chats chat-id :messages]))))
1454+
continue-existing-response? (or (not (string/blank? partial-text))
1455+
user-messages-recorded?)
1456+
retry-messages (if continue-existing-response?
1457+
[{:role "user"
1458+
:content [{:type :text
1459+
:text "Your previous response was interrupted mid-stream. Continue from where you left off, do not redo completed steps."}]}]
1460+
user-messages)
1461+
retry-source-type (if continue-existing-response?
1462+
:auto-continue
1463+
:transient-error-retry)
14331464
can-auto-continue? (and (not stopping?)
14341465
(or transient-error?
14351466
(string/includes? (or message "") "idle timeout"))
1436-
(< (:auto-continue-count chat-ctx 0) max-auto-continues)
1467+
(< (:auto-continue-count chat-ctx 0) auto-continue-limit)
14371468
(not (or (:on-finished-side-effect chat-ctx)
14381469
(:on-after-finish! chat-ctx)))
14391470
(not compacting?))]
@@ -1447,7 +1478,12 @@
14471478
(logger/info logger-tag "Transient error during response, auto-continuing"
14481479
{:chat-id chat-id :error-type error-type})
14491480
(lifecycle/send-content! chat-ctx :system
1450-
{:type :progress :state :running :text (str (or message "Connection interrupted") ", continuing...")})
1481+
{:type :progress
1482+
:state :running
1483+
:text (str (or message "Connection interrupted")
1484+
(if continue-existing-response?
1485+
", continuing..."
1486+
", retrying original request..."))})
14511487
(swap! db* assoc-in [:chats chat-id :auto-compacting?] true)
14521488
(lifecycle/finish-chat-prompt! :idle
14531489
(assoc chat-ctx
@@ -1457,13 +1493,13 @@
14571493
:on-after-finish!
14581494
(fn []
14591495
(prompt-messages!
1460-
[{:role "user"
1461-
:content [{:type :text
1462-
:text "Your previous response was interrupted mid-stream. Continue from where you left off, do not redo completed steps."}]}]
1463-
:auto-continue
1496+
retry-messages
1497+
retry-source-type
14641498
(update chat-ctx :auto-continue-count (fnil inc 0)))))))
14651499
(do
14661500
(when-not stopping?
1501+
(swap! db* assoc-in [:chats chat-id :prompt-error]
1502+
(prompt-error-data error-data error-type))
14671503
(lifecycle/send-content! chat-ctx :system
14681504
{:type :text
14691505
:text (if (= :context-overflow error-type)
@@ -1485,6 +1521,8 @@
14851521
(catch Exception e
14861522
(when-not (:silent? (ex-data e))
14871523
(logger/error e)
1524+
(swap! db* assoc-in [:chats chat-id :prompt-error]
1525+
(prompt-error-data {:exception e} :unknown))
14881526
(swap! db* update-in [:chats chat-id] dissoc :auto-compacting? :compacting?)
14891527
(when-not (string/blank? @received-msgs*)
14901528
(add-to-history! {:role "assistant"

0 commit comments

Comments
 (0)