feat: Persist a durable retry marker and failed-attempt stderr when the - #162
Open
andrei-hasna wants to merge 1 commit into
Open
feat: Persist a durable retry marker and failed-attempt stderr when the#162andrei-hasna wants to merge 1 commit into
andrei-hasna wants to merge 1 commit into
Conversation
Persist a durable retry marker and failed-attempt stderr when the loops executor retries a Codewith agent
`@hasna/loops` retries a Codewith agent that fails fast at start, but nothing durable records that the retry happened, so a run that needed three attempts is stored as a plain success and a worsening transient failure is invisible.
Verified on origin/main. `src/lib/executor.ts` announces the retry only through an optional logger, at two sites (around lines 1241 and 1413):
```ts
opts.log?.(`retrying codewith agent after transient fast start failure (${attemptIndex + 1}/${CODEWITH_START_RETRY_DELAYS_MS.length + 1})`);
```
No production caller supplies `log`. Measured with a positive control so that an empty result is not mistaken for an empty set:
```
$ grep -rn "onSpawn:" src/ --include=*.ts | grep -v '\.test\.'
src/daemon/daemon.ts:252
src/lib/scheduler.ts:307
src/lib/workflow-runner.ts:276 <- the grep method does find real call sites
$ grep -rn "^\s*log:" src/ --include=*.ts | grep -v '\.test\.'
(no matches) <- nothing supplies the logger
```
Compounding it, each attempt allocates a fresh `BoundedOutputBuffer` for stdout and stderr (around lines 1156-1157 and 1324-1325), so the failing attempt's stderr is discarded when the next attempt starts.
Consequence: a run that succeeded on the third attempt is persisted as `succeeded`, carrying only the last attempt's output, with zero indication that two attempts failed first. This is not a false green — the work really did succeed — but on a runner that executes unattended work, "we silently papered over a failure N times" must stay visible. Otherwise the first observable symptom is the day the retries stop being enough.
Expected behaviour after the fix: when a retry fires, evidence of it survives the run and is visible to somebody reading the stored run afterwards, without a caller having to opt in to a logger.
Required:
- Persist a retry marker somewhere durable on the run: append it to the returned `stderr`, or record it on the run/receipt structure the executor returns. Whichever you choose, it must be present in what gets stored, not only in an optional callback.
- Carry the failed attempts' captured stderr forward rather than discarding it when the buffer is reset, so the reason each attempt failed is still readable. Respect the existing output-size bound — do not let carried-forward output grow unbounded; truncate within the existing `maxOutputBytes` budget.
- Apply the change to BOTH retry sites in `src/lib/executor.ts`; they are near-duplicates and fixing only one leaves half the paths silent.
- Keep the existing `opts.log?.()` call as well; this adds a durable record, it does not replace the logger.
Acceptance:
- A failing-first regression test that drives the executor through a simulated transient fast start failure followed by a success, and asserts (a) the returned/stored result contains a retry marker, and (b) the failing attempt's stderr text is still present in the output. Confirm the test fails before the change and passes after.
- A test asserting that a run which succeeds on the first attempt gains no retry marker, so the marker is not noise on the normal path.
- `bun test`, `bun run typecheck` and `bun run build` all pass.
Keep the change minimal and scoped to the retry accounting in `src/lib/executor.ts`. Do not change the retry policy itself, the delay table, the matcher that decides what counts as a transient fast start failure, or anything about how runs are scheduled.
Upstream issue: #143
X-Factory-Run: run_86c67f977070
X-Factory-Task: 860bd291-40db-48cc-b168-ee0938e4b4c8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Objective
Persist a durable retry marker and failed-attempt stderr when the loops executor retries a Codewith agent
@hasna/loopsretries a Codewith agent that fails fast at start, but nothing durable records that the retry happened, so a run that needed three attempts is stored as a plain success and a worsening transient failure is invisible.Verified on origin/main.
src/lib/executor.tsannounces the retry only through an optional logger, at two sites (around lines 1241 and 1413):No production caller supplies
log. Measured with a positive control so that an empty result is not mistaken for an empty set:Compounding it, each attempt allocates a fresh
BoundedOutputBufferfor stdout and stderr (around lines 1156-1157 and 1324-1325), so the failing attempt's stderr is discarded when the next attempt starts.Consequence: a run that succeeded on the third attempt is persisted as
succeeded, carrying only the last attempt's output, with zero indication that two attempts failed first. This is not a false green — the work really did succeed — but on a runner that executes unattended work, "we silently papered over a failure N times" must stay visible. Otherwise the first observable symptom is the day the retries stop being enough.Expected behaviour after the fix: when a retry fires, evidence of it survives the run and is visible to somebody reading the stored run afterwards, without a caller having to opt in to a logger.
Required:
stderr, or record it on the run/receipt structure the executor returns. Whichever you choose, it must be present in what gets stored, not only in an optional callback.maxOutputBytesbudget.src/lib/executor.ts; they are near-duplicates and fixing only one leaves half the paths silent.opts.log?.()call as well; this adds a durable record, it does not replace the logger.Acceptance:
bun test,bun run typecheckandbun run buildall pass.Keep the change minimal and scoped to the retry accounting in
src/lib/executor.ts. Do not change the retry policy itself, the delay table, the matcher that decides what counts as a transient fast start failure, or anything about how runs are scheduled.Upstream issue: #143
Verification
Run
run_86c67f977070· backendcodewith· task860bd291-40db-48cc-b168-ee0938e4b4c8🏭 Generated by @hasnaxyz/factory
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.