Skip to content

feat: Persist a durable retry marker and failed-attempt stderr when the - #162

Open
andrei-hasna wants to merge 1 commit into
mainfrom
factory/860bd291-40db-48cc-b168-ee0938e4-00fae939
Open

feat: Persist a durable retry marker and failed-attempt stderr when the#162
andrei-hasna wants to merge 1 commit into
mainfrom
factory/860bd291-40db-48cc-b168-ee0938e4-00fae939

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Objective

Persist a durable retry marker and failed-attempt stderr when the loops executor retries a Codewith agent

@hasna/loops retries a Codewith agent that fails fast at start, but nothing durable records that the retry happened, so a run that needed three attempts is stored as a plain success and a worsening transient failure is invisible.

Verified on origin/main. src/lib/executor.ts announces the retry only through an optional logger, at two sites (around lines 1241 and 1413):

opts.log?.(`retrying codewith agent after transient fast start failure (${attemptIndex + 1}/${CODEWITH_START_RETRY_DELAYS_MS.length + 1})`);

No production caller supplies log. Measured with a positive control so that an empty result is not mistaken for an empty set:

$ grep -rn "onSpawn:" src/ --include=*.ts | grep -v '\.test\.'
src/daemon/daemon.ts:252
src/lib/scheduler.ts:307
src/lib/workflow-runner.ts:276          <- the grep method does find real call sites

$ grep -rn "^\s*log:" src/ --include=*.ts | grep -v '\.test\.'
(no matches)                            <- nothing supplies the logger

Compounding it, each attempt allocates a fresh BoundedOutputBuffer for stdout and stderr (around lines 1156-1157 and 1324-1325), so the failing attempt's stderr is discarded when the next attempt starts.

Consequence: a run that succeeded on the third attempt is persisted as succeeded, carrying only the last attempt's output, with zero indication that two attempts failed first. This is not a false green — the work really did succeed — but on a runner that executes unattended work, "we silently papered over a failure N times" must stay visible. Otherwise the first observable symptom is the day the retries stop being enough.

Expected behaviour after the fix: when a retry fires, evidence of it survives the run and is visible to somebody reading the stored run afterwards, without a caller having to opt in to a logger.

Required:

  • Persist a retry marker somewhere durable on the run: append it to the returned stderr, or record it on the run/receipt structure the executor returns. Whichever you choose, it must be present in what gets stored, not only in an optional callback.
  • Carry the failed attempts' captured stderr forward rather than discarding it when the buffer is reset, so the reason each attempt failed is still readable. Respect the existing output-size bound — do not let carried-forward output grow unbounded; truncate within the existing maxOutputBytes budget.
  • Apply the change to BOTH retry sites in src/lib/executor.ts; they are near-duplicates and fixing only one leaves half the paths silent.
  • Keep the existing opts.log?.() call as well; this adds a durable record, it does not replace the logger.

Acceptance:

  • A failing-first regression test that drives the executor through a simulated transient fast start failure followed by a success, and asserts (a) the returned/stored result contains a retry marker, and (b) the failing attempt's stderr text is still present in the output. Confirm the test fails before the change and passes after.
  • A test asserting that a run which succeeds on the first attempt gains no retry marker, so the marker is not noise on the normal path.
  • bun test, bun run typecheck and bun run build all pass.

Keep the change minimal and scoped to the retry accounting in src/lib/executor.ts. Do not change the retry policy itself, the delay table, the matcher that decides what counts as a transient fast start failure, or anything about how runs are scheduled.

Upstream issue: #143

Verification

  • policy source: base 4461670 (immutable commit — agent-proof)
  • containment: env — allowlist env, non-login shell, run-scoped HOME (registry auth seeded for install)
  • install: pass
  • typecheck: pass
  • build: pass
  • test: pass
  • doctor (ci): ok — 11 checks passed (1 advisory)

Run run_86c67f977070 · backend codewith · task 860bd291-40db-48cc-b168-ee0938e4b4c8
🏭 Generated by @hasnaxyz/factory


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Persist a durable retry marker and failed-attempt stderr when the loops executor retries a Codewith agent

`@hasna/loops` retries a Codewith agent that fails fast at start, but nothing durable records that the retry happened, so a run that needed three attempts is stored as a plain success and a worsening transient failure is invisible.

Verified on origin/main. `src/lib/executor.ts` announces the retry only through an optional logger, at two sites (around lines 1241 and 1413):

```ts
opts.log?.(`retrying codewith agent after transient fast start failure (${attemptIndex + 1}/${CODEWITH_START_RETRY_DELAYS_MS.length + 1})`);
```

No production caller supplies `log`. Measured with a positive control so that an empty result is not mistaken for an empty set:

```
$ grep -rn "onSpawn:" src/ --include=*.ts | grep -v '\.test\.'
src/daemon/daemon.ts:252
src/lib/scheduler.ts:307
src/lib/workflow-runner.ts:276          <- the grep method does find real call sites

$ grep -rn "^\s*log:" src/ --include=*.ts | grep -v '\.test\.'
(no matches)                            <- nothing supplies the logger
```

Compounding it, each attempt allocates a fresh `BoundedOutputBuffer` for stdout and stderr (around lines 1156-1157 and 1324-1325), so the failing attempt's stderr is discarded when the next attempt starts.

Consequence: a run that succeeded on the third attempt is persisted as `succeeded`, carrying only the last attempt's output, with zero indication that two attempts failed first. This is not a false green — the work really did succeed — but on a runner that executes unattended work, "we silently papered over a failure N times" must stay visible. Otherwise the first observable symptom is the day the retries stop being enough.

Expected behaviour after the fix: when a retry fires, evidence of it survives the run and is visible to somebody reading the stored run afterwards, without a caller having to opt in to a logger.

Required:
- Persist a retry marker somewhere durable on the run: append it to the returned `stderr`, or record it on the run/receipt structure the executor returns. Whichever you choose, it must be present in what gets stored, not only in an optional callback.
- Carry the failed attempts' captured stderr forward rather than discarding it when the buffer is reset, so the reason each attempt failed is still readable. Respect the existing output-size bound — do not let carried-forward output grow unbounded; truncate within the existing `maxOutputBytes` budget.
- Apply the change to BOTH retry sites in `src/lib/executor.ts`; they are near-duplicates and fixing only one leaves half the paths silent.
- Keep the existing `opts.log?.()` call as well; this adds a durable record, it does not replace the logger.

Acceptance:
- A failing-first regression test that drives the executor through a simulated transient fast start failure followed by a success, and asserts (a) the returned/stored result contains a retry marker, and (b) the failing attempt's stderr text is still present in the output. Confirm the test fails before the change and passes after.
- A test asserting that a run which succeeds on the first attempt gains no retry marker, so the marker is not noise on the normal path.
- `bun test`, `bun run typecheck` and `bun run build` all pass.

Keep the change minimal and scoped to the retry accounting in `src/lib/executor.ts`. Do not change the retry policy itself, the delay table, the matcher that decides what counts as a transient fast start failure, or anything about how runs are scheduled.

Upstream issue: #143

X-Factory-Run: run_86c67f977070
X-Factory-Task: 860bd291-40db-48cc-b168-ee0938e4b4c8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant