Skip to content

feat(v2-orchestrator): optionally ask to retry a failed unit lane stage straight away - #766

Open
thenom wants to merge 1 commit into
aws-samples:mainfrom
thenom:feat/lane-failure-retry-761
Open

thenom wants to merge 1 commit into
aws-samples:mainfrom
thenom:feat/lane-failure-retry-761

Conversation

@thenom

@thenom thenom commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #761

Problem

When a stage fails in a parallel unit lane, the lane ends and the section asks retry, skip or abort only once every other lane in the round has settled. With PR per unit that includes lanes waiting for a person to review and merge, so a failed unit can wait days for a retry while its dependents stay blocked.

Solution

A new opt-in Terraform variable, lane_failure_policy ("halt" by default, which leaves behaviour unchanged, or "ask"), passed to the v2 orchestrator as LANE_FAILURE_POLICY. With "ask", a failed lane stage opens a lane gate owned by the unit, with options retry and leave-failed:

  • Lane gate: unlike a barrier engine gate it never parks the execution. createHumanTask writes it in the same transaction as the run-ownership check on META, but leaves META's status and pendingHumanTaskId alone, so sibling lanes stay RUNNING (the credential broker serves only CREATED and RUNNING executions). It is unparked through unparkGate's lane path, as lane questions are. The row carries laneGate: true, and the intents API's resumeOwed treats an answered, unconsumed lane gate of the current run as owed, so "Resume run" still works after a failed callback send.
  • While it waits: the lane frees its concurrency permit and stops its runtime session. The unit row is FAILED with a blockedReason; dependents wait on the lane rather than being blocked.
  • Retry runs the stage again on the same unit branch, keeping completed stages, with its own durable identities (-a<n>, as halt-and-ask rounds use -r<n>). The agent is told why the last attempt failed, with the remote's fuller output for a rejected push, and given any guidance in the answer. After a rejected push the engine first undoes the unpushed commits (discardUnpushedCommits), keeping their changes in the working tree, so the retry can remove what the remote rejected. A repeat failure asks again.
  • Leave-failed ends the lane as before, so halt-and-ask still offers retry, skip and abort. An unrecognised answer is asked again up to HALT_REASK_LIMIT, then treated as leave-failed. A superseded gate retires the run.
  • Events: v2.unit.retry and v2.unit.retry_declined.
  • Durability: the policy is read in the run's existing load-meta checkpoint, as completionPolicyVersion and laneWaitPolicyVersion are. A history recorded before the change has no field and keeps halting, so no steps are added or reordered for runs in progress. With halt the step sequence is unchanged.
  • Docs: docs/concepts/execution.md.

Known limits: only stage failures in the lane's stage loop ask; failures in init-lane, merge-back, reconcile, unit PR creation or conflict resolution still end the lane. The gate shows with the intent's other pending gates, not on the unit's card.

This and #764 both add a block to run-stage.js before the CLI starts, so whichever merges second needs a small rebase.

User experience

With the setting on, a failed unit shows "Waiting for a decision: retry this stage or leave it failed", and a gate for that unit appears straight away while the other units keep running. The gate card's options are labelled for a lane retry.

Testing

  • Orchestrator: retry without halt-and-ask with fresh callback identities; the gate's unit ownership, options and lane scope (no META status or pointer written or conditioned on); unit state transitions and session release; a retry that fails again; leave-failed falling back to halt-and-ask; re-asking an unrecognised answer; a superseded gate retiring the run; off by default.
  • Store (the lane gate transaction has no status or pointer), intents (resume owed for a lane gate of the current run, not a retired one), git-engine (undoing unpushed commits) and run-stage (retry prompt and undo).
  • Full backend vitest: 7931 passed. The only failures are the four yjs-server files, which fail to import yjs from the root locally, as on clean main. Frontend: tsc -b --noEmit and the full suite (1321 tests).
  • npm run lint (warnings only), npm run format:check, npm run dep:check, the pre-commit hook.
  • Terraform: fmt -check, validate -no-tests with the CI backend override, and the seven test suites as in CI except lambda-packaging.
  • The lane gate is deployed with the setting on to a test environment, where the end-to-end checks are still being confirmed. The retry guidance and the undo of a rejected push are not deployed yet.

@github-actions github-actions Bot added area/frontend React application in frontend/ area/backend Lambda APIs, lambda/shared, v2-orchestrator area/agents AgentCore runtime: CLI drivers, MCP tools, workspace and Git operations area/infra Terraform, installer and deployment labels Nov 13, 2026
@github-actions github-actions Bot added the size/L < 3000 changed lines (lockfiles excluded) label Nov 13, 2026
github-actions Bot added a commit that referenced this pull request Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Automatically computed PR review signals (deterministic)

Complexity and branch coverage help focus review on functions and their tests; these signals are advisory.

Complexity and test coverage

8 changed JS/TS files analyzed · 8 flagged functions.
Coverage unavailable; test protection cannot be compared.

File / function Complexity Δ Branch coverage Δ Uncovered branches Depending files
🔴 lambda/v2-orchestrator/section.js:1808 — runParallelSection / runLaneBody 56 → 74 +18 — — — 1
🟠 lambda/v2-orchestrator/section.js:2367 — runParallelSection / askLaneRetry — → 32 — — — — 1
🟠 lambda/v2-orchestrator/section.js:278 — awaitEngineGate / <anonymous> — → 27 — — — — 1
🟠 lambda/v2-orchestrator/index.js:3386 — runStage / <anonymous> — → 21 — — — — 0
🟠 lambda/agentcore/git-engine.js:1565 — discardUnpushedCommits — → 20 — — — — 4
🟠 lambda/agentcore/commands/run-stage.js:1902 — runStageAttempt 902 → 908 +6 — — — 1
🟠 lambda/v2-orchestrator/index.js:482 — handler 415 → 416 +1 — — — 0
🟠 lambda/v2-orchestrator/index.js:1157 — handler / executeStage 44 → 45 +1 — — — 0

🟢 Improving · 🟠 Needs attention · 🔴 High risk · 🟣 Highest risk

— = unavailable · n/a = not applicable · * = incomplete tests


Isobar repository map

PR impact map

This fixed repository map locates changed files and their possible reach through static links.
Red marks changed files; green marks static links, not proven runtime impact or a risk grade.

Isobar by r3al1tym.

Signals
  • A: lambda/v2-orchestrator/section.js — modified file · 26 file-level import-link targets.
  • B: lambda/agentcore/commands/run-stage.js — modified file · 21 file-level import-link targets.
  • Total static reach: 61 unchanged production files; 141 unchanged test files.
  • Static reach unavailable for 4 changed files.
  • Revision: 1d97e05; updated 2026-11-18 18:04 UTC.
Legend
  • ○ modified · ◇ added · × deleted. Blue markers: tests; gray markers: documentation.
  • Solid green: import-linked name matches. Dotted green: broader file-level import fallback.

Full report artifacts

github-actions Bot added a commit that referenced this pull request Nov 13, 2026
…ge straight away

When a stage fails in a parallel unit lane the lane ends, and the section
asks retry, skip or abort only once every other lane has settled. With PR per
unit that includes lanes waiting for a person to review and merge, so a
failed unit can wait days for a retry while its dependents stay blocked.

With lane_failure_policy = "ask" (default "halt", unchanged), a failed lane
stage opens a gate owned by that unit with retry and leave-failed:

- The gate parks only its unit. META stays RUNNING, so sibling lanes keep
  running and pushing; the lane releases its concurrency permit and runtime
  session while it waits, and dependents wait on it.
- Retry runs the stage again on the same branch, telling the agent why the
  last attempt failed and passing on any guidance in the answer. After a
  rejected push, the engine first undoes the unpushed commits and keeps
  their changes in the working tree, so the retry can remove what the remote
  rejected. A repeat failure asks again.
- Leave-failed ends the lane as before, so halt-and-ask still applies.
- The policy is read in the existing load-meta checkpoint, so a run in
  progress keeps halting; no steps are added or reordered for it.
@thenom
thenom force-pushed the feat/lane-failure-retry-761 branch from 07e1b85 to 1d97e05 Compare November 18, 2026 17:56
github-actions Bot added a commit that referenced this pull request Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/agents AgentCore runtime: CLI drivers, MCP tools, workspace and Git operations area/backend Lambda APIs, lambda/shared, v2-orchestrator area/frontend React application in frontend/ area/infra Terraform, installer and deployment size/L < 3000 changed lines (lockfiles excluded)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Ask to retry a failed unit lane stage straight away, not after every lane settles

1 participant