Skip to content

persist sessionId on failure so a failed run stays resumable - #47

Closed
matt-wright86 wants to merge 517 commits into
mainfrom
fix/resumable-failed-runs
Closed

matt-wright86 wants to merge 517 commits into
mainfrom
fix/resumable-failed-runs

Conversation

@matt-wright86

Copy link
Copy Markdown
Collaborator

Summary

runTaskAgent persisted the engine sessionId only after the success/failure
branch, so a failed run lost its session — a retry had to rebuild context from
scratch. Persist it before the branch (failed runs stay resumable), and
return telemetry on the failure path too.

Type of change

  • Bug fix (non-breaking)
  • Test coverage

Checklist

  • Raw go/npx were not used
  • make gates — N/A: orchestrator-v2 uses its own vp toolchain
  • vp run -r build clean · vp run -r test green
  • Regression test added (engine fails → sessionId persisted + telemetry returned)
  • Commit message short, lowercase, imperative
  • No force-push to main
  • Self-reviewed (/simplify)

Notes for reviewers

First of the orchestrator-v2 hardening pieces (from the earlier experiment branch)
re-expressed onto the work-item model as focused PRs. Next up: I3 (callback
resilience), Route A (telemetry as a terminal outcome), I1 (capability routing).

🤖 Generated with Claude Code

devzeebo added 30 commits March 18, 2026 14:41
- Add CreateTable method to ProjectionStore interface
- Add RegisteredTables method to ProjectionEngine interface
- Implement auto-create table in Register() method
- Track registered tables for rebuild command
- Add tests for table creation and RegisteredTables()

Closes bf-ca98.4
- Add AccountDirectoryProjector that maintains pats array instead of pat_count
- PATCount derived from len(pats) - no auxiliary keys needed
- Idempotent handling for all events including duplicate PATs
- Add CreateTable method to mockProjectionStore in test helpers
Implements projector for PATCreated (insert) and PATRevoked (delete) events.
Table: projection_pat_by_id
Key: pat_id
Doc: {pat_id, key_hash, account_id}
Implements RealmDirectoryProjector for the projection refactor.
- Handles RealmCreated events (insert with status active)
- Handles RealmSuspended events (read-modify-write to suspended)
- Table: projection_realm_directory
- Key: realm_id
…solution

- Add RealmNameLookupProjector with projection_realm_name_lookup table
- Key: name, Doc: {name, realm_id}
- Handles RealmCreated event with idempotency
- Enables O(1) realm-name-to-ID resolution instead of scanning
Implements username to account_id lookup projector:
- Table: projection_username_lookup
- Key: username
- Document: {username, account_id}
- Handles AccountCreated event with idempotency

Rune: bf-ca98.14
Write DependencyExistenceProjector to maintain a table where each row
represents a single dependency. Row existence is the answer to whether
a dependency exists.

- Table: projection_dependency_existence
- Key: {rune_id}:{target_id}:{relationship}
- Events: DependencyAdded (insert), DependencyRemoved (delete)
- Skips inverse events (handled by forward events)

This replaces the dep:* boolean keys from the old dependency_graph projector.
Implements projector for tracking system-wide status:
- Tracks admin/owner accounts in _admin realm (admin_account_ids)
- Tracks existing realms (realm_ids)
- Single-row-per-realm invariant with key 'status'
- Handles AccountCreated, RoleAssigned, RoleRevoked, RealmCreated
- All handlers are idempotent
Implement projector that maintains cycle-check edges in projection_dependency_cycle_check table.
- Handles DependencyAdded (insert) and DependencyRemoved (delete) events
- Key: {source_id}:{target_id}, Doc: {source_id, target_id}
- Row existence indicates cycle would occur if edge is added
- Skips inverse events, handles unknown events gracefully
devzeebo and others added 26 commits June 19, 2026 12:27
Define ScriptContext, ScriptTaskDefinition, ScriptResult, and ExecutionStats in interfaces-task as the foundation contract for runner and agent packages.
… (GH #33).

Introduces ed25519 signing with deterministic JSON canonicalization, direction-agnostic RPC frames, and peer APIs where the runner dials and both sides communicate via subscribe/send. The orchestrator multiplexes connected runners through onPeerConnect/onPeerDisconnect and send(peerId), with loopback vitest-gwt specs covering crypto, bidirectional RPC, streaming, and tamper rejection.
Merge interfaces-task and interfaces-task-source into @bifrost-ai/interfaces-work.
Replace the split Task/Script concepts with WorkItem instances and WorkItemHandlers.
Handlers receive the work item and an execution context separately; RPC methods and
runner registration APIs use the same vocabulary. Reserve "task" for the LLM task
agent (kind: "task").
unify orchestrator v2 execution model under work item terminology
runTaskAgent persisted the engine sessionId only after the success/failure branch, so a failed run lost its session and a retry had to rebuild context from scratch. Persist it before the branch, and return telemetry on the failure path too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 4, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Improved retry handling for task runs that fail after starting, so session state is saved and can be resumed later.
    • Failure responses now include available execution telemetry, making it easier to diagnose unsuccessful runs.

Walkthrough

The runTaskAgent function now persists engineResult.sessionId to state immediately after engine execution, before checking success or failure, rather than only after success. The failure-path return now includes telemetry from engineResult.stats. A corresponding test was added.

Changes

Session persistence and telemetry on failure

Layer / File(s) Summary
Persist sessionId early and attach telemetry on failure
orchestrator-v2/packages/agent-3-task/src/run-task-agent.ts
ctx.setState now runs before the success/failure branch to persist engineResult.sessionId, and the failure return object now includes telemetry: engineResult.stats ?? undefined.
Test coverage for failure resumability
orchestrator-v2/packages/agent-3-task/src/run-task-agent.spec.ts
New test with failing_engine and valid_state_context asserts session_id_is_persisted and telemetry_is_returned in the outcome.

Possibly related PRs

  • devzeebo/bifrost#20: Related changes to engine execution and sessionId continuation flow that this PR's session persistence logic depends on.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: persisting sessionId on failure so retries remain resumable.
Description check ✅ Passed The description accurately explains the bug fix, telemetry change, and added regression test coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@orchestrator-v2/packages/agent-3-task/src/run-task-agent.ts`:
- Around line 51-58: The unguarded ctx.setState call in runTaskAgent can now
fail the whole function on both success and failure paths, so wrap the
persistence step in try/catch and keep returning a structured WorkItemResult if
state saving rejects. Use the existing runTaskAgent flow and the
engineResult.sessionId / ctx.setState block to locate the change, and make sure
any persistence error is handled without throwing past the function boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: b73d428d-7978-42eb-89b0-4759a9fc82f0

📥 Commits

Reviewing files that changed from the base of the PR and between a411816 and 3f30ef5.

📒 Files selected for processing (2)
  • orchestrator-v2/packages/agent-3-task/src/run-task-agent.spec.ts
  • orchestrator-v2/packages/agent-3-task/src/run-task-agent.ts

Comment on lines +51 to 58
// Persist the sessionId before branching on the outcome, so a failed run stays
// resumable — a retry can --resume instead of rebuilding context from scratch.
if (engineResult.sessionId !== undefined) {
await ctx.setState({
...workItem.state,
sessionId: engineResult.sessionId,
});
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Unhandled ctx.setState failure now impacts both success and failure paths.

This persistence call isn't wrapped in try/catch. If ctx.setState rejects, the function throws instead of returning a structured WorkItemResult, defeating the purpose of making failed runs resumable. Since this block now runs unconditionally (previously only on the success path), the blast radius of an unhandled rejection here has doubled.

🛡️ Suggested guard
   if (engineResult.sessionId !== undefined) {
-    await ctx.setState({
-      ...workItem.state,
-      sessionId: engineResult.sessionId,
-    });
+    try {
+      await ctx.setState({
+        ...workItem.state,
+        sessionId: engineResult.sessionId,
+      });
+    } catch {
+      // Swallow persistence errors so a failing setState doesn't crash the run;
+      // resumability is best-effort and shouldn't mask the underlying engine result.
+    }
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Persist the sessionId before branching on the outcome, so a failed run stays
// resumable — a retry can --resume instead of rebuilding context from scratch.
if (engineResult.sessionId !== undefined) {
await ctx.setState({
...workItem.state,
sessionId: engineResult.sessionId,
});
}
// Persist the sessionId before branching on the outcome, so a failed run stays
// resumable — a retry can --resume instead of rebuilding context from scratch.
if (engineResult.sessionId !== undefined) {
try {
await ctx.setState({
...workItem.state,
sessionId: engineResult.sessionId,
});
} catch {
// Swallow persistence errors so a failing setState doesn't crash the run;
// resumability is best-effort and shouldn't mask the underlying engine result.
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@orchestrator-v2/packages/agent-3-task/src/run-task-agent.ts` around lines 51
- 58, The unguarded ctx.setState call in runTaskAgent can now fail the whole
function on both success and failure paths, so wrap the persistence step in
try/catch and keep returning a structured WorkItemResult if state saving
rejects. Use the existing runTaskAgent flow and the engineResult.sessionId /
ctx.setState block to locate the change, and make sure any persistence error is
handled without throwing past the function boundary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants