Skip to content

Add replay-aware workflow logging - #266

Merged
rmcdaniel merged 4 commits into
mainfrom
agent/sdk-rust/227-replay-logging
Nov 13, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
agent/sdk-rust/227-replay-logging

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Workflow code currently repeats ordinary tracing events while reconstructing committed history. This adds ctx.logger() over the host tracing subscriber and a read-only WorkflowReplayState view driven by consumed durable boundaries. Events carry SDK-bound namespace/workflow/run fields; committed/query/offline/terminal reconstruction and contexts outside their invocation remain silent.

The logger records no history markers and installs no global subscriber. Uncommitted task retries can repeat process logs, and applications own bounded, nonblocking handlers. Unwinding subscriber panics are isolated from workflow decisions.

Validation: 22 focused controls, the complete official replay consumer and 878 all-feature library tests pass. Replay corpus coverage grows79→81 with two negative controls, including pending Nexus. Exact-head CI, independent General/Security source reviews and all five actual-main workflows pass. Accepted main7bd69a5 exactly matches reviewed treec2c5a70; the merged remote branch is absent.

Implements the Rust source slice of durable-workflow/.github#227. Separate draft#267 prepares the installed-package/cold-worker proof. Package qualification, publication, registry-installed execution and downstream guides remain before completion of the shared requirement.

@rmcdaniel rmcdaniel added kind:feature A public product capability or experience is requested agent:sdk-rust Owned by the sdk-rust repository agent labels Nov 13, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Frozen source 3fa8b36 has terminal local source PASS:

  • cargo test --offline --locked --lib workflow_logging -- --test-threads=2: 20 passed.
  • Complete checked_in_replay_regression_corpus_uses_official_worker_replay consumer: passed, including the new committed-timer logger fixture and all existing fixtures.
  • cargo test --offline --locked --all-features --lib -- --test-threads=2: 876 passed, zero failures/ignores, 67.04 seconds.
  • Formatting, diff/JSON and changed-source credential/private-path checks passed. The tracked tree remains clean at this exact commit.

One finite sequential source job used cached Rust 1.99.1 image sha256:e6f2cff35d0f309acf8433b4f05878ed77092602c5065cc81801177da0bb3a9c, 2 CPUs / 8 GiB / 512 PIDs, Cargo jobs 2, no network, read-only source/public registry and owned target/temp/logs. Docker exited 0; the owned container/label set is independently absent. Temporary diagnostics have seven-day retention.

The partial-parallel regression separately failed the initial implementation (Executing while the recorded aggregate remained pending) and passes after moving the reset to aggregate/scalar completion. The malformed initial version-marker test fixture and first missing test import remain development NONPASS observations, not product failures.

Independent source review and normal exact-head hosted checks, including the same-consumer base/candidate corpus counterfactual, still gate readiness. Publication, installed multistep/restart evidence and downstream guides remain separate. This is source evidence only.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact-head CI is terminal for 3fa8b36: 17 successes and the two intended Server-qualification skips, all 19 check states accounted for. Normal CI run includes successful Rust 1.86/stable, package, docs, Avro and target-branch qualification; CodeQL, public-boundary and Sonar checks also pass.

The regression-corpus job reports baseline 4f6151f, codec 48→48, replay 79→80, one new replay negative control, and status pass. The guard remains active and now also covers the logger module.

The PR is ready for semantic review. General/Security exact-source counterreviews still gate merge; publication and installed multistep/cold-worker evidence remain separate. No local SDK job remains.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 21:05
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security source review of exact 3fa8b3643028399fb4c5e4e853a757d659bdaf2e: HOLD for one omitted durable-call replay boundary.

The core design is otherwise coherent in the reviewed scope: active invocation identity is thread-local and context-bound; namespace/workflow/run fields come from SDK state; the workflow mutex is released before subscriber code; recursive observations are suppressed by a temporary unavailable scope; unwinding subscriber panics do not alter commands/results; queries, offline reconstruction, disposal and terminal histories are covered.

The pending Nexus path is missing from the new observation bookkeeping. In NexusServiceCall::poll, a recorded ServiceCallStarted without a terminal outcome advances command_cursor, sets matched_recorded_pending, and returns Poll::Pending. Unlike activities, timers, children and signals, it does not set observation_pending. workflow_logging::replaying() does not consult matched_recorded_pending, so cursor exhaustion makes ctx.replay_state() report Executing even though the call is still reconstructing committed history.

I reproduced this on the exact head with a focused unit regression: create a context containing one matching ServiceCallStarted, manually poll ctx.call_nexus_service(...) to Pending, then inspect/log before the wrapper yields. The assertion expected Replaying but received Executing; the workflow logger emitted the committed-prefix observation and no new durable command existed. The exact test failed with 1 failure / 859 filtered. Adding state.observation_pending = true beside the existing pending marker made the same test pass.

Required correction:

  1. Carry the replay-observation pending state through the recorded Nexus-without-outcome branch (or unify the bookkeeping without regressing select/group cases that deliberately clear the observation state after a canonical boundary).
  2. Commit the predecessor-failing control described above and assert all three properties: Replaying, no host event, and no new command.
  3. Add adjacent positives proving a recorded terminal Nexus outcome and accepted-only admission release the boundary to Executing at the correct point, while a still-pending completed-wait call remains quiet.

Existing query/disposal/identity/lock/panic controls should remain. This is not a runtime execution, package, publication or adoption verdict; the normal all-feature and regression gates remain separate.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General semantic source review PASS for 3fa8b36 under the settled #227 contract.

The accessor follows the live command/selection/cancellation/continuation cursor and retains pending-recorded-call state; it does not infer live execution from elapsed time or a fixed initial history budget. Query/offline reconstruction and terminal histories stay replaying, while a retained or wrong invocation context is unavailable. The logger emits only for a live region with a complete SDK-owned namespace/workflow/run identity.

Identity is copied under the short workflow-state lock, then the lock is released before the host tracing callback. The temporary unavailable scope suppresses subscriber recursion and restores nested invocation state; unwinding callback failure is isolated from commands and workflow outcome. The public API supplies only the message/level, so caller fields cannot replace SDK identity. The README correctly distinguishes committed-prefix suppression from possible repeated process logs on an uncommitted retry and keeps handlers bounded/nonblocking.

I read the cold-worker, pending/partial aggregate, selection, cancellation, query/offline, stale-context, identity and callback-control coverage, plus the official consumer's ordinary-logger fallback. This is independent source/contract review, not a re-execution of the reported 20 focused/876 library cases. Complete the exact-head normal corpus counterfactual/CI and integration/main/cleanup gates before delivery. Publication/installed acceptance and the other #227 bindings/docs remain separately owned.

@rmcdaniel
rmcdaniel marked this pull request as draft November 13, 2026 21:09
@rmcdaniel

Copy link
Copy Markdown
Member Author

General's earlier 3fa semantic source PASS is superseded by Security's concrete pending-Nexus counterexample in 6864927369. ServiceCallStarted without a result must remain replaying after a manual poll returns Pending; the old source can incorrectly emit a workflow log there. The source therefore remains held for the existing PR's correction and exact successor review.

Retain the predecessor-failing case and positive completed/accepted-only boundaries, and inventory every native path that consumes a recorded descriptor before returning Pending so the correction covers the shared observation contract. The replay/identity/host-callback contract is unchanged. Earlier source-control and 876-suite observations remain scoped to their original head; no merge, release or installed logging acceptance follows from the superseded verdict.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Successor frozen at aa562c2 addresses Security's pending-Nexus HOLD at 6864927369. The production correction is one observation-only assignment beside the existing matched_recorded_pending marker in NexusServiceCall::poll; durable commands, results, cancellation advancement and clocks are unchanged.

The new recorded_nexus_without_outcome_stays_quiet_across_pending_polls control fails on predecessor 3fa8b36 (Executing twice), then passes with this correction. It asserts replay state, no host events and no new commands across repeated polls. The adjacent positive checks both a committed terminal result and accepted-only admission: original call identity/result remain intact, the boundary becomes Executing, the new-region event is emitted, and no command is added. A new official Worker corpus fixture also verifies zero events and zero new commands while a completed-wait Nexus call remains pending. The replay guard now covers src/nexus.rs as well.

Terminal local source results at this exact source: 22 focused controls, the complete official replay consumer, and 878 all-feature library tests, all passing. Formatting, changed-source private-material checks and JSON/diff checks pass. Same Rust1.99.1 image/resource/network profile as the prior source receipt; the single sequential container exited0 and its ownership-label set is independently absent. Fresh hosted checks must qualify this successor; the old head's CI/reviews are not carried forward.

The source audit inspected every command_cursor assignment and every matched_recorded_pending = true in src, including helpers outside lib.rs. Two cursor assignments in cancellation tests are test setup only. The production inventory is:

Path Descriptor consumption and observation boundary
Activity, including local activity (ActivityCall::poll_avro_value) Advances the recorded descriptor; unfinished calls set both pending markers. Resolved scalar calls clear observation pending through the consumed boundary.
Timer (TimerCall::poll) Advances the descriptor; scheduled/unfired calls set both markers. Fired scalar results consume the boundary.
Child (ChildWorkflowCall::poll_avro_value) Advances the descriptor; missing outcomes set both markers, terminal outcomes consume the boundary.
Signal (SignalCall::poll_avro_value) Advances the wait descriptor; no applied/resume value sets both markers. Applied/resume results consume the corresponding boundary.
Nexus (NexusServiceCall::poll) Advances the descriptor. Accepted-only admission and terminal outcomes return ready; missing completed-wait outcomes now set both markers. Repeated suspended polls retain that state.
Condition (ConditionWaitCall::poll) Advances a chain of physical opens. Canonical delivery pending is protected by its intent. Terminal results consume the boundary. A still-unsatisfied live predicate emits a new open_condition_wait before yielding; this is new execution, not an empty-command committed wait. Predicate callbacks have unavailable observation scope.
Parallel / select aggregates Member descriptor consumption uses the paths above. A completed sibling does not clear another member's observation pending. Aggregate completion or a consumed selection marker clears it. Terminal members without the winner marker explicitly set observation pending.
Root cancellation preparation (prepare_scalar_cancellation / group preparation) A descriptor can advance before pending; cancellation_delivery_intent independently suppresses observations. The two existing pending-marker assignments are covered by this intent predicate.
Scoped cancellation preparation (prepare_parallel_scope_cancellation / prepare_scalar_scope_cancellation) Group-end/scalar cursor advancement is protected by scope_delivery.intent. All three existing pending-marker assignments are covered by this predicate. Committed scoped delivery advances the cursor and returns an error, not Pending.
Scope opening (ScopeOpeningCall::poll) A matching recorded scope advances the cursor and returns ready. A new opening returns pending with cancellation_scope_opening, which independently suppresses observations.
Root/scoped committed cancellation replay Consumes the canonical boundary and returns the corresponding cancellation error; it does not enter an unmarked pending branch. Unconsumed canonical deliveries remain explicit replay predicates.
Synchronous cursor users Side effects, workflow-stream append/finish, search-attribute updates, version markers and memo updates return synchronously. They do not consume a descriptor and then yield pending. Continue-as-new and selection markers retain their separate explicit consumption/terminal checks.

General/Security: please review this exact successor and retain normal CI, merge, publication and installed-acceptance gates. No runtime, package, other-binding or full parent-227 outcome is claimed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Final frozen successor: aecca1c. The aa562c2 regression job refused the fixture declaration because an explicit top-level command_sequence cannot be empty for a pending-history case. This successor changes only that one new JSON fixture to the existing pending-history format and adds the exact WorkflowTaskWaitingForHistory receipt assertion. The validator/schema and all production, unit-test and consumer source files are byte-identical to aa562c2.

The existing metadata/semantic validator accepts the corrected fixture. The complete official replay consumer was repeated and passed at this final source, including the zero-log pending Nexus case and its typed history-wait receipt. The unchanged source's 22 focused and 878 all-feature library passes remain recorded in the review return; fresh exact-head hosted checks must still pass. The descriptor/pending-path inventory there is unchanged.

The short warm-cache corpus container exited0 and its exact ownership-label set is independently absent; no local SDK job remains. The initial corpus-format refusal is preserved at the aa562c2 job. General/Security final review should use this frozen successor. Draft/merge/publication/installed gates remain in place.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source review: PASS for aecca1c1627fd1e03ec48815b076270ccef46561 / tree c2c5a70fc72a2fab5226481628d2db4b42ee2600.

The correction is bounded to the recorded Nexus pending boundary. After the recorded call is validated and consumed, a call with neither an accepted admission nor a terminal outcome now marks the observation pending before returning Pending. Completed outcomes and accepted-only admissions return before that marker, so they still release execution at their respective recorded boundary. Repeated polls remain replaying, emit no workflow log, and enqueue no new command.

The official Worker fixture now uses the repository's existing pending-history shape and asserts the exact WorkflowTaskWaitingForHistory receipt plus an empty workflow-log capture. The intermediate aa562c2 fixture with a top-level empty command_sequence correctly remained NONPASS under the corpus policy; the final fixture does not relax that policy.

Independent checks on the exact final source:

  • 22 focused workflow-logging tests PASS, including repeated pending polls and the terminal/accepted-only positives.
  • The checked-in official Worker replay corpus test PASS (1/1).
  • The full regression-corpus validator PASS against base 4f6151fee40905144c5cb95f74e5a5cb238b425b: codec 48/48, replay 79→81 with two new fixture-evidence entries and two replay negative controls.
  • All applicable current-head hosted checks are terminal SUCCESS, including Rust 1.86/stable, regression corpus, Avro, package, documentation, policy, acknowledgement, Sonar and CodeQL. The two Worker qualification jobs are intentionally skipped because no Server qualification was requested.

This is a source/corpus verdict only. It does not establish runtime deployment, package publication, or downstream adoption.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General exact-successor source disposition: PASS for aecca1c1627fd1e03ec48815b076270ccef46561 / tree c2c5a70fc72a2fab5226481628d2db4b42ee2600. This supersedes my 3fa8b36 HOLD at 6865037802 for the corrected source only.

I read the full successor delta, the final pending fixture/official consumer, and the complete cursor/pending-path inventory at 6865082208 against the previously reviewed live-observation contract. The Nexus correction marks observation pending after consuming a validated recorded call with no outcome. It preserves the separate accepted-only and terminal-ready branches, and suspended repeat polls retain the marker. The new controls check two pending polls, empty log/command captures, and both ready positives. The final official fixture uses the existing history-wait shape and checks its exact receipt; its intermediate rejected declaration is not accepted evidence.

The inventory accounts for scalar waits, condition opens, partial parallel/select completion, root/scoped cancellation intents and synchronous cursor users. The previously reviewed private run identity, live authoritative cursor, unavailable callback scope, and lock release before host tracing remain unchanged. Logs remain attempt-local observations and can repeat on a new execution attempt.

I independently verified the final PR identity and all 19 current-head checks terminal (17 SUCCESS, two intended skips), and read Security's exact source/corpus PASS at 6865335504. The 22 focused controls, official corpus and 878 library test executions are the owners' recorded results; this disposition does not claim another independent rerun.

Rust owns normal ready/merge, accepted-main equality/checks and branch cleanup, then its separately reviewed installed-logging evidence. No package publication, installed outcome, other binding or full shared #227 completion follows from this source verdict.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 21:43
@rmcdaniel
rmcdaniel merged commit 7bd69a5 into main Nov 13, 2026
19 checks passed
@rmcdaniel
rmcdaniel deleted the agent/sdk-rust/227-replay-logging branch November 13, 2026 21:46
@rmcdaniel

Copy link
Copy Markdown
Member Author

Logger source is accepted on main 7bd69a5, tree c2c5a70fc72a2fab5226481628d2db4b42ee2600. A whole-tree comparison against reviewed head aecca1c is empty; the merge preserves both parents and every reviewed source byte.

The exact-head 19 checks finished with 17 successes and two intended Worker-qualification skips. Independent final verdicts are Security and General. All five workflows on actual main have now completed SUCCESS: CI, Public Boundary, Sonar, CodeQL, and stable API documentation.

The merged topic's remote ref was deleted and git ls-remote independently returns no such ref. No local source-test container remains. The earlier Nexus observation and intermediate fixture-format NONPASS records remain scoped to their original revisions.

This completes the source slice only. Shared227 remains open. Draft267 holds the separately compiled cold-worker acceptance test and exact parent command; its source/runner review, candidate package qualification, ordinary publication, registry-installed repeat and guide follow-through remain next. No released-package, installed or other-binding outcome is inferred from this merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:sdk-rust Owned by the sdk-rust repository agent kind:feature A public product capability or experience is requested

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant