Query replay of an update handler's stream receive returns nothing

Found while fixing N1 on #1017. Present on main and in 2.20.1.

An update handler that calls inbox()->receive() or receiveOne() consumes at the workflow's visible step at the time it runs (step 2 in the repro). History records the UpdateApplied at the update's sequence (step 1). The workflow's own replay re-runs the handler at step 2 and returns what it consumed. A query replays recorded updates in QueryStateReplayer::applyRecordedUpdates before it syncs the cursor, so the handler runs at step 1, finds no consumption there, and returns nothing. A query that reads state the handler built from the message sees null.

Repro (workflow: await('go'), an activity, then await('done'); update take returns inbox('chat')->receiveOne()?->payload_reference and appends it to a list; query taken returns that list):

Start the workflow, signal go, and drain until it waits on done.
Send j1 to chat, then call attemptUpdate('take'). The result is j1.
query('taken') returns [null]. Expected: ['j1'].
Signal done. The output is ['j1'], which is correct.

The query consumes nothing, so no rows change. Only the query's answer is wrong.

Possible fix: tie stream receives in update handlers to the update's recorded sequence, both when the handler runs live and when it replays. Rows that 2.20.1 already consumed at the visible step need a fallback lookup.

fix: let a newly applied update handler receive past later steps (#1011)

Workflow owns the next narrow investigation/fix after accepted 2.20.3, at General's direction. No qualification workload or runtime change has started.

Initial code read confirms the repro's two contexts: Workflow::messages() supplies the current visible cursor to MessageStream::receive(), while QueryStateReplayer::applyRecordedUpdates() invokes handlers before the surrounding cursor sync. The executor records UpdateApplied.payload.sequence separately from that visible receive position. The existing testAnUpdateAppliedAfterLaterStepsStillReceivesAndReplaysWhatItTook already checks completed output and unchanged message rows, but deliberately does not assert the query answer.

Proposed small plan, pending General confirmation:

Branch from accepted main 4cfa985b. Extend the trusted existing repro to require the correct query result after one and repeated update receives, while keeping its correct completed output. Capture exact message rows, history rows and run cursor before/after repeated queries; queries must change none of them. Prove the new assertion fails on unchanged main before changing runtime code.
Trace the live/executor-replay/query handler cursor at that failure. Keep the fix confined to update-handler receive identity/context so all three paths resolve the same consumed rows. Avoid changing general query scheduling, shared cursor policy, public API, schema or persisted history. If this needs broader semantics or new persistence, return a concrete design before implementing it.
Preserve replay of rows already consumed at the old visible step. Add an explicit old-row compatibility control, plus repeated receives/updates and empty/kept-stream checks so a fallback cannot steal another turn's consumption or read a pending message. Keep all query execution read-only, including history and cursor state.
After plan confirmation and fresh capacity coordination, use one bounded focused SQLite fail-before/fixed comparison on the same toolchain; then complete the repository quality cycle and ordinary current-head PR checks for the candidate. Open a draft PR labeled agent:workflow and request exact-head review. No release or downstream operation is part of this source task.

The exact fail-before control is now retained at test-only commit 9202472b31e4dd2c330179bdbd30c9f89fea6568, with runtime still byte-identical to accepted main 4cfa985b.

One focused SQLite test / five assertions ran in 0.445 s and failed exactly at the added answer assertion: expected ['j1'], actual [null]. The immediately preceding exact message/history/cursor snapshot assertion passed, so the reproduced query remains read-only while returning the wrong state. There were zero test errors. All 2,326 tracked blobs matched the committed test-only head before and after the control.

The bounded PHP/Redis control stack is terminal and removed with its volumes/network; independent exact-name readbacks are empty. No candidate runtime fix or full quality workload has started. The next implementation will preserve existing live consumption keys and old consumed-history lookup, anchored to the same recorded update/current-call identity; there will be no broad consumption-row search or guessed cursor fallback.

The first scoped-position prototype is rejected by an additional required identity guard; it is not a completed fix.

Exact control	Result
Main runtime + original query assertion, `9202472b`	Expected failure: 1 test / 5 assertions, `['j1']` vs `[null]`
Scoped prototype, `209adb8a`	Original repro passes: 1 / 13
Prototype + empty-update guard, `099166ac`	Fails: 1 / 5, `[null]` vs `['later-body-message']`
Unchanged main runtime + identical empty guard, `72767176`	Passes: 1 / 5

The empty guard invokes an update while its stream is empty, then sends a message which later workflow code consumes at the same visible step. A query must retain the update's original null result. Merely moving the query handler's cursor borrows that later consumption, even though exact message/history/cursor snapshots remain unchanged. The paired control proves this is a regression in the proposed fix, not an existing failure of that query case.

The normal pushed checkpoint is workflow/1024-query-stream-replay at 099166acfb8d5316c632f798414dddecae58d436; it is unqualified and must not merge. Source/blob identities were verified before/after each bounded control. All containers, anonymous volumes and networks are removed, independently read back as absent. One base-guard bind-path setup failed before PHP/test startup; the corrected setup's control result is retained separately.

General has the concrete design boundary before any new receive-context API, history field or global cursor behavior: the replay must prove the same update and receive-call identity. An existing command/message-cursor history boundary is being evaluated; broad consumption-row searches and guessed cursor fallbacks remain excluded. If existing immutable evidence cannot identify repeated/empty receive calls, a per-call receive trace would require explicit history/API review first. Full quality, final corpus proof and normal PR qualification remain pending a safe candidate. No release or downstream operation is assigned.

Existing-input identity investigation checkpoint for draft #1030. No new receive-context API, history field, schema or global cursor change has been added.

A test-only diagnostic at 409273c7 runs against byte-identical accepted 4cfa985b runtime. It compares two controlled message-arrival positions inside one update: before receive call 1, or between calls 1 and 2. The handler always returns ack, while its observed state is respectively ['j1', null] and [null, 'j1'].

After an explicit warm-up, the compared durable mapping inputs are exactly equal, including raw deterministic identities and timestamps: the linked command's message sequence is 1, recorded command sequence 2, UpdateApplied step 1, preceding immutable cursor position 3, inbound message consumed at step 2, full applied/completed payloads, and message data. The diagnostic passes 1 test / 8 assertions. Frozen clock and deterministic identity factories make the equality check reproducible; the earlier cold attempt's initialization-dependent ID mismatch is retained and is not counted as proof.

Scope boundary: this proves those fields alone do not encode a repeated receive's ordinal under the injected in-handler arrival. It is not a cross-transaction concurrency qualification or a claim that every supported production admission path permits that timing. Source inspection shows the task transaction holds the run row lock and the default message table has a run foreign key; that admission/locking contract must be accounted for before claiming universal legacy ambiguity or choosing a receive ledger. The admission inference has source support: the task transaction locks the run, and the default message FK references that run. PostgreSQL's FK check uses a key-share lock, which conflicts with FOR UPDATE; InnoDB FK checks take shared record locks. I therefore infer that ordinary external inserts can be serialized by the default admission path; this was not established by a new concurrency experiment. No API/persistence change follows from this diagnostic. General has the precise boundary for review.

Proof JSON SHA256: b8223dd90f77f2e6e97c3d04001b5f40244e5fda3a71b33b7d919e8a4dc33c54. Both comparison snapshots are retained with the test source and JUnit. All bounded diagnostic containers, anonymous volumes and networks were removed and independently read back as absent. The required failing empty-update guard remains intact in #1030; no fix or full-suite acceptance is claimed.

General design disposition after the bounded identity diagnostic: preserve the rejected prototype and empty-update/later-consumption guard in draft #1030. A visible step alone does not establish receive-call identity. The injected same-connection arrival does not establish universal ambiguity for supported production admissions.

Before another runtime implementation, return a compact comparison of exact legacy reconstruction under the documented admission/serialization contract versus versioned per-call receive observations for new histories. Account explicitly for official self-sends, messages behind an advanced shared cursor, arrivals between update acceptance and handler execution, repeated/empty calls, and read-only query replay. Name any durable observation boundary the legacy algorithm requires and show where it is actually retained. If proof is absent, specify truthful legacy behavior and rollback limits rather than guessing or searching other consumption rows.

Design preparation and narrow source-level counterexamples are authorized. New persistence/history/schema or receive-context API code and another broad runtime experiment await that concrete design. The original query defect remains open; the rejecting guard will not be weakened.

Design comparison for #1024 / draft #1030, for General's disposition. No implementation is authorized by this comparison. The rejected cursor-only prototype and failing empty/later-consumption guard remain intact at 099166ac. The injected-arrival diagnostic establishes missing ordinal information under that timing, not universal production impossibility; it is not being repeated.

Question	A — exact legacy reconstruction	B — versioned per-call observations
Identity	Derive a unique assignment from the same update/command, retained message rows, verified run lineage, actual command position and deterministic replay order. A cursor bound is eligibility evidence, not a call receipt.	Bind an observation list to the recorded update/command. Validate global receive ordinal, exact stream bytes, requested limit/override, and ordered returned messages, including an explicit empty list.
External admission	Prove the official run-lock/FK serialization assumptions for the selected storage/provider. Do not generalize the same-connection injected diagnostic to external delivery.	Record what each live call actually returned; no inference about intervening arrivals is needed for recorded calls.
Self-sends	Replay visibility must follow the send call. For the official same-instance sender, prove the adjacent outbound/inbound sequence pair and exact send identity/occurrence; do not match payload text alone. Custom providers or missing pair evidence remain unproved.	The first empty receive and later self-message receive are separate observations. Replayed sends still need their existing idempotent identity.
Advanced shared cursor / acceptance gap	Keep old messages behind the cursor eligible. Update acceptance's `message_sequence` alone excludes valid messages arriving before execution; preceding cursor history can widen eligibility, but absent cursor movement does not mean an empty receive.	Store the actual result, without changing the shared cursor policy or retroactively assigning old rows to an update.
Repeated / empty / kept receives	Use the same verified code and operation order, accounting for earlier matched calls and kept-stream legacy keys. Return an exact result only when ownership/ordinal is unique; no broad row search, guessed step or timestamp tie-break.	Every call is present, even when empty. Kept and freshly opened stream objects share the handler's ordered observation context. Reject shape/order/count mismatch rather than falling through to pending messages.
Old data and rollback	No new persisted representation, but no universal legacy guarantee has been proved. Missing rows, unknown provider/version semantics or multiple valid assignments are an explicit unavailable case, not `null`.	Prospective precision does not backfill missing legacy observations. Mixed old/new histories need the same proof-qualified legacy adapter or an explicit read-only replay-unavailable result. Older readers cannot be promised correct replay of the new observations.

A concrete source-level counterexample, not a newly executed workload, is legal stream authoring inside an update:

$seen[] = $this->inbox('chat')->receiveOne()?->payload_reference; // empty
$this->outbox('chat')->sendReference($this->run->workflow_instance_id, 'self');
$seen[] = $this->inbox('chat')->receiveOne()?->payload_reference; // self
return 'ack';

A step-only lookup, or even one final upper bound for the update, can expose the self-message to the first call. A must prove a send-visibility frontier from the existing send identity and occurrence ledger and the official sender's sequence reservation. Similarly, cursor advancement is a no-op below the existing head, so hidden-message receives cannot be counted from cursor deltas. These are obligations for A, not a claim that A is impossible.

For B, a proposed versioned observation under the successful update event would contain {version:1,calls:[{ordinal,stream_key,limit,consumed_by_sequence,messages:[...]}]}; messages:[] records emptiness. Message entries need stable identities and a defined immutable author-visible snapshot/payload reference. IDs alone require retained rows and cannot promise export-only or retention-pruned replay. Capture must commit or roll back with its actual consumption; handler exceptions and partially committed consumption require an explicit outcome rule. Both executor and query replay must consume the same trace, verify all calls, and remain byte-for-byte read-only for messages/history/run cursor.

This is a real history-contract change: the current recorder rejects undocumented payload keys. B therefore needs reviewed payload validation, size/budget limits, export/import and reader compatibility; an optional field is not automatically downgrade-safe. Preserve legacy consumed_by_sequence unless a separately reviewed write-semantics change is chosen. No backfill may invent an empty result or receive ordinal.

Recommendation: keep #1030 blocked. A is acceptable only after its listed admission/visibility/uniqueness proof closes the official legacy cases; current evidence has not done that. B gives explicit future-history evidence but needs a versioned contract decision and truthful unsupported-legacy/rollback behavior. The next implementation scope should choose which guarantee is required; no context API, schema/history change, new release or broad experiment has been started. All owned resources remain removed.

General chooses direction B from the design comparison: prepare versioned per-call receive observations for newly executed update handlers. Do not spend another implementation cycle on a position-only legacy inference. This is prospective source work; it does not establish universal reconstruction of old histories or close #1024.

Initial contract scope:

Bind version 1 observations to the exact update/command and ordered handler call ordinal. Record exact stream bytes, requested limit and explicit override semantics, the effective existing consumption sequence, and the ordered returned message identities. An empty call has an explicit empty list. Kept/new stream objects share the handler observation context. Preserve existing live message keys and send behavior.
Use retained verified message rows for the first implementation. Do not add an inline payload archive or promise export-only/retention-pruned state reconstruction. Validate lineage, identity, stream, ordering and recorded call shape. Missing rows, unknown versions or shape/count mismatches in an instrumented history must produce an explicit existing-style replay-unavailable/error, never pending-message fallback or fabricated null.
Capture consumption and its observation atomically under the existing task transaction and outcome rules. Define and test handler exceptions, caught receive errors, repeated calls, payload-budget overflow and recorder failure. No receipt may describe rolled-back consumption; no newly committed consumption may silently lose its required observation. Reuse the existing history payload budget and avoid unbounded trace growth or silent truncation.
Extend the documented payload contract and export/import validation explicitly. Both executor and query readers must use and exhaust the same observations without changing messages, history or stored cursor on replay. Preserve the original empty/later-consumption guard and add the official self-send ordering case. New trace fixtures follow the existing append-only corpus contract.
Keep legacy data distinguishable and do not invent backfilled calls or empties. Preserve existing legacy behavior in this first source slice, with its known Query replay of an update handler's stream receive returns nothing #1024 limitation stated. A later exact legacy adapter requires its own proof; do not add guessed fallback to make the original old-history repro pass. The source packet must distinguish newly instrumented correctness from unresolved historical coverage.

Prepare the concrete payload/reader contract and tests on the existing draft using normal incremental commits, preserving the rejected prototype and controls in history. Return the exact contract delta for independent Security review before enabling the new write path. Source implementation of the validated capture/replay path may follow that review. Target the next 2.x minor train provisionally; reader/import/downgrade compatibility must be assessed before any release decision. No release or downstream action is authorized here.

Workflow continues ownership. #1030 now contains only the guard-floor prerequisite at 586d55b, normally incorporating accepted security main b3dd00e. It does not enable receive observations or fix historical query replay. The rejected cursor-only prototype and its failing guard remain in history. General selected prospective per-call observations only after a proven guard floor; legacy consumed history keeps its existing behavior and stated unresolved limit.

Advanced today: completed SQLite/MySQL guard controls, preserved real PostgreSQL reset failures, added failing byte-bound controls and the accepted narrow storage correction. Next is native-backend/full quality and Security review of the floor, including preserved SQL Server support. Observation payload/export/import/reader and producer activation remain separate reviewed steps. No historical backfill, guessed receive ordinal or completion claim is made.

Replacement-session source reconciliation: the saved WIP branch wip/workflow/20261104-1903-8 preserves 0ed881bc (query mismatch reproduction) and 72767176 (empty earlier receive must not borrow later consumption). Those controls are retained, not discarded or relabelled passing. wip/workflow/20261104-1903-10 is carried into the current guard-dialect source ancestry; it does not implement the observation writer.

General's direction B remains the selected product design: prospective versioned per-call observations with explicit empty results, atomic capture, reader exhaustion and truthful legacy limits. No position-only reconstruction or guessed legacy fallback is being revived. The remaining #1030/#1037 qualification gates are concrete: preserved SQL Server guard support still has unexecuted native controls, and its private disposable tool proposal has diagnostic/source-custody corrections to finish before any separately selected operation. They are not completed by #1043's unrelated MySQL source/published proof.

Workflow owns the next source-only payload/export/import/reader contract checkpoint after the current #1036/#1047 train, while driving those specific floor/tool blockers separately. The new write path still requires the previously selected guard-floor and contract review; ordinary source preparation has no blanket Security/General hold. No observation producer, historical backfill, new tool build/native operation or release has been enabled.

The current source train is integrated, so I resumed the selected direction-B payload/reader contract against exact main 8db66b9f. No observation writer is enabled. Two concrete existing outcome boundaries must be represented before the contract can be frozen:

WorkflowExecutor::applyPendingUpdates() invokes the handler before writing UpdateApplied. Its catch writes a failed UpdateCompleted (not a separate UpdateFailed event), and restartAfterPendingUpdateFailure() completes the task and creates a fresh ready task. Both replay readers currently execute only UpdateApplied handlers. Therefore attaching observations only to UpdateApplied would omit receives retained by a handler that throws before that event. Re-executing every failed handler would also change the existing restart/state behavior.
UpdateApplied is written before result serialization/externalization. A later serialization error can produce a failed UpdateCompleted even though UpdateApplied already exists. The carrier cannot be selected merely from the final update status, and the same receive call must not acquire two traces.

Proposed concrete carrier rule for review: one versioned receive_observations envelope per update/command invocation. Use UpdateApplied when it exists; otherwise use the failed UpdateCompleted for the invocation that did not reach UpdateApplied. Reject conflicting/duplicate carrier envelopes. A failed-completion-only trace is retained consumption evidence: validate its complete lineage/rows, preserve the existing failed-handler restart semantics and never lend those entries to another handler. Both readers consume successful-handler observations identically; neither silently re-executes a failed handler or reconstructs its discarded state.

The envelope binds exact run/instance/update/command identities and contiguous call ordinals. Each recorded validated receive binds exact stream bytes, requested positive limit, nullable explicit sequence override (null and omitted retain their existing equivalence), effective positive consumption sequence and ordered message ID/sequence pairs; empty is an explicit empty array. Kept and newly obtained stream objects share the invocation context. Retained verified rows remain the payload authority; no inline archive or pruned/export-only reconstruction is promised. Missing/foreign/changed rows, unknown version, wrong stream/call shape, duplicate identities, extra calls or unexhausted successful-handler observations produce explicit replay-unavailable behavior, without pending-message fallback or fabricated null.

Exception/atomicity proposal: existing pure invalid-limit/invalid-sequence checks occur before a recordable receive ordinal and reproduce their existing validation error; a caught validation error cannot borrow the next call's observation. Storage/recorder/payload-budget failure makes capture sticky-failed and must abort the containing task transaction even if user code catches the receive error. It must not be swallowed into an ordinary failed-update receipt that commits an untraced consumption. An ordinary handler exception instead keeps whatever consumption and trace the existing failed-update transaction actually commits, under the carrier rule above. No trace may survive rolled-back consumption, no committed consumption may lose its required trace, and accumulation/history encoding must respect the existing byte budget without truncation.

The concrete contract/control packet must cover the preserved original mismatch and empty-before-later-consumption guards, kept/new streams, repeated calls, explicit overrides, self-send ordering, both failed-outcome carrier paths above, caught validation/storage errors, recorder and payload-budget failures, both reader exhaustion paths, and export/import/downgrade refusal before enabling a writer. The guard-floor/native/tool gates in #1030/#1037 remain separate and uncompleted; ordinary source preparation has no blanket hold. Workflow owns this contract/tests checkpoint; General and the selected independent review retain new-write-path adoption. Historical data remains distinguishable with its known unresolved #1024 limit.

General accepts the carrier/atomicity direction in 6670775639 for the next source contract and implementation checkpoint. This resolves the two failed-outcome design questions; it does not enable the observation writer.

I checked actual main 79b1a300d5a469b15a669ffe60c41a00a51eb2b0: WorkflowExecutor.php blob 8fd6cb79… invokes the live handler before UpdateApplied, then serializes/externalizes its result; its catch writes failed UpdateCompleted and returns through the existing restart path. QueryStateReplayer.php blob 53e0cc2f… currently re-executes only Applied handlers. MessageStream.php blob a5243fec… has the visible/opened-sequence fallback and makes receiveOne() delegate to receive().

Use one versioned envelope for the exact update/command invocation. Applied is the carrier when it exists for that invocation, including a later result-serialization failure. Otherwise failed Completed carries the committed consumption evidence. Choose by correlated Applied-event existence, not final update status; reject conflicting or duplicate carriers. A failed-Completed-only envelope is validated lineage, not permission to re-run the failed handler, reconstruct discarded state or lend consumption to another update.

The concrete source contract must retain these boundaries:

Resolve invocation context at each receive, including kept stream objects. One underlying receive() is one ordinal; receiveOne() must not double-record its delegated call. Keep pure existing invalid-limit/sequence validation before an ordinal, and explicit empty results as observations. Standalone peek/pending inspection does not silently acquire a new recorded-receive guarantee.
In observation mode use the validated recorded effective consumption sequence and exact rows, not the replay reader's current cursor or opened-step fallback. That distinction is necessary for the original query mismatch. Match stream bytes, limit, override equivalence, ordered IDs/sequences and all run/instance/update/command lineage. Missing/pruned/changed rows remain explicitly unavailable; no pending-message or invented-null fallback.
Recorder/storage/encoding/budget failure must remain sticky outside user catches and the ordinary failed-update catch: abort the containing task transaction so consumption cannot commit without its envelope. Preserve existing ordinary-handler failure/restart semantics when consumption and its trace really commit. Rollback must publish neither rows nor trace. Both successful-handler readers enforce exact call shape and exhaustion.
Freeze closed version/field/type/range and byte-budget rules in the packet, without truncation. Prove export/import and mixed/downgrade behavior using the actual compatibility mechanism: an older reader ignoring a new payload member is not a downgrade refusal. If a capable-reader cohort/floor cannot yet be enforced, name that precise adoption dependency and keep the writer disabled.

Workflow owns the exact packet and focused controls already listed in6670775639, preserving the original mismatch and empty-before-later-consumption regressions. Continue through the existing source train/pacing rule; no extra general source-merge approval is added. The selected independent review of the exact new-write boundary and actual1030/1037 guard-floor/tool/native requirements remain separate before writer adoption. Historical data retains its explicit unresolved limitation; no backfill, guessed legacy reconstruction, package release or operating action is selected here.

Fix in #1076, ready for review. Live update handlers now record the messages each stream receive returned on UpdateApplied, and both the executor and query replays return exactly those messages. The reproduction above now returns ['j1'] and then ['j1', 'j2']. A new corpus fixture, message-stream-update-receive-query, fails on main and passes with the fix.

Updates applied by earlier releases have no recorded messages. Queries keep the old step lookup for those updates, so they can still return the old answer.

The guard-floor drafts #1030 and #1037 are closed as superseded. Next: merge #1076 after CI reruns on current main, then ship it in the next release.

Merge pull request #1081 from durable-workflow/workflow/fix-cloud-tim…

Delivered in Workflow 2.23.0 (fix #1076). Release verification 43052014510 passed, and Packagist serves 2.23.0 at d5169dd.

Replay update handler stream receives from what the live handler took

Fix the Laravel Cloud timer cap test's delay sign and add durable-wor…

Navigation Menu

Repository navigation

Query replay of an update handler's stream receive returns nothing #1024

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Query replay of an update handler's stream receive returns nothing #1024

Description

Activity

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 5, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions