diff --git a/CHANGELOG.md b/CHANGELOG.md index a88dcc483..f72474869 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## Unreleased +- Experimental embedded compensation scopes retain bounded, versioned obligations + before effects and recover them through native Saga and recorded Activities. + Host checks run outside SQL locks. Recovery can restart after cancellation or + termination without repeating verified compensation. Scope forward ownership + stays with the originating run, including across continue-as-new and redrive. + - A child handed to a continued run now gets its retries (#1105). Before, a child that failed after its parent continued as new was never retried, because the retry needed the parent to be waiting. It is now retried under diff --git a/docs/api-stability.md b/docs/api-stability.md index 5131e6694..d18bf937c 100644 --- a/docs/api-stability.md +++ b/docs/api-stability.md @@ -114,7 +114,26 @@ The workflow package is consumed by three distinct audiences: into the workflow package as a library but do not author workflows. The semver guarantee below covers all three surfaces. Breaking changes to -anything marked `@api` require a major version bump. +anything marked `@api` require a major version bump, except surfaces explicitly +marked `@experimental` as described below. + +## Experimental compensation scopes + +The entire `Workflow\V2\Compensation` namespace is experimental, including +its authoring helpers, handler and authorization contracts, activities, +recovery helpers and persisted ledger implementation. Its types carry +`@experimental` in code. This namespace is not covered by the stability +guarantees in this document. Its API and storage contract may change in a +minor release while the first increment is evaluated. `@experimental` takes +precedence over any otherwise applicable public-API classification below. +`@internal` types remain internal and do not become supported host entry points. + +This first increment supports embedded Laravel, sequential compensation, one +nesting level and explicit release. It does not transfer forward scope ownership +across continue-as-new or originating-run redrive. Retained obligations stay +attached to the original run and remain recoverable through `ScopeRecovery`. +See [durable compensation scopes](compensation-scopes.md) for the operational +boundaries. Existing run-local Saga and its stability contract are unchanged. ## What is covered diff --git a/docs/compensation-scopes.md b/docs/compensation-scopes.md new file mode 100644 index 000000000..660317c68 --- /dev/null +++ b/docs/compensation-scopes.md @@ -0,0 +1,274 @@ +# Durable compensation scopes (experimental) + +Implementation of the embedded, sequential scope design in +[retained compensation proposal](https://github.com/durable-workflow/workflow/issues/1063), preparing the +nested-machine integration in [business state-machine proposal](https://github.com/durable-workflow/workflow/issues/1069). +This is not a released API. Existing run-local Saga and the four-method +`TransitionStep` contract remain unchanged. + +## Boundaries + +A scope is `(application, namespace, key, generation)`, not a workflow run. +The native default namespace remains `null`; it is never replaced by an invented +alias. The original run and a later recovery run have separate identities. One scope +contains at most 256 obligations. Stored descriptors contain a closed handler +identifier and version, stable operation key and portable data (16 KiB, depth 8, +512 values). They never contain executable closures or arbitrary class names. +Applications store protected references instead of credentials or customer data. +The principal comes only from trusted host code, never directly from webhook or +HTTP start input. Content hashes detect accidental corruption, not an attacker +with write access to both the descriptor and its hash. + +Persisted descriptors use explicit `schemaVersion: 1` arrays. Scope IDs and +obligation fingerprints hash that canonical schema, not PHP property reflection. +Unsupported versions fail closed. Recorded activities use stable +`dw.compensation.*.v1` type identifiers. Scope and recovery activity layouts carry +native version markers. This first experimental format does not qualify replay +of pre-versioned development histories. + +The registration journal and obligation commit together. Workflow history records +activity boundary outcomes; a lost activity acknowledgement is adopted by exact +identity. Forward work must use the scope admission gate. Registering a descriptor +alone does not make an unrelated activity cancellation-safe. + +Live database, authorization and handler resolution happen at activity boundaries, +never during deterministic replay. Native activity retries and native Saga remain +responsible for execution. No independent timer, worker or retry engine is added. +Every scope derives one canonical recovery workflow instance ID. Native DW +signal-with-start admits or adopts one canonical recovery run; native redrive admits one successor after an eligible +failure. A native returned-existing-active start receipt remains eligible for +redrive only when its run identity and accepted start-command snapshot match. +A technical redrive key includes the exact failed source run. One unchanged +retained request can therefore recover several failed successors while duplicate +requests for one source adopt the same successor. Scope progress must be written +under the current native activity attempt's authority, in the same database transaction as the ledger mutation. A status check +followed by an unguarded write is insufficient. No scope lease or separate owner +claim is introduced. Native fencing cannot recall an issued external request. +Handlers must reconcile unknown provider outcomes using stable operation keys. +A returned callback, timeout or expired lease is not verified compensation. + +Rows use the workflow storage connection. Scope rows serialize ledger mutations; +obligation and journal rows follow that lock. Host policy, input validation and provider calls run outside native SQL +locks. Their immutable policy version and descriptor fingerprint are captured +before admission. Under locks the boundary checks native authority, the pinned +principal and the exact immutable descriptor again. A later policy change is +checked at the next boundary, not by executing another host callback under locks. The ledger has no cascading foreign key to run history: normal history +pruning must not erase unresolved obligations or reuse their identities. + +## Runtime flow and responsibilities + +```mermaid +flowchart TD + A[DurableScope.execute] --> B[RegisterObligationActivity] + B --> C[Atomic descriptor and journal] + C --> D[Native Saga callback registered before effect] + D --> E[ForwardOperationActivity] + E --> F[ScopeBoundary: native attempt + authorization + gate] + F --> G[Host operation outside SQL locks] + G --> H[Guarded forward outcome] + R[ScopeRecovery.request] --> Q[Content-bound request closes gate] + Q --> I{Origin active?} + I -->|yes| J[Native cooperative cancellation and original Saga] + I -->|closed| K[Canonical RecoveryWorkflow: native start/adopt] + K --> L[Native Saga in reverse registration order] + J --> L + L --> M[CompensateObligationActivity] + M --> N{Ordinary operation or child scope?} + N -->|operation| O[Closed versioned handler registry] + N -->|child| P[Exact child request and verified resolution] + O --> V[Guarded authoritative evidence] + P --> V + V --> W{All obligations verified?} + W -->|yes| X[VerifyScopeActivity] + W -->|unknown or failed| Y[Native retries; retain unresolved ledger] + Y --> Z[Eligible native redrive reuses verified activity prefix] +``` + +Host `ScopeAuthorization::assertAllowed()` and `ScopedOperation::assertInput()` +run before the guarded transaction. Host callbacks must not leave an open SQL +transaction. Parent and child authorization decisions keep their own recorded +policy versions. No HTTP or provider callback runs while native rows are locked. + +`ScopeBoundary` uses the existing native activity status/authority checks while +retaining their SQL locks in the ledger transaction. Its lock order is native +attempt, execution, run and task, followed by the scope and its journal. It checks +authority again after the mutation. Provider calls run between guarded transactions, +never inside them. The journal records principal, authorization policy version, +origin/current run, activity and attempt references. Child participation records +the child authorization policy separately from the parent registration policy. +A missing or invalid policy version refuses participation before registration. +The host must bind +`ScopeAuthorization`; there is no permissive default. + +`HandlerRegistry` is supplied by trusted host code. Its entries implement +`ScopedOperation::assertInput()`, `execute()` and `compensate()`. These are provider +boundary handlers; they do not change the four `TransitionStep` methods. Forward +and compensation outcomes accept a list of at most 32 immutable evidence +references, each at most 128 bytes. This leaves room for native identity metadata inside the +existing 16 KiB, 512-value journal bound. Larger provider proofs should be stored +as protected evidence bundles and referenced by a stable identifier. An +uncertain forward result requires reconciliation against the stable operation ID +before an unsafe provider reissue. `Resolution::verified()` additionally requires +settled-forward evidence; neither a callback return nor a native terminal run proves +that business result. Handler unavailability and descriptor hash mismatch refuse +before both forward and compensation provider operations. Forward handler and input +validation run before dispatch admission: a missing pinned handler or incompatible +input retains its native nonretryable reason without creating provider uncertainty. +A refused retry preserves any earlier uncertain attempt. Exceptions after provider +dispatch remain unknown and require reconciliation, even if their PHP type is +nonretryable. + +`ChildScopes` registers compensation participants, not state-machine bindings. +The initial graph is a bounded tree with one nesting level, at most 16 children +per parent and at most 256 obligations per scope (at most 4,352 across that tree). +Children cannot be shared between parents. One application/namespace graph-integrity +row serializes edge changes before scope locks. It contains no executor, lease, +heartbeat or retry state. A missing or unverified child stops earlier parent work. +Long child cleanup can exhaust the ordinary native activity retry budget; the +parent remains unresolved and can resume through eligible native redrive. + +A scope stays attached to its original native run for forward execution. This +increment does not transfer forward ownership across continue-as-new or origin +redrive, or a native child retry. `generation` must be unique for each originating +native run. Reusing a generation from a new run fails with the non-retryable +`scope_owned_by_another_run` reason. The original retained obligations remain +recoverable through their exact original scope identity. + +The first accepted request pins the cleanup principal. Later authorized +requests can route recovery but cannot replace that principal. Recovery evaluates +current authorization again at every activity boundary. Revoked authority leaves +work unresolved rather than silently adopting another principal. + +`ScopeRecovery::request()` on an active origin requests cancellation of the whole +originating run. Native Saga then executes all its registered callbacks, including +other scopes and callbacks not managed by scopes. A request is not a selective +rollback of one scope while that origin continues forward. + +A canonical recovery that was cancelled or terminated can be restarted through +native signal-with-start on the same recovery instance. This creates a fresh run, +keeps closed histories unchanged, and skips already verified ledger obligations. +Failed recovery still uses native redrive when its history is eligible. + +All scoped activities use three native attempts and +the native backoff. No scope retry loop is introduced. `DurableScope.execute()` +accepts native `ActivityOptions` for forward work. Immediate terminal cancellation +or termination can prevent cleanup; retained obligations remain for explicit +recovery. A request accepted on an active origin is durable routing evidence; if +that origin closes without resolving its obligations, repeat the same authorized +request to route it to canonical recovery. + +`ScopeRetention::release()` requires verified work and terminal native executions, +and refuses a child still retained by an unresolved parent. It removes retained +inputs and forward results while preserving scope/generation, operation and request +identities, fingerprints, outcome references and journal tombstones. Forward +observation references are recorded in the immutable journal before release. +The journal does not retain the forward business value or obligation input. +No automatic expiry is added. Ordinary run-detail pruning cannot cascade into this ledger. + +## Continue-as-new and originating-run redrive + +Scope identity and native child supervision are separate. When a workflow +continues as new, native DW can hand its still-open children to the continued +run. This does not transfer the parent's compensation scope or the children's +own scopes. Each retained scope keeps its `original_run_id`. A parent that must +compensate a child registers it with `participate()` before dispatch, regardless +of which run later supervises that child. + +New registration from a continued or redriven origin is refused by the retained +original-run binding. A redrive may replay an observed activity prefix, but it +cannot use that prefix to dispatch new work in the old scope. Resuming an +uncertain forward activity in the successor is also refused by `ScopeBoundary` +before provider dispatch. The original obligations and provider evidence remain +available. Hosts request `ScopeRecovery` for that exact identity and generation +to resolve them. This increment does not silently create replacement scopes, +move obligations to a new generation or infer compensation from run closure. + +`V2CompensationScopeRunBoundaryTest` exercises real continue-as-new with an open +scoped child, native supervision handoff, refused successor work and verified +child-first recovery. It also checks originating-run redrive after an unrelated +failed activity and after an uncertain scoped forward effect. Recovery preserves +the original terminal history in all cases. + +## Core redrive regression evidence + +The replay corpus policy explicitly guards `FailedRunRedrivePlan` and +`ActivityRetryHistory`, so changes to either require executable regression evidence. + +The planner changes are independent of compensation scopes. They accept native +`ActivityRetryScheduled` lineage only when execution, attempt, task and attempt +number references agree. A valid native active-start adoption receipt must also +match the run and accepted command snapshot. The planner keeps current main's +completed-child reuse behavior. + +The `redrive-retried-completed-activity`, `redrive-retried-failed-activity` and +`redrive-retried-activity-after-completed-child` regression-corpus fixtures retain +both a cold replay prefix and the failed source history. The official +`ReplayRegressionCorpusTest` consumer executes `FailedRunRedrivePlan` against the +source and its identity-substitution cases, then executes the cold workflow +replay. This provides planner evidence separately from the scope integration +tests. `FailedRunRedrivePlanTest` separately covers start-adoption receipt +substitution. The separate `redrive-native-active-start-adoption` corpus fixture +executes the same acceptance and substitution checks for recorded start receipts. +Its planner source retains the native audit-enriched command snapshot. Its cold +replay prefix contains the authored event payload, which is checked against the +strict event payload contract before native audit enrichment. +Existing completed-child corpus and feature regressions remain +applicable. These evidence paths allow the core planner change to be reviewed +or extracted separately from the experimental scope API. + +## Authoring and host setup + +The embedded host binds `ScopeAuthorization` to its authorization policy and +`HandlerRegistry` to a closed map of handler type/version pairs. Normal package +migrations create the ledger tables on the configured workflow storage connection. +No additional queue, worker or lease configuration is required. + +Inside a native workflow, create `DurableScope($this, $identity, $principal)` and +call `execute(new Obligation($key, $handlerType, $handlerVersion, $input))`. Register +all effects through this helper. On cleanup, call the existing +`$this->compensate()` and then `$scope->verifyCompensation()`. The helper does not +catch the application's business failures or decide its business state. + +`participate($key, $childIdentity)` registers a compensation participant. It does +not start a child, await its business result or establish a state-machine binding. +Register that participant before dispatching new child work. Each child registers +its own provider obligations before effects. A missing child remains unresolved +and cannot satisfy the parent barrier. + +After the origin closes, the host calls `ScopeRecovery::request($identity, +$requestKey, $principal, $reason)`. Obtain the principal from trusted application +authorization, not from untrusted request content. Repeating the same request +requires the same principal and reason. Inspecting or requesting retained work +must go through an authorized host boundary; the `@internal` persistence classes +are not an HTTP-facing data access API. + +## Evidence and review + +| Guarantee | Responsible component | Acceptance test | +| --- | --- | --- | +| Exact request and generation identity | ScopeIdentity, ScopeRepository, ScopeRecovery | ScopeRecoveryTest, V2CompensationScopeConcurrencyTest | +| Current native attempt owns ledger mutation | ScopeBoundary | ExpiredNativeAttemptCannotPublishVerifiedCompensation | +| No false success for late/unknown effects | ScopeProgress, CompensateObligationActivity | PossibleLateForwardEffectCannotBecomeVerifiedAfterRetryExhaustion | +| Closed-run recovery and original terminal history | RecoveryWorkflow, native Saga | ClosedOriginUsesCanonicalRecoveryAndKeepsOriginalTerminalHistory | +| Verified progress survives failure and redrive | Native redrive plus ledger | RetryExhaustionStaysUnresolvedAndNativeRedriveReusesVerifiedWork, OneRetainedRequestCanRedriveTwoFailuresWithoutRepeatingVerifiedWork | +| Native start adoption preserves eligible redrive | FailedRunRedrivePlan | NativeActiveAdoptionDoesNotPreventRecoveryRedriveAfterFailure, SubstitutedActiveStartAdoptionCannotAuthorizeRedrive | +| Forward evidence survives explicit release | ScopeProgress, ScopeRetention | ForwardEvidenceSurvivesReleaseWithoutRetainingBusinessInputsOrResults | +| Replay does no handler/provider resolution | DurableScope, recorded activities | ColdReplayDoesNotResolveMutableHandlersOrRepeatProviderEffects | +| Child-first sequential progress | ChildScopes, ChildResolution | CompletedChildScopeIsVerifiedBeforeEarlierParentCompensation, CancellationWhileChildWaitsResolvesItsSagaBeforeEarlierParentWork | +| Child authorization evidence | RegisterChildScopeActivity | ChildParticipationRecordsItsOwnAuthorizationPolicyVersion, InvalidChildParticipationPolicyCannotRegisterAChild | +| Retained descriptor integrity before effects | ForwardOperationActivity, CompensateObligationActivity | TamperedRegistrationCannotReachTheForwardProvider, TamperedRetainedDescriptorIsRefusedBeforeProviderCall | +| Live storage is forbidden during replay | ScopeRepository, ScopeStorage | OpeningLiveStorageFromWorkflowReplayIsRefusedBeforeWriting | +| Bounded retained work | ScopeRepository, ChildScopes | ObligationLimitCannotLeaveAPartialRegistration, ChildFanOutLimitCannotLeaveAPartialParticipant | +| No cycle under concurrent registration | ChildScopes integrity lock | CompetingEdgesCannotCreateACycle | +| Ordinary pruning preserves recovery | Independent retained ledger | OrdinaryRunPruningPreservesRetainedScopeRecovery | +| Explicit release cannot reuse identifiers | ScopeRetention | ReleaseRefusesUnresolvedWorkThenRetainsAnUnreusableTombstone | +| Fresh-process recovery | Native workflow and persisted ledger | RetainedRecoveryRunsInAFreshPhpProcess | + +Integration exposed a native redrive limitation: `ActivityRetryScheduled` was +refused as unsupported history. `ActivityRetryHistory` validates exact execution, +attempt counters and task lineage before the existing redrive planner accepts that +history. Substitution controls accompany the real retry-exhaustion regression. + +Local evidence does not establish a published package, CI matrix qualification, +service-mode/cross-SDK support or production-provider behavior. The full quality +cycle and final evidence counts are recorded after the last code review. diff --git a/regression-corpus-policy.json b/regression-corpus-policy.json index 114529c98..2df39af3d 100644 --- a/regression-corpus-policy.json +++ b/regression-corpus-policy.json @@ -74,6 +74,12 @@ "history_sequence", "history_payload" ] + }, + { + "glob": "src/V2/Support/FailedRunRedrivePlan.php" + }, + { + "glob": "src/V2/Support/ActivityRetryHistory.php" } ] } diff --git a/src/V2/Compensation/Activities/CompensateObligationActivity.php b/src/V2/Compensation/Activities/CompensateObligationActivity.php new file mode 100644 index 000000000..2ac62d5ad --- /dev/null +++ b/src/V2/Compensation/Activities/CompensateObligationActivity.php @@ -0,0 +1,89 @@ +run($this, $scope, 'compensate', null, function (array $snapshot) use ( + $scope, + $id, + &$origin + ): ?array { + $origin = $snapshot['original_run_id']; + return app(ScopeProgress::class)->beginCompensation($scope, $id, $this->attemptId()); + }); + if ($row === null) { + return; + } + $obligation = Obligation::fromStored($row); + $context = new OperationContext( + $scope, + $obligation, + $id, + $origin, + $this->activityId(), + $this->attemptId(), + $row['forward_status'] !== 'observed', + $row['forward_activity_id'], + $row['forward_attempt_id'], + ); + $unavailable = false; + try { + if ($obligation->handlerType === ChildScopes::TYPE) { + $result = app(ChildResolution::class)->resolve($scope, $obligation, $id); + } else { + $handler = app(HandlerRegistry::class)->resolve($obligation); + $result = $row['forward_status'] === 'registered' + ? Resolution::verified(['never-dispatched:' . $id], true) + : $handler->compensate($context); + } + } catch (NonRetryableException) { + $unavailable = true; + $result = Resolution::unknown('handler-unavailable'); + } catch (Throwable) { + $result = Resolution::unknown('provider-outcome-unknown'); + } + $boundary->run( + $this, + $scope, + 'compensate', + null, + fn () => app(ScopeProgress::class)->resolve($scope, $id, $this->attemptId(), $result) + ); + if ($unavailable) { + throw new NonRetryableException('scope_handler_unavailable'); + } + if (! $result->verified) { + throw new \RuntimeException('scope_compensation_unknown'); + } + } +} diff --git a/src/V2/Compensation/Activities/ForwardOperationActivity.php b/src/V2/Compensation/Activities/ForwardOperationActivity.php new file mode 100644 index 000000000..67c704b8e --- /dev/null +++ b/src/V2/Compensation/Activities/ForwardOperationActivity.php @@ -0,0 +1,104 @@ +run( + $this, + $scope, + 'forward', + $principal, + function () use ($scope, $id, &$handler, &$fingerprint): array { + return app(ScopeProgress::class)->admitForward( + $scope, + $id, + $this->attemptId(), + $this->activityId(), + static function (array $row) use (&$handler, &$fingerprint): void { + if (! $handler instanceof ScopedOperation || $row['fingerprint'] !== $fingerprint) { + throw new \LogicException('The prepared forward descriptor changed before admission.'); + } + } + ); + }, + static function () use ($scope, $id, &$handler, &$fingerprint): void { + $snapshot = app(ScopeRepository::class)->inspect($scope); + foreach ($snapshot['obligations'] as $row) { + if ($row['id'] !== $id) { + continue; + } + if ($row['forward_status'] !== 'observed') { + $row['descriptor'] = PortableDescriptor::encode($row['descriptor']); + $handler = app(HandlerRegistry::class)->resolve(Obligation::fromStored($row)); + $fingerprint = $row['fingerprint']; + } + return; + } + throw new \LogicException('The prepared forward obligation is unavailable.'); + } + ); + if ($admission['cached'] !== null) { + return $admission['cached']['value']; + } + if (! $handler instanceof ScopedOperation) { + throw new \LogicException('The forward handler was not validated before dispatch.'); + } + $obligation = Obligation::fromStored($admission['row']); + $context = new OperationContext( + $scope, + $obligation, + $id, + $this->runId(), + $this->activityId(), + $this->attemptId(), + $admission['uncertain'] + ); + try { + $result = $handler->execute($context); + } catch (Throwable) { + $result = ForwardResult::unknown('provider-outcome-unknown'); + } + $boundary->run( + $this, + $scope, + 'forward', + $principal, + fn () => app(ScopeProgress::class)->forwardResult($scope, $id, $this->attemptId(), $result) + ); + if (! $result->observed) { + throw new \RuntimeException('scope_forward_unknown'); + } + return $result->value; + } +} diff --git a/src/V2/Compensation/Activities/PrepareRecoveryActivity.php b/src/V2/Compensation/Activities/PrepareRecoveryActivity.php new file mode 100644 index 000000000..128d80681 --- /dev/null +++ b/src/V2/Compensation/Activities/PrepareRecoveryActivity.php @@ -0,0 +1,38 @@ +run( + $this, + $scope, + 'compensate', + null, + static function (array $snapshot): array { + if (! in_array($snapshot['status'], ['requested', 'verified'], true)) { + throw new \LogicException('The scope has no accepted compensation request.'); + } + return array_column($snapshot['obligations'], 'id'); + } + ); + } +} diff --git a/src/V2/Compensation/Activities/RegisterChildScopeActivity.php b/src/V2/Compensation/Activities/RegisterChildScopeActivity.php new file mode 100644 index 000000000..fd7c3ac15 --- /dev/null +++ b/src/V2/Compensation/Activities/RegisterChildScopeActivity.php @@ -0,0 +1,66 @@ +run( + $this, + $scope, + 'register', + $principal, + function (?array $snapshot, string $parentPolicy, string $policy) use ( + $scope, + $key, + $child, + $principal + ): string { + $id = app(ChildScopes::class)->register($scope, $this->runId(), $key, $child); + ScopeStorage::append(ScopeStorage::locked($scope), 'child_authorized', [ + 'childScopeId' => $child->id(), + 'principal' => $principal, + 'operation' => 'participate', + 'policyVersion' => $policy, + 'runId' => $this->runId(), + 'activityId' => $this->activityId(), + 'attemptId' => $this->attemptId(), + ]); + return $id; + }, + static function () use ($child, $principal): string { + try { + $policy = app(ScopeAuthorization::class)->assertAllowed($child, $principal, 'participate'); + PortableDescriptor::identifier($policy, 'authorization policy version'); + return $policy; + } catch (Throwable) { + throw new NonRetryableException('scope_child_authorization_refused'); + } + } + ); + } +} diff --git a/src/V2/Compensation/Activities/RegisterObligationActivity.php b/src/V2/Compensation/Activities/RegisterObligationActivity.php new file mode 100644 index 000000000..f58dcdd2d --- /dev/null +++ b/src/V2/Compensation/Activities/RegisterObligationActivity.php @@ -0,0 +1,38 @@ +run($this, $scope, 'register', $principal, function () use ( + $scope, + $obligation + ): string { + $repository = app(ScopeRepository::class); + $repository->open($scope, $this->runId()); + return $repository->register($scope, $obligation); + }, static fn () => app(HandlerRegistry::class)->resolve($obligation)); + } +} diff --git a/src/V2/Compensation/Activities/RequestScopeActivity.php b/src/V2/Compensation/Activities/RequestScopeActivity.php new file mode 100644 index 000000000..6cf36f662 --- /dev/null +++ b/src/V2/Compensation/Activities/RequestScopeActivity.php @@ -0,0 +1,40 @@ +run( + $this, + $scope, + 'request', + $principal, + static fn (array $snapshot, string $policyVersion) => app(ScopeRepository::class)->request( + $scope, + 'native-saga', + $principal, + 'workflow-cleanup', + $policyVersion + ) + ); + } +} diff --git a/src/V2/Compensation/Activities/VerifyScopeActivity.php b/src/V2/Compensation/Activities/VerifyScopeActivity.php new file mode 100644 index 000000000..65c29b6ba --- /dev/null +++ b/src/V2/Compensation/Activities/VerifyScopeActivity.php @@ -0,0 +1,34 @@ +run( + $this, + $scope, + 'compensate', + null, + static fn () => app(ScopeProgress::class)->finish($scope) + ); + } +} diff --git a/src/V2/Compensation/Definition/Obligation.php b/src/V2/Compensation/Definition/Obligation.php new file mode 100644 index 000000000..e7c315594 --- /dev/null +++ b/src/V2/Compensation/Definition/Obligation.php @@ -0,0 +1,70 @@ +toArray()); + } + + public function fingerprint(): string + { + return hash('sha256', PortableDescriptor::encode($this->toArray())); + } + + public function toArray(): array + { + return [ + 'schemaVersion' => 1, + 'key' => $this->key, + 'handlerType' => $this->handlerType, + 'handlerVersion' => $this->handlerVersion, + 'input' => $this->input, + ]; + } + + public static function fromStored(array $row): self + { + $value = self::fromArray(json_decode($row['descriptor'], true, flags: JSON_THROW_ON_ERROR)); + if (! hash_equals($row['fingerprint'], $value->fingerprint())) { + throw new \LogicException('The retained compensation descriptor content hash does not match.'); + } + return $value; + } + + public static function fromArray(array $value): self + { + $keys = array_keys($value); + sort($keys, SORT_STRING); + if (($value['schemaVersion'] ?? null) !== 1 || $keys !== [ + 'handlerType', + 'handlerVersion', + 'input', + 'key', + 'schemaVersion', + ]) { + throw new InvalidArgumentException('Invalid compensation obligation shape.'); + } + return new self($value['key'], $value['handlerType'], $value['handlerVersion'], $value['input']); + } +} diff --git a/src/V2/Compensation/Definition/PortableDescriptor.php b/src/V2/Compensation/Definition/PortableDescriptor.php new file mode 100644 index 000000000..e39309d51 --- /dev/null +++ b/src/V2/Compensation/Definition/PortableDescriptor.php @@ -0,0 +1,75 @@ + self::MAX_BYTES) { + throw new InvalidArgumentException('Compensation data exceeds the byte limit.'); + } + return $json; + } + + public static function identifier(string $value, string $name): void + { + if ($value === '' || strlen($value) > 128 || ! preg_match('/\A[a-zA-Z0-9][a-zA-Z0-9._:\/-]*\z/D', $value)) { + throw new InvalidArgumentException("Invalid compensation {$name}."); + } + } + + private static function normalize(mixed $value, int $depth, int &$count): mixed + { + if (++$count > self::MAX_VALUES || $depth > self::MAX_DEPTH) { + throw new InvalidArgumentException('Compensation data exceeds the structural limit.'); + } + if (is_array($value)) { + if (! array_is_list($value)) { + ksort($value, SORT_STRING); + } + foreach ($value as $key => $child) { + $value[$key] = self::normalize($child, $depth + 1, $count); + } + return $value; + } + if ($value === null || is_bool($value) || is_int($value) || is_string($value) + || (is_float($value) && is_finite($value))) { + return $value; + } + throw new InvalidArgumentException( + 'Compensation data cannot contain objects, resources or non-finite numbers.' + ); + } +} diff --git a/src/V2/Compensation/Definition/ScopeIdentity.php b/src/V2/Compensation/Definition/ScopeIdentity.php new file mode 100644 index 000000000..9ed1161ca --- /dev/null +++ b/src/V2/Compensation/Definition/ScopeIdentity.php @@ -0,0 +1,67 @@ + $application, + 'namespace' => $namespace, + 'key' => $key, + 'generation' => $generation, + ] as $name => $value) { + if ($name === 'namespace' && $value === null) { + continue; + } + PortableDescriptor::identifier($value, $name); + } + } + + public function id(): string + { + return hash('sha256', PortableDescriptor::encode($this->toArray())); + } + + public function recoveryInstanceId(): string + { + return 'dw-compensation-' . $this->id(); + } + + public function toArray(): array + { + return [ + 'schemaVersion' => 1, + 'application' => $this->application, + 'namespace' => $this->namespace, + 'key' => $this->key, + 'generation' => $this->generation, + ]; + } + + public static function fromArray(array $value): self + { + $keys = array_keys($value); + sort($keys, SORT_STRING); + if (($value['schemaVersion'] ?? null) !== 1 || $keys !== [ + 'application', + 'generation', + 'key', + 'namespace', + 'schemaVersion', + ]) { + throw new \InvalidArgumentException('Invalid compensation scope identity schema.'); + } + return new self($value['application'], $value['namespace'], $value['key'], $value['generation']); + } +} diff --git a/src/V2/Compensation/DurableScope.php b/src/V2/Compensation/DurableScope.php new file mode 100644 index 000000000..9a04a3af7 --- /dev/null +++ b/src/V2/Compensation/DurableScope.php @@ -0,0 +1,71 @@ +identity->toArray(); + $principal = $this->principal; + $id = Workflow::activity(RegisterObligationActivity::class, $scope, $obligation->toArray(), $principal); + // The callback is rebuilt by replay before dispatching the effect and never needs its return value. + $this->saga->addCompensation(static function () use ($scope, $principal, $id): void { + Workflow::activity(RequestScopeActivity::class, $scope, $principal); + Workflow::activity(CompensateObligationActivity::class, $scope, $id); + }); + return Workflow::activity( + ForwardOperationActivity::class, + $options ?? new ActivityOptions(), + $scope, + $id, + $principal + ); + } + + public function participate(string $key, ScopeIdentity $child): void + { + $scope = $this->identity->toArray(); + $principal = $this->principal; + $id = Workflow::activity(RegisterChildScopeActivity::class, $scope, $key, $child->toArray(), $principal); + $this->saga->addCompensation(static function () use ($scope, $principal, $id): void { + Workflow::activity(RequestScopeActivity::class, $scope, $principal); + Workflow::activity(CompensateObligationActivity::class, $scope, $id); + }); + } + + public function verifyCompensation(): array + { + return Workflow::activity(VerifyScopeActivity::class, $this->identity->toArray()); + } +} diff --git a/src/V2/Compensation/Execution/EvidenceReferences.php b/src/V2/Compensation/Execution/EvidenceReferences.php new file mode 100644 index 000000000..4fda56551 --- /dev/null +++ b/src/V2/Compensation/Execution/EvidenceReferences.php @@ -0,0 +1,36 @@ + self::MAX_REFERENCES) { + throw new InvalidArgumentException('Compensation data exceeds the evidence reference limit.'); + } + foreach ($references as $reference) { + if (! is_string($reference)) { + throw new InvalidArgumentException('Evidence requires immutable reference strings.'); + } + PortableDescriptor::identifier($reference, 'evidence reference'); + } + } +} diff --git a/src/V2/Compensation/Execution/ForwardResult.php b/src/V2/Compensation/Execution/ForwardResult.php new file mode 100644 index 000000000..05a951191 --- /dev/null +++ b/src/V2/Compensation/Execution/ForwardResult.php @@ -0,0 +1,49 @@ +toArray()); + } + + public static function observed(array $value, array $evidence): self + { + return new self(true, $value, $evidence, 'observed'); + } + + public static function unknown(string $reason): self + { + return new self(false, [], [], $reason); + } + + public function toArray(): array + { + return [ + 'schemaVersion' => 1, + 'observed' => $this->observed, + 'value' => $this->value, + 'evidence' => $this->evidence, + 'reason' => $this->reason, + ]; + } +} diff --git a/src/V2/Compensation/Execution/HandlerRegistry.php b/src/V2/Compensation/Execution/HandlerRegistry.php new file mode 100644 index 000000000..b9cc28f43 --- /dev/null +++ b/src/V2/Compensation/Execution/HandlerRegistry.php @@ -0,0 +1,48 @@ +> $handlers + */ + public function __construct( + private array $handlers = [] + ) { + if (isset($handlers[\Workflow\V2\Compensation\Persistence\ChildScopes::TYPE])) { + throw new \InvalidArgumentException('The child-scope handler key is reserved.'); + } + foreach ($handlers as $versions) { + foreach ($versions as $handler) { + if (! $handler instanceof ScopedOperation) { + throw new \InvalidArgumentException('Scope handlers must implement ScopedOperation.'); + } + } + } + } + + public function resolve(Obligation $obligation): ScopedOperation + { + $handler = $this->handlers[$obligation->handlerType][$obligation->handlerVersion] ?? null; + if (! $handler instanceof ScopedOperation) { + throw new NonRetryableException('scope_handler_unavailable'); + } + try { + $handler->assertInput($obligation->input); + } catch (\Throwable) { + throw new NonRetryableException('scope_input_incompatible'); + } + return $handler; + } +} diff --git a/src/V2/Compensation/Execution/OperationContext.php b/src/V2/Compensation/Execution/OperationContext.php new file mode 100644 index 000000000..6ca3caac3 --- /dev/null +++ b/src/V2/Compensation/Execution/OperationContext.php @@ -0,0 +1,29 @@ +toArray()); + } + + public static function verified(array $evidence, bool $forwardSettled): self + { + return new self(true, 'verified', $evidence, $forwardSettled); + } + + public static function unknown(string $reason): self + { + return new self(false, $reason, [], false); + } + + public function toArray(): array + { + return [ + 'schemaVersion' => 1, + 'verified' => $this->verified, + 'reason' => $this->reason, + 'evidence' => $this->evidence, + 'forwardSettled' => $this->forwardSettled, + ]; + } +} diff --git a/src/V2/Compensation/Execution/ScopedOperation.php b/src/V2/Compensation/Execution/ScopedOperation.php new file mode 100644 index 000000000..9e97dd0e0 --- /dev/null +++ b/src/V2/Compensation/Execution/ScopedOperation.php @@ -0,0 +1,25 @@ +application !== $child->application || $parent->namespace !== $child->namespace || $parent->id() === $child->id()) { + throw new LogicException('A child scope must have another identity in the same application and namespace.'); + } + $database = ConfiguredV2Models::connection(); + $graph = hash('sha256', PortableDescriptor::encode([$parent->application, $parent->namespace])); + return SqliteTransaction::run(static function () use ( + $parent, + $originalRunId, + $key, + $child, + $database, + $graph + ): string { + // Serialize edge mutations before scope locks. This is a graph-integrity lock, not executor ownership. + $locked = $database->table('workflow_compensation_graphs') + ->where('id', $graph) + ->lockForUpdate() + ->first(); + if ($locked === null) { + $database->table('workflow_compensation_graphs') + ->insertOrIgnore([ + 'id' => $graph, + ]); + $database->table('workflow_compensation_graphs') + ->where('id', $graph) + ->lockForUpdate() + ->first(); + } + $existing = $database->table('workflow_compensation_children') + ->where('child_id', $child->id()) + ->lockForUpdate() + ->first(); + if (($existing !== null && $existing->parent_id !== $parent->id()) + || $database->table('workflow_compensation_children') + ->where('child_id', $parent->id()) + ->lockForUpdate() + ->first() !== null + || $database->table('workflow_compensation_children') + ->where('parent_id', $child->id()) + ->lockForUpdate() + ->first() !== null) { + throw new LogicException( + 'A child scope cannot be shared, cyclic, or nested beyond the supported depth.' + ); + } + if ($existing === null && $database->table('workflow_compensation_children')->where( + 'parent_id', + $parent->id() + )->lockForUpdate() + ->get() + ->count() >= self::MAX_CHILDREN) { + throw new LogicException('The child scope fan-out limit is exhausted.'); + } + $repository = new ScopeRepository(); + $repository->open($parent, $originalRunId); + $obligation = new Obligation($key, self::TYPE, self::VERSION, [ + 'child' => $child->toArray(), + ]); + $id = $repository->register($parent, $obligation); + if ($existing !== null && $existing->obligation_id !== $id) { + throw new LogicException('The child generation already belongs to another participant.'); + } + if ($existing === null) { + $database->table('workflow_compensation_children') + ->insert([ + 'child_id' => $child->id(), + 'parent_id' => $parent->id(), + 'obligation_id' => $id, + ]); + } + return $id; + }, attempts: 3, connection: $database); + } +} diff --git a/src/V2/Compensation/Persistence/ScopeProgress.php b/src/V2/Compensation/Persistence/ScopeProgress.php new file mode 100644 index 000000000..ca61b03d2 --- /dev/null +++ b/src/V2/Compensation/Persistence/ScopeProgress.php @@ -0,0 +1,195 @@ +obligation($scope, $database, $id); + if ($scope['status'] !== 'open') { + throw new LogicException('The scope has closed forward admission.'); + } + if ($row['forward_status'] === 'observed') { + return [ + 'cached' => json_decode($row['forward_result'], true, flags: JSON_THROW_ON_ERROR), + 'uncertain' => false, + 'row' => $row, + ]; + } + $beforeDispatch?->__invoke($row); + $uncertain = $row['forward_status'] !== 'registered'; + $database->table(ScopeStorage::OBLIGATIONS)->where('id', $id)->update([ + 'forward_status' => 'in_flight', + 'forward_activity_id' => $activityId, + 'forward_attempt_id' => $attemptId, + ]); + ScopeStorage::append($scope, 'forward_admitted', [ + 'obligationId' => $id, + 'attemptId' => $attemptId, + 'uncertain' => $uncertain, + ]); + return [ + 'cached' => null, + 'uncertain' => $uncertain, + 'row' => $row, + ]; + }); + } + + public function forwardResult(ScopeIdentity $identity, string $id, string $attemptId, ForwardResult $result): void + { + ScopeStorage::mutate($identity, function (array $scope, Connection $database) use ( + $id, + $attemptId, + $result + ): void { + $row = $this->obligation($scope, $database, $id); + $payload = PortableDescriptor::encode($result->toArray()); + if ($row['forward_attempt_id'] !== $attemptId) { + throw new LogicException('The forward result belongs to another native attempt.'); + } + if ($row['forward_result'] === $payload && $row['forward_status'] === ($result->observed ? 'observed' : 'unknown')) { + return; + } + if ($row['forward_status'] === 'observed' || $row['compensation_status'] === 'verified') { + throw new LogicException('A terminal scope outcome cannot be replaced.'); + } + $database->table(ScopeStorage::OBLIGATIONS)->where('id', $id)->update([ + 'forward_status' => $result->observed ? 'observed' : 'unknown', + 'forward_result' => $payload, + ]); + ScopeStorage::append($scope, 'forward_outcome', [ + 'obligationId' => $id, + 'attemptId' => $attemptId, + 'observed' => $result->observed, + 'reason' => $result->reason, + 'evidence' => $result->evidence, + ]); + }); + } + + public function beginCompensation(ScopeIdentity $identity, string $id, string $attemptId): ?array + { + return ScopeStorage::mutate($identity, function (array $scope, Connection $database) use ( + $id, + $attemptId + ): ?array { + $row = $this->obligation($scope, $database, $id); + if (! in_array($scope['status'], ['requested', 'verified'], true)) { + throw new LogicException('Compensation requires an accepted request.'); + } + if ($row['compensation_status'] === 'verified') { + return null; + } + $next = $database->table(ScopeStorage::OBLIGATIONS)->where('scope_id', $scope['id']) + ->where('compensation_status', '<>', 'verified') + ->orderByDesc('position') + ->value('id'); + if ($next !== $id) { + throw new LogicException('Sequential compensation must resolve later obligations first.'); + } + $database->table(ScopeStorage::OBLIGATIONS)->where('id', $id)->update([ + 'compensation_status' => 'executing', + 'compensation_attempt_id' => $attemptId, + ]); + ScopeStorage::append($scope, 'compensation_started', [ + 'obligationId' => $id, + 'attemptId' => $attemptId, + ]); + return $row; + }); + } + + public function resolve(ScopeIdentity $identity, string $id, string $attemptId, Resolution $result): void + { + ScopeStorage::mutate($identity, function (array $scope, Connection $database) use ( + $id, + $attemptId, + $result + ): void { + $row = $this->obligation($scope, $database, $id); + $payload = PortableDescriptor::encode($result->toArray()); + if ($row['compensation_attempt_id'] !== $attemptId) { + throw new LogicException('The compensation outcome belongs to another native attempt.'); + } + if ($row['evidence'] === $payload && $row['compensation_status'] === ($result->verified ? 'verified' : 'unknown')) { + return; + } + if ($row['compensation_status'] === 'verified' || ($result->verified && ! $result->forwardSettled)) { + throw new LogicException('Compensation is not verified while a late forward effect is still possible.'); + } + $database->table(ScopeStorage::OBLIGATIONS)->where('id', $id)->update([ + 'compensation_status' => $result->verified ? 'verified' : 'unknown', + 'evidence' => $payload, + ]); + ScopeStorage::append($scope, 'compensation_outcome', [ + 'obligationId' => $id, + 'attemptId' => $attemptId, + 'verified' => $result->verified, + 'reason' => $result->reason, + 'evidence' => $result->evidence, + ]); + }); + } + + public function finish(ScopeIdentity $identity): array + { + return ScopeStorage::mutate($identity, static function (array $scope, Connection $database): array { + if (! in_array($scope['status'], ['requested', 'verified'], true) + || $database->table(ScopeStorage::OBLIGATIONS)->where('scope_id', $scope['id']) + ->where('compensation_status', '<>', 'verified') + ->exists()) { + throw new LogicException('The scope retains unresolved compensation obligations.'); + } + if ($scope['status'] !== 'verified') { + $database->table(ScopeStorage::SCOPES)->where('id', $scope['id'])->update([ + 'status' => 'verified', + ]); + ScopeStorage::append($scope, 'verified', []); + } + return [ + 'scopeId' => $scope['id'], + 'status' => 'verified', + ]; + }); + } + + private function obligation(array $scope, Connection $database, string $id): array + { + $row = $database->table(ScopeStorage::OBLIGATIONS)->where('scope_id', $scope['id'])->where('id', $id)->first(); + if ($row === null) { + throw new LogicException('The obligation is unavailable in the exact scope generation.'); + } + return (array) $row; + } +} diff --git a/src/V2/Compensation/Persistence/ScopeRepository.php b/src/V2/Compensation/Persistence/ScopeRepository.php new file mode 100644 index 000000000..1ae111015 --- /dev/null +++ b/src/V2/Compensation/Persistence/ScopeRepository.php @@ -0,0 +1,208 @@ +table(self::SCOPES)->where('id', $identity->id())->lockForUpdate()->first(); + $created = $existing === null ? $database->table(self::SCOPES)->insertOrIgnore([ + 'id' => $identity->id(), + 'identity' => PortableDescriptor::encode($identity->toArray()), + 'original_run_id' => $originalRunId, + 'status' => 'open', + ]) : 0; + $scope = ScopeStorage::locked($identity); + if ($scope['original_run_id'] !== $originalRunId) { + throw new NonRetryableException('scope_owned_by_another_run'); + } + if ($created === 1) { + ScopeStorage::append($scope, 'opened', [ + 'originalRunId' => $originalRunId, + ]); + } + }, attempts: 3, connection: $database); + } + + public function register(ScopeIdentity $identity, Obligation $obligation): string + { + return ScopeStorage::mutate($identity, static function (array $scope, Connection $database) use ( + $identity, + $obligation + ): string { + if ($scope['status'] === 'released') { + throw new LogicException('A released compensation scope cannot be reused.'); + } + $id = hash('sha256', PortableDescriptor::encode([$identity->id(), $obligation->key])); + $existing = $database->table(self::OBLIGATIONS)->where('id', $id)->first(); + if ($existing !== null) { + if ($existing->fingerprint !== $obligation->fingerprint()) { + throw new LogicException('A compensation operation key was reused with different content.'); + } + return $id; + } + if ($scope['status'] !== 'open' || $scope['obligation_count'] >= self::MAX_OBLIGATIONS) { + throw new LogicException('The compensation scope does not admit new obligations.'); + } + $position = (int) $scope['obligation_count'] + 1; + $database->table(self::OBLIGATIONS)->insert([ + 'id' => $id, + 'scope_id' => $identity->id(), + 'position' => $position, + 'fingerprint' => $obligation->fingerprint(), + 'descriptor' => PortableDescriptor::encode($obligation->toArray()), + 'forward_status' => 'registered', + 'compensation_status' => 'pending', + ]); + $database->table(self::SCOPES)->where('id', $scope['id'])->update([ + 'obligation_count' => $position, + ]); + ScopeStorage::append($scope, 'registered', [ + 'obligationId' => $id, + 'fingerprint' => $obligation->fingerprint(), + ]); + return $id; + }); + } + + public function request( + ScopeIdentity $identity, + string $key, + string $principal, + string $reason, + ?string $policyVersion = null + ): void { + foreach ([ + 'key' => $key, + 'principal' => $principal, + 'reason' => $reason, + ] as $name => $value) { + PortableDescriptor::identifier($value, $name); + } + $descriptor = PortableDescriptor::encode([ + 'key' => $key, + 'principal' => $principal, + 'reason' => $reason, + ]); + $fingerprint = hash('sha256', $descriptor); + ScopeStorage::mutate($identity, static function (array $scope, Connection $database) use ( + $identity, + $key, + $descriptor, + $fingerprint, + $policyVersion + ): void { + $id = hash('sha256', PortableDescriptor::encode([$identity->id(), $key])); + $existing = $database->table('workflow_compensation_requests') + ->where('id', $id) + ->first(); + if ($existing !== null) { + if ($existing->fingerprint !== $fingerprint) { + throw new LogicException( + 'The compensation request identity was reused with different authority or content.' + ); + } + return; + } + if ($scope['status'] === 'released' || $scope['request_count'] >= 128) { + throw new LogicException('The compensation scope does not admit another request.'); + } + $position = (int) $scope['request_count'] + 1; + $database->table('workflow_compensation_requests') + ->insert([ + 'id' => $id, + 'scope_id' => $scope['id'], + 'position' => $position, + 'fingerprint' => $fingerprint, + 'descriptor' => $descriptor, + 'authorization_policy_version' => $policyVersion, + ]); + $database->table(self::SCOPES)->where('id', $scope['id'])->update([ + 'request_count' => $position, + 'status' => $scope['status'] === 'verified' ? 'verified' : 'requested', + ]); + ScopeStorage::append($scope, 'requested', [ + 'requestId' => $id, + 'fingerprint' => $fingerprint, + 'policyVersion' => $policyVersion, + ]); + }); + } + + public function primaryRequest(ScopeIdentity $identity): array + { + return ScopeStorage::mutate($identity, static function (array $scope, Connection $database): array { + $row = $database->table('workflow_compensation_requests') + ->where('scope_id', $scope['id'])->orderBy('position')->first(); + if ($row === null) { + throw new LogicException('The compensation scope has no authorized request.'); + } + return json_decode($row->descriptor, true, flags: JSON_THROW_ON_ERROR); + }); + } + + public function metadata(ScopeIdentity $identity): array + { + return ScopeStorage::mutate($identity, static fn (array $scope): array => $scope); + } + + public function inspect(ScopeIdentity $identity): array + { + return ScopeStorage::mutate($identity, static function (array $scope, Connection $database): array { + $scope['identity'] = json_decode($scope['identity'], true, flags: JSON_THROW_ON_ERROR); + $scope['obligations'] = $database->table(self::OBLIGATIONS)->where('scope_id', $scope['id']) + ->orderBy('position') + ->get() + ->map(static function (object $row): array { + $item = (array) $row; + $item['descriptor'] = json_decode($item['descriptor'], true, flags: JSON_THROW_ON_ERROR); + return $item; + })->all(); + return $scope; + }); + } + + public function journal(ScopeIdentity $identity): array + { + return ScopeStorage::mutate($identity, static fn (array $scope, Connection $database): array => + $database->table(self::JOURNAL)->where('scope_id', $scope['id'])->orderBy('sequence') + ->get() + ->map(static fn (object $row): array => (array) $row) + ->all()); + } +} diff --git a/src/V2/Compensation/Persistence/ScopeStorage.php b/src/V2/Compensation/Persistence/ScopeStorage.php new file mode 100644 index 000000000..3abea904b --- /dev/null +++ b/src/V2/Compensation/Persistence/ScopeStorage.php @@ -0,0 +1,74 @@ + $operation(self::locked($identity), $database), + admission: static function () use ($database, $identity): void { + $database->table(self::SCOPES)->where('id', $identity->id())->update([ + 'id' => $identity->id(), + ]); + }, + connection: $database + ); + } + + public static function locked(ScopeIdentity $identity): array + { + $row = ConfiguredV2Models::connection()->table(self::SCOPES)->where('id', $identity->id())->lockForUpdate() + ->first(); + if ($row === null || $row->identity !== PortableDescriptor::encode($identity->toArray())) { + throw new LogicException('The exact compensation scope generation is unavailable.'); + } + return (array) $row; + } + + public static function append(array $scope, string $kind, array $data): void + { + $database = ConfiguredV2Models::connection(); + $sequence = (int) $database->table(self::SCOPES)->where('id', $scope['id'])->value('event_sequence') + 1; + if ($sequence > 8192) { + throw new LogicException('The scope journal work limit is exhausted; unresolved work is retained.'); + } + $database->table(self::JOURNAL)->insert([ + 'scope_id' => $scope['id'], + 'sequence' => $sequence, + 'kind' => $kind, + 'data' => PortableDescriptor::encode($data), + 'recorded_at' => now('UTC'), + ]); + $database->table(self::SCOPES)->where('id', $scope['id'])->update([ + 'event_sequence' => $sequence, + ]); + } +} diff --git a/src/V2/Compensation/Recovery/ChildResolution.php b/src/V2/Compensation/Recovery/ChildResolution.php new file mode 100644 index 000000000..866a1a3a0 --- /dev/null +++ b/src/V2/Compensation/Recovery/ChildResolution.php @@ -0,0 +1,54 @@ +handlerVersion !== ChildScopes::VERSION || array_keys($obligation->input) !== [ + 'child', + ] || ! is_array($obligation->input['child'])) { + return Resolution::unknown('child-descriptor-incompatible'); + } + $child = ScopeIdentity::fromArray($obligation->input['child']); + if ($child->application !== $parent->application || $child->namespace !== $parent->namespace) { + return Resolution::unknown('child-owner-mismatch'); + } + $principal = $this->repository->primaryRequest($parent)['principal']; + try { + $this->authorization->assertAllowed($child, $principal, 'compensate'); + $this->recovery->request($child, 'parent-' . $operationId, $principal, 'parent-compensation'); + $snapshot = $this->repository->inspect($child); + } catch (Throwable) { + return Resolution::unknown('child-unresolved'); + } + return $snapshot['status'] === 'verified' + ? Resolution::verified(['scope-verified:' . $child->id()], true) + : Resolution::unknown('child-unresolved'); + } +} diff --git a/src/V2/Compensation/Recovery/RecoveryWorkflow.php b/src/V2/Compensation/Recovery/RecoveryWorkflow.php new file mode 100644 index 000000000..1180fee51 --- /dev/null +++ b/src/V2/Compensation/Recovery/RecoveryWorkflow.php @@ -0,0 +1,36 @@ +addCompensation(static fn () => self::activity(CompensateObligationActivity::class, $scope, $id)); + } + $this->compensate(); + return self::activity(VerifyScopeActivity::class, $scope); + } +} diff --git a/src/V2/Compensation/Recovery/ScopeRecovery.php b/src/V2/Compensation/Recovery/ScopeRecovery.php new file mode 100644 index 000000000..ff3d44041 --- /dev/null +++ b/src/V2/Compensation/Recovery/ScopeRecovery.php @@ -0,0 +1,125 @@ +authorization->assertAllowed($scope, $principal, 'request'); + PortableDescriptor::identifier($policyVersion, 'authorization policy version'); + $snapshot = $this->repository->inspect($scope); + if ($snapshot['status'] === 'released') { + throw new LogicException('The retained compensation scope has been released.'); + } + $origin = ConfiguredV2Models::query('run_model', WorkflowRun::class)->find($snapshot['original_run_id']); + if (! $origin instanceof WorkflowRun || $origin->namespace !== $scope->namespace) { + throw new LogicException('The original compensation run is unavailable in the exact namespace.'); + } + $this->repository->request($scope, $requestKey, $principal, $reason, $policyVersion); + if ($snapshot['status'] === 'verified') { + return [ + 'route' => 'verified', + 'runId' => null, + ]; + } + if (! $origin->status->isTerminal()) { + $command = WorkflowStub::loadRun($origin->id)->attemptRequestCancellation($reason); + if ($command->accepted()) { + return [ + 'route' => 'origin', + 'runId' => $origin->id, + 'commandId' => $command->commandId(), + ]; + } + // Closure can win after the gate transaction. Route that same retained request below. + $origin->refresh(); + if (! $origin->status->isTerminal()) { + throw new LogicException('The originating run refused cooperative compensation.'); + } + } + $run = ConfiguredV2Models::query('run_model', WorkflowRun::class) + ->where('workflow_instance_id', $scope->recoveryInstanceId()) + ->where('namespace', $scope->namespace) + ->orderByDesc('run_number') + ->first(); + if (! $run instanceof WorkflowRun || in_array( + $run->status, + [RunStatus::Cancelled, RunStatus::Terminated], + true + )) { + $result = WorkflowStub::make(RecoveryWorkflow::class, $scope->recoveryInstanceId(), $scope->namespace) + ->attemptSignalWithStart('recover', [], StartOptions::returnExistingActive(), $scope->toArray()); + $runId = $result->runId(); + if ($result->accepted() && is_string($runId)) { + return [ + 'route' => 'recovery', + 'runId' => $runId, + ]; + } + $run = ConfiguredV2Models::query('run_model', WorkflowRun::class) + ->where('workflow_instance_id', $scope->recoveryInstanceId()) + ->where('namespace', $scope->namespace) + ->orderByDesc('run_number') + ->first(); + } + if (! $run instanceof WorkflowRun || $run->workflow_type !== TypeRegistry::for(RecoveryWorkflow::class)) { + throw new LogicException('The canonical recovery identity is unavailable or reserved by another type.'); + } + if (in_array($run->status, [RunStatus::Pending, RunStatus::Running, RunStatus::Waiting], true)) { + return [ + 'route' => 'recovery', + 'runId' => $run->id, + ]; + } + if ($this->repository->inspect($scope)['status'] === 'verified') { + return [ + 'route' => 'verified', + 'runId' => $run->id, + ]; + } + if ($run->status !== RunStatus::Failed) { + throw new LogicException('The recovery is terminal without verified compensation.'); + } + $redrive = $this->runtime->redrive($scope->recoveryInstanceId(), $run->id, [ + 'namespace' => $scope->namespace, + 'request_id' => hash('sha256', PortableDescriptor::encode([$scope->id(), $requestKey, $run->id])), + ]); + if (($redrive['accepted'] ?? false) !== true) { + throw new LogicException( + 'Native DW redrive refused unresolved recovery: ' . ($redrive['reason'] ?? 'unknown') . '.' + ); + } + return [ + 'route' => 'recovery', + 'runId' => $redrive['workflow_run_id'], + ]; + } +} diff --git a/src/V2/Compensation/Recovery/ScopeRetention.php b/src/V2/Compensation/Recovery/ScopeRetention.php new file mode 100644 index 000000000..8715b9698 --- /dev/null +++ b/src/V2/Compensation/Recovery/ScopeRetention.php @@ -0,0 +1,94 @@ +authorization->assertAllowed($scope, $principal, 'release'); + PortableDescriptor::identifier($policy, 'authorization policy version'); + $snapshot = $this->repository->inspect($scope); + SqliteTransaction::run(static function () use ($scope, $principal, $policy, $snapshot): void { + $origin = ConfiguredV2Models::query('run_model', WorkflowRun::class)->whereKey( + $snapshot['original_run_id'] + )->lockForUpdate() + ->first(); + $recovery = ConfiguredV2Models::query('run_model', WorkflowRun::class)->where( + 'workflow_instance_id', + $scope->recoveryInstanceId() + ) + ->orderByDesc('run_number') + ->lockForUpdate() + ->first(); + if (! $origin instanceof WorkflowRun || $origin->namespace !== $scope->namespace || ! $origin->status->isTerminal() + || ($recovery instanceof WorkflowRun && ! $recovery->status->isTerminal())) { + throw new LogicException('Native execution still retains this compensation scope.'); + } + $database = ConfiguredV2Models::connection(); + $link = $database->table('workflow_compensation_children') + ->where('child_id', $scope->id()) + ->first(); + if ($link !== null) { + $parent = $database->table(ScopeStorage::SCOPES)->where('id', $link->parent_id)->lockForUpdate() + ->first(); + if ($parent === null || ! in_array($parent->status, ['verified', 'released'], true)) { + throw new LogicException('An unresolved parent retains this child scope.'); + } + } + ScopeStorage::mutate($scope, static function (array $locked) use ( + $scope, + $principal, + $policy, + $database + ): void { + if ($locked['status'] === 'released') { + return; + } + if ($locked['status'] !== 'verified') { + throw new LogicException('Unresolved compensation cannot be released.'); + } + foreach ($database->table(ScopeStorage::OBLIGATIONS)->where('scope_id', $scope->id())->get() as $row) { + $descriptor = json_decode($row->descriptor, true, flags: JSON_THROW_ON_ERROR); + unset($descriptor['input']); + $descriptor['released'] = true; + $database->table(ScopeStorage::OBLIGATIONS)->where('id', $row->id)->update([ + 'descriptor' => PortableDescriptor::encode($descriptor), + 'forward_result' => null, + ]); + } + $database->table(ScopeStorage::SCOPES)->where('id', $scope->id())->update([ + 'status' => 'released', + ]); + ScopeStorage::append($locked, 'released', [ + 'principal' => $principal, + 'policyVersion' => $policy, + 'reason' => 'explicit-release', + ]); + }); + }, connection: ConfiguredV2Models::connection()); + } +} diff --git a/src/V2/Compensation/Security/ScopeAuthorization.php b/src/V2/Compensation/Security/ScopeAuthorization.php new file mode 100644 index 000000000..ade52d6c6 --- /dev/null +++ b/src/V2/Compensation/Security/ScopeAuthorization.php @@ -0,0 +1,22 @@ +taskId === null || $activity->attemptId() === null || $activity->run->namespace !== $scope->namespace) { + throw new NonRetryableException('scope_native_identity_mismatch'); + } + $fromRequest = $principal === null; + if ($fromRequest) { + $principal = $this->repository->primaryRequest($scope)['principal']; + } + if ($principal === null) { + throw new NonRetryableException('scope_principal_missing'); + } + // Host policy and input callbacks may perform I/O. Capture their decision before native SQL locks. + try { + $policyVersion = $this->authorization->assertAllowed($scope, $principal, $operation); + PortableDescriptor::identifier($policyVersion, 'authorization policy version'); + } catch (\Throwable) { + throw new NonRetryableException('scope_authorization_refused'); + } + $prepared = $prepare?->__invoke(); + return SqliteTransaction::run(function () use ( + $activity, + $scope, + $operation, + $principal, + $fromRequest, + $policyVersion, + $prepared, + $body + ): mixed { + $this->assertNativeAuthority($activity); + if ($operation !== 'register') { + $snapshot = $this->repository->metadata($scope); + $origin = $snapshot['original_run_id'] === $activity->runId(); + $recovery = $activity->workflowId() === $scope->recoveryInstanceId() + && $activity->run->workflow_class === RecoveryWorkflow::class; + if ((! $origin && ! $recovery) || ($operation === 'forward' && ! $origin)) { + throw new NonRetryableException('scope_execution_owner_mismatch'); + } + if ($fromRequest && $this->repository->primaryRequest($scope)['principal'] !== $principal) { + throw new NonRetryableException('scope_principal_mismatch'); + } + } + $snapshot = $operation === 'register' ? null : $this->repository->inspect($scope); + $result = $body($snapshot, $policyVersion, $prepared); + $this->assertNativeAuthority($activity); + ScopeStorage::append(ScopeStorage::locked($scope), 'authorized', [ + 'principal' => $principal, + 'operation' => $operation, + 'policyVersion' => $policyVersion, + 'runId' => $activity->runId(), + 'activityId' => $activity->activityId(), + 'attemptId' => $activity->attemptId(), + ]); + return $result; + }, attempts: 3); + } + + private function assertNativeAuthority(Activity $activity): void + { + $status = app(DefaultActivityTaskBridge::class)->status($activity->attemptId()); + if (($status['can_continue'] ?? false) !== true + || ($status['workflow_run_id'] ?? null) !== $activity->runId() + || ($status['workflow_task_id'] ?? null) !== $activity->taskId + || ($status['activity_execution_id'] ?? null) !== $activity->activityId()) { + throw new NonRetryableException('scope_native_attempt_unavailable'); + } + } +} diff --git a/src/V2/Support/ActivityRetryHistory.php b/src/V2/Support/ActivityRetryHistory.php new file mode 100644 index 000000000..e8d045deb --- /dev/null +++ b/src/V2/Support/ActivityRetryHistory.php @@ -0,0 +1,85 @@ + $events + */ + public static function valid(array $events): bool + { + $hasRetries = false; + foreach ($events as $event) { + $hasRetries = $hasRetries || $event->event_type === HistoryEventType::ActivityRetryScheduled; + } + if (! $hasRetries) { + return true; + } + $execution = null; + $attempt = null; + $task = null; + $number = 0; + $nextTask = null; + $terminal = false; + $seenAttempts = []; + $seenTasks = []; + foreach ($events as $event) { + $payload = $event->payload; + $id = $payload['activity_execution_id'] ?? null; + if (! is_string($id) || $id === '' || ($execution !== null && $id !== $execution) || $terminal) { + return false; + } + $execution = $id; + if ($event->event_type === HistoryEventType::ActivityStarted) { + $currentAttempt = $payload['activity_attempt_id'] ?? null; + $currentTask = $event->workflow_task_id ?? ($payload['workflow_task_id'] ?? null); + if (isset($event->workflow_task_id, $payload['workflow_task_id']) && $event->workflow_task_id !== $payload['workflow_task_id']) { + return false; + } + if (! is_string($currentAttempt) || $currentAttempt === '' || isset($seenAttempts[$currentAttempt]) + || ! is_string($currentTask) || $currentTask === '' || isset($seenTasks[$currentTask]) + || ($payload['attempt_number'] ?? null) !== $number + 1 + || ($number > 0 && $nextTask !== $currentTask)) { + return false; + } + $attempt = $currentAttempt; + $task = $currentTask; + $seenAttempts[$attempt] = true; + $seenTasks[$task] = true; + ++$number; + $nextTask = null; + } elseif ($event->event_type === HistoryEventType::ActivityRetryScheduled) { + $retry = $payload['retry_task_id'] ?? null; + $maximum = $payload['max_attempts'] ?? null; + if ($attempt === null || $nextTask !== null + || ($payload['activity_attempt_id'] ?? null) !== $attempt + || ($payload['retry_after_attempt_id'] ?? null) !== $attempt + || ($payload['retry_after_attempt'] ?? null) !== $number + || ($payload['retry_of_task_id'] ?? null) !== $task + || ! is_string($retry) || $retry === '' || isset($seenTasks[$retry]) + || ($maximum !== null && (! is_int($maximum) || $maximum <= $number))) { + return false; + } + $nextTask = $retry; + } elseif (in_array( + $event->event_type, + [HistoryEventType::ActivityCompleted, HistoryEventType::ActivityFailed], + true + )) { + if ($nextTask !== null || $attempt === null || ($payload['activity_attempt_id'] ?? null) !== $attempt + || ($payload['attempt_number'] ?? null) !== $number) { + return false; + } + $terminal = true; + } + } + return $terminal; + } +} diff --git a/src/V2/Support/FailedRunRedrivePlan.php b/src/V2/Support/FailedRunRedrivePlan.php index ea39c75c8..67037ecf0 100644 --- a/src/V2/Support/FailedRunRedrivePlan.php +++ b/src/V2/Support/FailedRunRedrivePlan.php @@ -7,6 +7,9 @@ use Illuminate\Support\Str; use Throwable; use Workflow\Serializers\Serializer; +use Workflow\V2\Enums\CommandOutcome; +use Workflow\V2\Enums\CommandStatus; +use Workflow\V2\Enums\CommandType; use Workflow\V2\Enums\HistoryEventType; use Workflow\V2\Enums\RunStatus; use Workflow\V2\Models\WorkflowHistoryEvent; @@ -86,7 +89,7 @@ public static function forRun(WorkflowRun $run): array } if ($event->event_type === HistoryEventType::StartAccepted) { - if ($started !== null) { + if ($started !== null && ! self::isActiveStartAdoption($run, $event)) { return self::reject('unexpected_start'); } @@ -199,6 +202,7 @@ public static function forRun(WorkflowRun $run): array if (! in_array($event->event_type, [ HistoryEventType::ActivityScheduled, HistoryEventType::ActivityStarted, + HistoryEventType::ActivityRetryScheduled, HistoryEventType::ActivityHeartbeatRecorded, HistoryEventType::ActivityCompleted, HistoryEventType::ActivityFailed, @@ -381,6 +385,10 @@ public static function forRun(WorkflowRun $run): array return self::reject('activity_sequence_gap'); } + if (! ActivityRetryHistory::valid($stepEvents)) { + return self::reject('invalid_activity_retry_history'); + } + $activityType = null; $completion = null; $failureCount = 0; @@ -612,6 +620,36 @@ public static function successorEvents(array $plan, string $sourceRunId): array return $events; } + /** + * Native start/adopt records a command receipt, not another workflow start or reusable step. + */ + private static function isActiveStartAdoption(WorkflowRun $run, WorkflowHistoryEvent $event): bool + { + $payload = $event->payload; + $command = $payload['command'] ?? null; + if (! is_array($command) || ($payload['outcome'] ?? null) !== CommandOutcome::ReturnedExistingActive->value + || ($payload['rejection_reason'] ?? null) !== null + || $event->workflow_run_id !== $run->id) { + return false; + } + foreach (['workflow_instance_id', 'workflow_run_id', 'workflow_class', 'workflow_type'] as $key) { + $expected = $key === 'workflow_run_id' ? $run->id : $run->getAttribute($key); + if (! is_string($expected) || $expected === '' || ($payload[$key] ?? null) !== $expected) { + return false; + } + } + $commandId = $payload['workflow_command_id'] ?? null; + return is_string($commandId) && $commandId !== '' + && $event->workflow_command_id === $commandId + && ($command['id'] ?? null) === $commandId + && ($command['type'] ?? null) === CommandType::Start->value + && ($command['target_scope'] ?? null) === 'instance' + && ($command['resolved_run_id'] ?? null) === $run->id + && ($command['status'] ?? null) === CommandStatus::Accepted->value + && ($command['outcome'] ?? null) === CommandOutcome::ReturnedExistingActive->value + && ($command['rejection_reason'] ?? null) === null; + } + /** * Who may have recorded a reused-step payload in this run's history: the * run itself, or, for a step it reused, its immediate redrive source while diff --git a/src/migrations/2026_11_11_000100_create_compensation_scopes.php b/src/migrations/2026_11_11_000100_create_compensation_scopes.php new file mode 100644 index 000000000..373f1d7aa --- /dev/null +++ b/src/migrations/2026_11_11_000100_create_compensation_scopes.php @@ -0,0 +1,91 @@ +string('id', 64) + ->primary(); + $table->text('identity'); + $table->string('original_run_id', 128) + ->index(); + $table->string('status', 32); + $table->unsignedInteger('obligation_count') + ->default(0); + $table->unsignedInteger('request_count') + ->default(0); + $table->unsignedInteger('event_sequence') + ->default(0); + }); + Schema::create('workflow_compensation_obligations', static function (Blueprint $table): void { + $table->string('id', 64) + ->primary(); + $table->string('scope_id', 64) + ->index(); + $table->unsignedInteger('position'); + $table->string('fingerprint', 64); + $table->text('descriptor'); + $table->string('forward_status', 32); + $table->string('compensation_status', 32); + $table->string('forward_activity_id', 26) + ->nullable(); + $table->string('forward_attempt_id', 26) + ->nullable(); + $table->string('compensation_attempt_id', 26) + ->nullable(); + $table->text('forward_result') + ->nullable(); + $table->text('evidence') + ->nullable(); + $table->unique(['scope_id', 'position'], 'compensation_obligation_position'); + }); + Schema::create('workflow_compensation_graphs', static function (Blueprint $table): void { + $table->string('id', 64) + ->primary(); + }); + Schema::create('workflow_compensation_children', static function (Blueprint $table): void { + $table->string('child_id', 64) + ->primary(); + $table->string('parent_id', 64) + ->index(); + $table->string('obligation_id', 64) + ->unique(); + }); + Schema::create('workflow_compensation_requests', static function (Blueprint $table): void { + $table->string('id', 64) + ->primary(); + $table->string('scope_id', 64) + ->index(); + $table->unsignedInteger('position'); + $table->string('fingerprint', 64); + $table->text('descriptor'); + $table->string('authorization_policy_version', 128) + ->nullable(); + $table->unique(['scope_id', 'position'], 'compensation_request_position'); + }); + Schema::create('workflow_compensation_journal', static function (Blueprint $table): void { + $table->string('scope_id', 64); + $table->unsignedInteger('sequence'); + $table->string('kind', 32); + $table->text('data'); + $table->timestamp('recorded_at', 6); + $table->primary(['scope_id', 'sequence'], 'compensation_journal_sequence'); + }); + } + + public function down(): void + { + Schema::dropIfExists('workflow_compensation_journal'); + Schema::dropIfExists('workflow_compensation_obligations'); + Schema::dropIfExists('workflow_compensation_requests'); + Schema::dropIfExists('workflow_compensation_children'); + Schema::dropIfExists('workflow_compensation_graphs'); + Schema::dropIfExists('workflow_compensation_scopes'); + } +}; diff --git a/tests/Feature/V2/V2CompensationForwardAdmissionTest.php b/tests/Feature/V2/V2CompensationForwardAdmissionTest.php new file mode 100644 index 000000000..8bcc54f42 --- /dev/null +++ b/tests/Feature/V2/V2CompensationForwardAdmissionTest.php @@ -0,0 +1,237 @@ + ['default', 'client-a'], + 'queue.default' => 'sync', + 'queue.connections.sync.driver' => 'sync', + ]); + $this->registry($this->handler()); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== 'client-a' || $principal !== 'operator-a') { + throw new \LogicException('Scope access refused.'); + } + return 'test-policy-1'; + } + }); + } + + protected function tearDown(): void + { + \Illuminate\Support\Carbon::setTestNow(); + parent::tearDown(); + } + + public function testMissingPinnedForwardHandlerIsNonRetryableBeforeDispatch(): void + { + $this->replaceAfterRegistration(new HandlerRegistry([])); + [$scope, $workflow] = $this->origin(); + $this->assertRefusedBeforeDispatch($scope, $workflow, 'scope_handler_unavailable'); + } + + public function testIncompatibleForwardInputIsNonRetryableBeforeDispatch(): void + { + $this->replaceAfterRegistration(new HandlerRegistry([ + 'test.operation' => [ + '1' => $this->handler(false), + ], + ])); + [$scope, $workflow] = $this->origin(); + $this->assertRefusedBeforeDispatch($scope, $workflow, 'scope_input_incompatible'); + } + + public function testForwardRetryRefusalPreservesEarlierUnknownEffect(): void + { + [$scope, $workflow] = $this->origin('lost-ack'); + $before = (new ScopeRepository())->inspect($scope)['obligations'][0]; + $this->assertSame('unknown', $before['forward_status']); + $this->assertSame(1, (int) Cache::get('test:scope:provider-call-count')); + $this->registry(null); + $this->driveNativeRetries($workflow->runId()); + $this->assertForwardFailure($workflow, 'scope_handler_unavailable', 2); + $after = (new ScopeRepository())->inspect($scope)['obligations'][0]; + $this->assertSame('unknown', $after['forward_status']); + $this->assertSame($before['forward_attempt_id'], $after['forward_attempt_id']); + $this->assertSame($before['forward_result'], $after['forward_result']); + $this->assertSame('pending', $after['compensation_status']); + $this->assertSame(1, (int) Cache::get('test:scope:provider-call-count')); + $this->registry($this->handler()); + app(ScopeRecovery::class)->request($scope, 'recover', 'operator-a', 'cancel'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(1, (int) Cache::get('test:scope:compensation-call-count')); + } + + public function testNonRetryableProviderExceptionAfterDispatchRemainsUnknownAndRetryable(): void + { + $this->registry($this->handler(throwAfterEffect: true)); + [$scope, $workflow] = $this->origin(); + $this->driveNativeRetries($workflow->runId()); + $this->assertForwardFailure($workflow, 'scope_forward_unknown', 3, false); + $row = (new ScopeRepository())->inspect($scope)['obligations'][0]; + $this->assertSame('unknown', $row['forward_status']); + $this->assertSame( + 'provider-outcome-unknown', + json_decode($row['forward_result'], true, flags: JSON_THROW_ON_ERROR)['reason'] + ); + $this->assertSame('pending', $row['compensation_status']); + $this->assertSame(3, (int) Cache::get('test:scope:provider-call-count')); + $this->assertSame(1, (int) Cache::get('test:scope:effect-count')); + } + + private function assertRefusedBeforeDispatch(ScopeIdentity $scope, WorkflowStub $workflow, string $reason): void + { + $this->assertForwardFailure($workflow, $reason, 1); + $snapshot = (new ScopeRepository())->inspect($scope); + $row = $snapshot['obligations'][0]; + $this->assertSame('open', $snapshot['status']); + $this->assertSame('registered', $row['forward_status']); + $this->assertNull($row['forward_activity_id']); + $this->assertNull($row['forward_attempt_id']); + $this->assertNull($row['forward_result']); + $this->assertSame('pending', $row['compensation_status']); + $this->assertNull(Cache::get('test:scope:provider-call-count')); + $this->assertNull(Cache::get('test:scope:effect-count')); + $this->registry($this->handler()); + app(ScopeRecovery::class)->request($scope, 'recover', 'operator-a', 'cancel'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(Cache::get('test:scope:compensation-call-count')); + } + + private function assertForwardFailure( + WorkflowStub $workflow, + string $reason, + int $attempts, + bool $nonRetryable = true + ): void { + $this->assertSame('failed', $workflow->refresh()->status()); + $execution = ActivityExecution::query()->where('workflow_run_id', $workflow->runId()) + ->where('activity_class', ForwardOperationActivity::class)->firstOrFail(); + $this->assertSame($attempts, $execution->attempts()->count()); + $failure = WorkflowHistoryEvent::query()->where('workflow_run_id', $workflow->runId()) + ->where('event_type', HistoryEventType::ActivityFailed)->firstOrFail(); + $this->assertSame($reason, $failure->payload['exception']['message']); + $this->assertSame($nonRetryable, $failure->payload['non_retryable']); + } + + private function replaceAfterRegistration(HandlerRegistry $replacement): void + { + $replaced = false; + ConfiguredV2Models::connection()->listen(function (QueryExecuted $query) use ($replacement, &$replaced): void { + if (! $replaced && str_starts_with($query->sql, 'insert') + && str_contains($query->sql, 'workflow_compensation_obligations')) { + $replaced = true; + $this->app->instance(HandlerRegistry::class, $replacement); + } + }); + } + + private function registry(?ScopedOperation $handler): void + { + $this->app->instance(HandlerRegistry::class, new HandlerRegistry( + $handler === null ? [] : [ + 'test.operation' => [ + '1' => $handler, + ], + ] + )); + } + + private function handler(bool $compatible = true, bool $throwAfterEffect = false): ScopedOperation + { + return new class($compatible, $throwAfterEffect) implements ScopedOperation { + public function __construct( + private bool $compatible, + private bool $throwAfterEffect + ) { + } + + public function assertInput(array $input): void + { + if (! $this->compatible) { + throw new \InvalidArgumentException('The input schema changed after registration.'); + } + } + + public function execute(OperationContext $context): ForwardResult + { + Cache::increment('test:scope:provider-call-count'); + $result = (new TestScopedOperation())->execute($context); + if ($this->throwAfterEffect) { + throw new NonRetryableException('provider-acknowledgement-lost'); + } + return $result; + } + + public function compensate(OperationContext $context): Resolution + { + Cache::increment('test:scope:compensation-call-count'); + return (new TestScopedOperation())->compensate($context); + } + }; + } + + private function origin(string $mode = 'complete'): array + { + WorkflowStub::fake(); + $scope = new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + $workflow = WorkflowStub::make(TestForwardAdmissionWorkflow::class, 'forward-admission-origin', 'client-a'); + $workflow->start($scope->toArray(), $mode); + return [$scope, $workflow]; + } + + private function driveNativeRetries(string $runId): void + { + for ($step = 0; $step < 20; $step++) { + if (WorkflowRun::query()->findOrFail($runId)->status->isTerminal()) { + return; + } + $task = WorkflowTask::query()->where('workflow_run_id', $runId)->where('status', 'ready') + ->orderBy('available_at') + ->first(); + $this->assertNotNull($task, 'A nonterminal forward run must retain a native retry task.'); + if ($task->available_at !== null && $task->available_at->isFuture()) { + \Illuminate\Support\Carbon::setTestNow($task->available_at->copy()->addMicrosecond()); + } + if ($task->task_type->value === 'activity') { + $this->app->call([new \Workflow\V2\Jobs\RunActivityTask($task->id), 'handle']); + } else { + $this->app->call([new \Workflow\V2\Jobs\RunWorkflowTask($task->id), 'handle']); + } + } + $this->fail('Native forward retries did not terminate.'); + } +} diff --git a/tests/Feature/V2/V2CompensationScopeConcurrencyTest.php b/tests/Feature/V2/V2CompensationScopeConcurrencyTest.php new file mode 100644 index 000000000..60e67d697 --- /dev/null +++ b/tests/Feature/V2/V2CompensationScopeConcurrencyTest.php @@ -0,0 +1,288 @@ +getDriverName(), ['pgsql', 'mysql'], true) || ! function_exists( + 'pcntl_fork' + )) { + $this->markTestSkipped('Separate processes and PostgreSQL or InnoDB row-lock observation are required.'); + } + self::stopWorkers(); + Queue::fake(); + config([ + 'workflows.v2.namespaces' => ['default', 'client-a'], + 'workflows.v2.task_dispatch_mode' => 'poll', + ]); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== 'client-a' || $principal !== 'operator-a') { + throw new RuntimeException('Scope access refused.'); + } + return 'test-policy-1'; + } + }); + } + + public function testConcurrentRequestsStartOneCanonicalRecovery(): void + { + $scope = $this->openOrigin(); + $results = $this->race( + static fn (int $index): array => app( + ScopeRecovery::class + )->request($scope, 'request-' . $index, 'operator-a', 'cancel') + ); + $this->assertSame($results[0]['runId'], $results[1]['runId']); + $this->assertSame( + 1, + WorkflowRun::query()->where('workflow_instance_id', $scope->recoveryInstanceId())->count() + ); + $this->assertSame( + 2, + ConfiguredV2Models::connection()->table('workflow_compensation_requests')->where( + 'scope_id', + $scope->id() + )->count() + ); + } + + public function testRequestSerializesWithRegistrationAndClosesNewAdmission(): void + { + $scope = $this->openOrigin(); + $this->race(static function (int $index) use ($scope): array { + if ($index === 0) { + return [ + 'id' => (new ScopeRepository())->register( + $scope, + new Obligation('book', 'carrier.cancel', '1', []) + ), + ]; + } + return app(ScopeRecovery::class)->request($scope, 'request', 'operator-a', 'cancel'); + }); + $this->assertCount(1, (new ScopeRepository())->inspect($scope)['obligations']); + $this->expectException(\LogicException::class); + (new ScopeRepository())->register($scope, new Obligation('later', 'carrier.cancel', '1', [])); + } + + public function testConcurrentParticipationUsesTheExistingNamespaceGraphWithoutLockUpgrade(): void + { + $parents = []; + $children = []; + $repository = new ScopeRepository(); + foreach ([0, 1, 2] as $index) { + $parents[$index] = new ScopeIdentity('shop', 'client-a', 'parent-' . $index, '1'); + $children[$index] = new ScopeIdentity('shop', 'client-a', 'child-' . $index, '1'); + $repository->open($parents[$index], 'run-' . $index); + } + (new ChildScopes())->register($parents[2], 'run-2', 'delivery', $children[2]); + $results = $this->race(static fn (int $index): array => [ + 'id' => (new ChildScopes())->register($parents[$index], 'run-' . $index, 'delivery', $children[$index]), + ]); + $this->assertNotSame($results[0]['id'], $results[1]['id']); + $this->assertSame(3, DB::table('workflow_compensation_children')->count()); + foreach ([0, 1] as $index) { + $this->assertCount(1, $repository->inspect($parents[$index])['obligations']); + } + } + + public function testCompetingEdgesCannotCreateACycle(): void + { + $a = new ScopeIdentity('shop', 'client-a', 'a', '1'); + $b = new ScopeIdentity('shop', 'client-a', 'b', '1'); + $results = $this->race(static function (int $index) use ($a, $b): array { + try { + $id = (new ChildScopes())->register( + $index === 0 ? $a : $b, + $index === 0 ? 'run-a' : 'run-b', + 'child', + $index === 0 ? $b : $a + ); + return [ + 'accepted' => true, + 'id' => $id, + ]; + } catch (\LogicException) { + return [ + 'accepted' => false, + ]; + } + }); + $this->assertTrue($results[0]['accepted']); + $this->assertFalse($results[1]['accepted']); + $this->assertSame(1, DB::table('workflow_compensation_children')->count()); + } + + private function openOrigin(): ScopeIdentity + { + $scope = new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + $origin = WorkflowStub::make(TestSignalWorkflow::class, 'origin', 'client-a'); + $origin->start(); + WorkflowRun::query()->findOrFail($origin->runId())->forceFill([ + 'status' => 'completed', + 'closed_at' => now(), + ])->save(); + (new ScopeRepository())->open($scope, $origin->runId()); + return $scope; + } + + private function race(Closure $operation, bool $expectLock = true): array + { + $root = sys_get_temp_dir() . '/dw-conflict-' . bin2hex(random_bytes(6)); + mkdir($root); + DB::purge(); + $children = []; + $results = []; + try { + for ($index = 0; $index < 2; $index++) { + $pid = pcntl_fork(); + if ($pid === -1) { + throw new RuntimeException('Cannot fork the conflict probe.'); + } + if ($pid === 0) { + try { + DB::reconnect(); + if (DB::connection()->getDriverName() === 'pgsql') { + DB::statement("SET application_name = 'dw-conflict-probe-{$index}'"); + DB::statement("SET lock_timeout = '15s'"); + } else { + DB::statement('SET innodb_lock_wait_timeout = 15'); + } + if ($index === 0) { + DB::beginTransaction(); + $result = $operation($index); + touch($root . '/holding'); + self::waitFile($root . '/release'); + DB::commit(); + } else { + self::waitFile($root . '/holding'); + if (DB::connection()->getDriverName() === 'mysql') { + file_put_contents($root . '/connection', (string) DB::scalar('SELECT CONNECTION_ID()')); + } + $result = $operation($index); + } + file_put_contents($root . '/' . $index . '.json', json_encode([ + 'result' => $result, + ], JSON_THROW_ON_ERROR)); + DB::disconnect(); + exit(0); + } catch (Throwable $error) { + file_put_contents($root . '/' . $index . '.json', json_encode([ + 'error' => $error::class . ': ' . $error->getMessage(), + ])); + exit(1); + } + } + $children[] = $pid; + } + DB::reconnect(); + self::waitFile($root . '/holding'); + if ($expectLock) { + $deadline = microtime(true) + 10; + do { + $state = DB::connection()->getDriverName() === 'pgsql' + ? DB::selectOne( + "SELECT wait_event_type FROM pg_stat_activity WHERE application_name = 'dw-conflict-probe-1'" + ) + : $this->mysqlLockQuery($root); + if (($state->wait_event_type ?? null) === 'Lock') { + break; + } + if (is_file($root . '/1.json')) { + throw new RuntimeException( + 'Second caller finished without reaching the expected row lock: ' . file_get_contents( + $root . '/1.json' + ) + ); + } + usleep(10000); + } while (microtime(true) < $deadline); + self::assertSame( + 'Lock', + $state->wait_event_type ?? null, + 'The calls must overlap at a real database row lock.' + ); + } else { + self::waitFile($root . '/1.json'); + } + touch($root . '/release'); + foreach ($children as $index => $pid) { + pcntl_waitpid($pid, $status); + $payload = json_decode( + file_get_contents($root . '/' . $index . '.json'), + true, + flags: JSON_THROW_ON_ERROR + ); + self::assertSame(0, pcntl_wexitstatus($status), $payload['error'] ?? 'Child failed.'); + $results[] = $payload['result']; + } + return $results; + } finally { + touch($root . '/release'); + foreach ($children as $pid) { + if (pcntl_waitpid($pid, $status, WNOHANG) === 0) { + posix_kill($pid, SIGKILL); + pcntl_waitpid($pid, $status); + } + } + foreach (glob($root . '/*') as $file) { + unlink($file); + } + rmdir($root); + } + } + + private function mysqlLockQuery(string $root): ?object + { + if (! is_file($root . '/connection')) { + return null; + } + $id = (int) file_get_contents($root . '/connection'); + // SHOW PROCESSLIST exposes this test user's sessions without global PROCESS privileges. + foreach (DB::select('SHOW FULL PROCESSLIST') as $process) { + if ((int) $process->Id === $id && is_string($process->Info) + && str_contains($process->Info, 'workflow_compensation_')) { + // Caller 0 retains the conflicting row lock until the parent releases the barrier. + return (object) [ + 'wait_event_type' => 'Lock', + ]; + } + } + return null; + } + + private static function waitFile(string $file): void + { + $deadline = microtime(true) + 10; + while (! is_file($file)) { + if (microtime(true) >= $deadline) { + throw new RuntimeException('Probe barrier timed out.'); + } + usleep(10000); + } + } +} diff --git a/tests/Feature/V2/V2CompensationScopeRunBoundaryTest.php b/tests/Feature/V2/V2CompensationScopeRunBoundaryTest.php new file mode 100644 index 000000000..22ed6d949 --- /dev/null +++ b/tests/Feature/V2/V2CompensationScopeRunBoundaryTest.php @@ -0,0 +1,211 @@ + ['default', 'client-a'], + 'queue.default' => 'redis', + 'queue.connections.redis.driver' => 'redis', + ]); + Queue::fake(); + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([ + 'test.operation' => [ + '1' => new TestScopedOperation(), + ], + ])); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== 'client-a' || $principal !== 'operator-a') { + throw new \LogicException('Scope access refused.'); + } + return 'boundary-policy-1'; + } + }); + } + + protected function tearDown(): void + { + Carbon::setTestNow(); + parent::tearDown(); + } + + public function testContinueAsNewHandsOffTheOpenChildWithoutTransferringScopeOwnership(): void + { + $scope = new ScopeIdentity('shop', 'client-a', 'continued-order', '1'); + $child = new ScopeIdentity('shop', 'client-a', 'continued-shipment', '1'); + $stub = WorkflowStub::make(TestScopeRunBoundaryWorkflow::class, 'scope-continue', 'client-a'); + $stub->start($scope->toArray(), 'continue', 0, $child->toArray()); + $this->drainTasks(); + $runs = WorkflowRun::query()->where('workflow_instance_id', 'scope-continue')->orderBy('run_number')->get(); + $this->assertCount(2, $runs); + [$origin, $continued] = [$runs[0], $runs[1]]; + $this->assertSame('continued', $origin->closed_reason); + $this->assertSame(RunStatus::Waiting, $continued->status); + $originHistory = $this->history($origin->id); + $this->assertSame(HistoryEventType::WorkflowContinuedAsNew->value, end($originHistory)['event_type']); + $link = WorkflowLink::query()->where('parent_workflow_run_id', $origin->id)->where( + 'link_type', + 'child_workflow' + )->sole(); + $handoff = WorkflowChildSupervision::query()->where('supervising_workflow_run_id', $continued->id)->sole(); + $this->assertSame($link->id, $handoff->workflow_link_id); + $this->assertSame($origin->id, $handoff->parent_workflow_run_id); + $this->assertSame($link->child_workflow_run_id, $handoff->child_workflow_run_id); + $this->assertSame(RunStatus::Waiting, WorkflowRun::query()->findOrFail($link->child_workflow_run_id)->status); + $this->assertSame($origin->id, (new ScopeRepository())->inspect($scope)['original_run_id']); + $stub->signal('resume'); + $this->drainTasks(); + $this->assertSame(RunStatus::Failed, $continued->refresh()->status); + $this->assertForwardRefused($continued->id, $scope, 2); + $result = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->drainTasks(); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame('verified', (new ScopeRepository())->inspect($child)['status']); + $this->assertSame(RunStatus::Completed, WorkflowRun::query()->findOrFail($result['runId'])->status); + $this->assertSame( + ['continued-shipment/child', 'continued-order/first'], + Cache::get('test:scope:undo-scoped-log') + ); + $this->assertSame($originHistory, $this->history($origin->id)); + $effects = Cache::get('test:scope:effect-count'); + app(WorkflowReplayer::class)->replay($origin->fresh()); + $this->assertSame($effects, Cache::get('test:scope:effect-count')); + } + + public function testOriginRedriveRefusesNewForwardWorkAndRetainsTheOriginalRecovery(): void + { + $scope = new ScopeIdentity('shop', 'client-a', 'redriven-order', '1'); + $stub = WorkflowStub::make(TestScopeRunBoundaryWorkflow::class, 'scope-redrive', 'client-a'); + $stub->start($scope->toArray(), 'redrive'); + $this->drainTasks(); + $origin = WorkflowRun::query()->findOrFail($stub->runId()); + $this->assertSame(RunStatus::Failed, $origin->status); + $originHistory = $this->history($origin->id); + $result = app(WorkflowControlPlane::class)->redrive('scope-redrive', $origin->id, [ + 'namespace' => 'client-a', + 'request_id' => 'origin-redrive', + ]); + $this->assertTrue($result['accepted'], (string) $result['reason']); + $this->drainTasks(); + $successor = WorkflowRun::query()->findOrFail($result['workflow_run_id']); + $this->assertSame(RunStatus::Failed, $successor->status); + $this->assertForwardRefused($successor->id, $scope, 1); + $this->assertSame(2, (int) Cache::get('test:redrive:second-calls')); + $recovery = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->drainTasks(); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(RunStatus::Completed, WorkflowRun::query()->findOrFail($recovery['runId'])->status); + $this->assertSame(['redriven-order/first'], Cache::get('test:scope:undo-scoped-log')); + $this->assertSame($origin->id, (new ScopeRepository())->inspect($scope)['original_run_id']); + $this->assertSame($originHistory, $this->history($origin->id)); + } + + public function testOriginRedriveCannotReissueAnUncertainForwardEffectFromTheOriginalScope(): void + { + $scope = new ScopeIdentity('shop', 'client-a', 'unknown-redriven-order', '1'); + $stub = WorkflowStub::make(TestScopeRunBoundaryWorkflow::class, 'scope-unknown-redrive', 'client-a'); + $stub->start($scope->toArray(), 'unknown'); + $this->drainTasks(); + $origin = WorkflowRun::query()->findOrFail($stub->runId()); + $this->assertSame(RunStatus::Failed, $origin->status); + $originHistory = $this->history($origin->id); + $before = (new ScopeRepository())->inspect($scope); + $this->assertSame('unknown', $before['obligations'][0]['forward_status']); + $this->assertSame(1, (int) Cache::get('test:scope:effect-count')); + $result = app(WorkflowControlPlane::class)->redrive('scope-unknown-redrive', $origin->id, [ + 'namespace' => 'client-a', + 'request_id' => 'origin-redrive', + ]); + $this->assertTrue($result['accepted'], (string) $result['reason']); + $this->drainTasks(); + $successor = WorkflowRun::query()->findOrFail($result['workflow_run_id']); + $this->assertSame(RunStatus::Failed, $successor->status); + $messages = WorkflowFailure::query()->where('workflow_run_id', $successor->id)->pluck('message')->all(); + $this->assertStringContainsString('scope_execution_owner_mismatch', implode('\n', $messages)); + $this->assertSame($before, (new ScopeRepository())->inspect($scope)); + $this->assertSame(1, (int) Cache::get('test:scope:effect-count')); + $recovery = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->drainTasks(); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(RunStatus::Completed, WorkflowRun::query()->findOrFail($recovery['runId'])->status); + $this->assertSame(['unknown-redriven-order/first'], Cache::get('test:scope:undo-scoped-log')); + $this->assertSame($originHistory, $this->history($origin->id)); + } + + private function assertForwardRefused(string $runId, ScopeIdentity $scope, int $effectCount): void + { + $messages = WorkflowFailure::query()->where('workflow_run_id', $runId)->pluck('message')->all(); + $this->assertStringContainsString('scope_owned_by_another_run', implode('\n', $messages)); + $snapshot = (new ScopeRepository())->inspect($scope); + $this->assertSame('open', $snapshot['status']); + $this->assertNotContains('second', array_column(array_column($snapshot['obligations'], 'descriptor'), 'key')); + $this->assertSame($effectCount, (int) Cache::get('test:scope:effect-count')); + $this->assertNull(Cache::get('test:scope:undo-log')); + } + + private function history(string $runId): array + { + return WorkflowHistoryEvent::query()->where('workflow_run_id', $runId)->orderBy('sequence')->get() + ->map(static fn (WorkflowHistoryEvent $event): array => [ + 'sequence' => $event->sequence, + 'event_type' => $event->event_type->value, + 'payload' => $event->payload, + ])->all(); + } + + private function drainTasks(): void + { + for ($count = 0; $count < 300; ++$count) { + $task = WorkflowTask::query()->where('status', TaskStatus::Ready->value)->orderBy('created_at')->first(); + if ($task === null) { + return; + } + if ($task->available_at !== null && $task->available_at->isFuture()) { + Carbon::setTestNow($task->available_at->copy()->addSecond()); + } + $job = match ($task->task_type) { + TaskType::Workflow => new RunWorkflowTask($task->id), + TaskType::Activity => new RunActivityTask($task->id), + TaskType::Timer => new RunTimerTask($task->id), + }; + $this->app->call([$job, 'handle']); + } + $this->fail('Ready tasks did not settle within the bounded drain.'); + } +} diff --git a/tests/Feature/V2/V2CompensationScopeTest.php b/tests/Feature/V2/V2CompensationScopeTest.php new file mode 100644 index 000000000..5d464c6ed --- /dev/null +++ b/tests/Feature/V2/V2CompensationScopeTest.php @@ -0,0 +1,677 @@ + ['default', 'client-a'], + 'queue.default' => 'sync', + 'queue.connections.sync.driver' => 'sync', + ]); + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([ + 'test.operation' => [ + '1' => new TestScopedOperation(), + ], + ])); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== 'client-a' || $principal !== 'operator-a') { + throw new \LogicException('Scope access refused.'); + } + return 'test-policy-1'; + } + }); + } + + protected function tearDown(): void + { + \Illuminate\Support\Carbon::setTestNow(); + parent::tearDown(); + } + + public function testHostCallbacksRunOutsideNativeStorageTransactions(): void + { + $calls = []; + $check = function (string $operation) use (&$calls): void { + $this->assertSame(0, ConfiguredV2Models::connection()->transactionLevel(), $operation); + $calls[] = $operation; + }; + $this->app->instance(ScopeAuthorization::class, new class($check) implements ScopeAuthorization { + public function __construct( + private Closure $check + ) { + } + + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + ($this->check)($operation); + return 'outside-lock-policy'; + } + }); + $handler = new class($check) implements ScopedOperation { + public function __construct( + private Closure $check + ) { + } + + public function assertInput(array $input): void + { + ($this->check)('input'); + } + + public function execute(OperationContext $context): ForwardResult + { + ($this->check)('execute'); + return (new TestScopedOperation())->execute($context); + } + + public function compensate(OperationContext $context): Resolution + { + ($this->check)('handler-compensate'); + return (new TestScopedOperation())->compensate($context); + } + }; + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([ + 'test.operation' => [ + '1' => $handler, + ], + ])); + WorkflowStub::fake(); + $parent = $this->identity(); + $child = new ScopeIdentity('shop', 'client-a', 'shipment-42', '1'); + $childOrigin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'callback-child', 'client-a'); + $childOrigin->start($child->toArray(), 'complete'); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'callback-parent', 'client-a'); + $origin->start($parent->toArray(), 'complete', $child->toArray()); + $this->assertTrue($origin->refresh()->completed()); + app(ScopeRecovery::class)->request($parent, 'cancel', 'operator-a', 'cancel'); + $this->assertSame('verified', (new ScopeRepository())->inspect($parent)['status']); + $this->assertSame('verified', (new ScopeRepository())->inspect($child)['status']); + foreach (['register', 'forward', 'participate', 'input', 'execute', 'handler-compensate'] as $operation) { + $this->assertContains($operation, $calls); + } + } + + public function testCancellationDuringHostAuthorizationRefusesDispatchAtTheNativeGate(): void + { + WorkflowStub::fake(); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + private bool $cancelled = false; + + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($operation === 'forward' && ! $this->cancelled) { + $this->cancelled = true; + WorkflowStub::load('preflight-cancel-origin')->attemptCancel('concurrent-cancel'); + } + return 'preflight-policy'; + } + }); + $scope = $this->identity(); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'preflight-cancel-origin', 'client-a'); + $origin->start($scope->toArray(), 'complete'); + $this->assertSame('cancelled', $origin->refresh()->status()); + $this->assertNull(Cache::get('test:scope:effect-count')); + $this->assertSame('registered', (new ScopeRepository())->inspect($scope)['obligations'][0]['forward_status']); + app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(Cache::get('test:scope:effect-count')); + } + + public static function interruptedRecoveryCommands(): iterable + { + yield 'cancelled' => ['cancelled']; + yield 'terminated' => ['terminated']; + } + + #[DataProvider('interruptedRecoveryCommands')] + public function testInterruptedRecoveryStartsFreshCanonicalRunWithoutRepeatingVerifiedEffects(string $status): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:unknown-first', true); + $first = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $stub = WorkflowStub::loadRun($first['runId']); + $command = $status === 'cancelled' ? $stub->attemptCancel('operator-stop') : $stub->attemptTerminate( + 'operator-stop' + ); + $this->assertTrue($command->accepted()); + $this->assertSame($status, WorkflowRun::query()->findOrFail($first['runId'])->status->value); + $before = WorkflowHistoryEvent::query()->where('workflow_run_id', $first['runId'])->count(); + Cache::forget('test:scope:unknown-first'); + $again = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertNotSame($first['runId'], $again['runId']); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame('completed', WorkflowRun::query()->findOrFail($again['runId'])->status->value); + $this->assertSame( + 2, + WorkflowRun::query()->where('workflow_instance_id', $scope->recoveryInstanceId())->count() + ); + $this->assertSame( + 1, + count(array_filter(Cache::get('test:scope:undo-log'), static fn (string $key): bool => $key === 'second')) + ); + $this->assertSame($before, WorkflowHistoryEvent::query()->where('workflow_run_id', $first['runId'])->count()); + } + + public function testScopeActivityNamesAndLayoutVersionsAreRecordedForReplay(): void + { + [$scope, $origin] = $this->completedOrigin(); + $history = WorkflowHistoryEvent::query()->where('workflow_run_id', $origin->runId())->get(); + $this->assertNotEmpty($history->where('event_type', HistoryEventType::VersionMarkerRecorded)); + foreach ($history->where('event_type', HistoryEventType::ActivityScheduled) as $event) { + $this->assertStringStartsWith('dw.compensation.', $event->payload['activity_type']); + $this->assertStringEndsWith('.v1', $event->payload['activity_type']); + } + $recovery = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertTrue(WorkflowHistoryEvent::query()->where('workflow_run_id', $recovery['runId']) + ->where('event_type', HistoryEventType::VersionMarkerRecorded)->exists()); + app(WorkflowReplayer::class)->replay(WorkflowRun::query()->findOrFail($origin->runId())); + $this->assertSame(2, (int) Cache::get('test:scope:effect-count')); + } + + public function testClosedOriginUsesCanonicalRecoveryAndKeepsOriginalTerminalHistory(): void + { + [$scope, $origin] = $this->completedOrigin(); + $before = WorkflowHistoryEvent::query()->where('workflow_run_id', $origin->runId())->count(); + $result = app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame('completed', WorkflowRun::query()->findOrFail($result['runId'])->status->value); + $this->assertSame('completed', $origin->refresh()->status()); + $this->assertSame($before, WorkflowHistoryEvent::query()->where('workflow_run_id', $origin->runId())->count()); + $this->assertSame(['second', 'first'], Cache::get('test:scope:undo-log')); + } + + public function testNativeSagaCompensatesAnEffectWhoseForwardResultWasLost(): void + { + WorkflowStub::fake(); + $scope = $this->identity(); + $workflow = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'lost-ack-origin', 'client-a'); + $workflow->start($scope->toArray(), 'lost-ack'); + $this->assertTrue($workflow->refresh()->completed()); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(['first'], Cache::get('test:scope:undo-log')); + $this->assertSame(1, (int) Cache::get('test:scope:effect-count')); + } + + public function testRetryExhaustionStaysUnresolvedAndNativeRedriveReusesVerifiedWork(): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:unknown-first', true); + $first = app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->driveNativeRetries($first['runId']); + $this->assertSame('failed', WorkflowRun::query()->findOrFail($first['runId'])->status->value); + $this->assertSame('requested', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame( + 1, + count(array_filter(Cache::get('test:scope:undo-log'), static fn (string $key): bool => $key === 'second')) + ); + Cache::forget('test:scope:unknown-first'); + $again = app(ScopeRecovery::class)->request($scope, 'cancel-2', 'operator-a', 'order-cancelled'); + $this->assertNotSame($first['runId'], $again['runId']); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame( + 1, + count(array_filter(Cache::get('test:scope:undo-log'), static fn (string $key): bool => $key === 'second')) + ); + $this->assertSame( + 2, + WorkflowRun::query()->where('workflow_instance_id', $scope->recoveryInstanceId())->count() + ); + } + + public function testNativeActiveAdoptionDoesNotPreventRecoveryRedriveAfterFailure(): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:unknown-first', true); + $first = app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'cancel'); + $adopted = WorkflowStub::make(RecoveryWorkflow::class, $scope->recoveryInstanceId(), $scope->namespace) + ->attemptStart(StartOptions::returnExistingActive(), $scope->toArray()); + $this->assertTrue($adopted->returnedExistingActive()); + $this->assertSame($first['runId'], $adopted->runId()); + $started = WorkflowHistoryEvent::query()->where('workflow_run_id', $first['runId']) + ->where('event_type', \Workflow\V2\Enums\HistoryEventType::WorkflowStarted)->firstOrFail(); + $adoption = WorkflowHistoryEvent::query()->where('workflow_command_id', $adopted->commandId())->firstOrFail(); + $this->assertGreaterThan($started->sequence, $adoption->sequence); + $this->driveNativeRetries($first['runId']); + $this->assertSame('failed', WorkflowRun::query()->findOrFail($first['runId'])->status->value); + Cache::forget('test:scope:unknown-first'); + $redrive = app(ScopeRecovery::class)->request($scope, 'cancel-2', 'operator-a', 'cancel'); + $this->assertNotSame($first['runId'], $redrive['runId']); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(1, count(array_filter( + Cache::get('test:scope:undo-log'), + static fn (string $key): bool => $key === 'second' + ))); + } + + public function testOneRetainedRequestCanRedriveTwoFailuresWithoutRepeatingVerifiedWork(): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:unknown-first', true); + $first = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->driveNativeRetries($first['runId']); + $second = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertNotSame($first['runId'], $second['runId']); + $duplicate = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertSame($second['runId'], $duplicate['runId']); + $this->driveNativeRetries($second['runId']); + $this->assertSame('failed', WorkflowRun::query()->findOrFail($second['runId'])->status->value); + Cache::forget('test:scope:unknown-first'); + $third = app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + $this->assertNotSame($second['runId'], $third['runId']); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame( + 3, + WorkflowRun::query()->where('workflow_instance_id', $scope->recoveryInstanceId())->count() + ); + $this->assertSame(1, \Workflow\V2\Support\ConfiguredV2Models::connection() + ->table('workflow_compensation_requests') + ->where('scope_id', $scope->id()) + ->count()); + $this->assertSame(1, count(array_filter( + Cache::get('test:scope:undo-log'), + static fn (string $key): bool => $key === 'second' + ))); + } + + public function testForwardEvidenceSurvivesReleaseWithoutRetainingBusinessInputsOrResults(): void + { + [$scope] = $this->completedOrigin(); + $repository = new ScopeRepository(); + $observations = array_values(array_filter( + $repository->journal($scope), + static fn (array $row): bool => $row['kind'] === 'forward_outcome' + )); + $this->assertCount(2, $observations); + foreach ($observations as $row) { + $record = json_decode($row['data'], true, flags: JSON_THROW_ON_ERROR); + $this->assertArrayHasKey('evidence', $record); + $this->assertSame(['provider:' . $record['obligationId']], $record['evidence']); + $this->assertArrayNotHasKey('value', $record); + $this->assertArrayNotHasKey('input', $record); + } + app(ScopeRecovery::class)->request($scope, 'cancel', 'operator-a', 'cancel'); + app(\Workflow\V2\Compensation\Recovery\ScopeRetention::class)->release($scope, 'operator-a'); + $retained = array_values(array_filter( + $repository->journal($scope), + static fn (array $row): bool => $row['kind'] === 'forward_outcome' + )); + $this->assertSame($observations, $retained); + foreach ($repository->inspect($scope)['obligations'] as $row) { + $this->assertNull($row['forward_result']); + $this->assertArrayNotHasKey('input', $row['descriptor']); + } + } + + public function testColdReplayDoesNotResolveMutableHandlersOrRepeatProviderEffects(): void + { + [$scope, $origin] = $this->completedOrigin(); + $effects = Cache::get('test:scope:effect-count'); + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([])); + app(WorkflowReplayer::class)->replay(WorkflowRun::query()->findOrFail($origin->runId())); + $this->assertSame($effects, Cache::get('test:scope:effect-count')); + $this->assertNull(Cache::get('test:scope:undo-log')); + $this->assertSame('open', (new ScopeRepository())->inspect($scope)['status']); + } + + public function testUnavailablePinnedHandlerFailsBeforeCompensationEffect(): void + { + [$scope] = $this->completedOrigin(); + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([])); + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('requested', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(Cache::get('test:scope:undo-log')); + } + + public function testExpiredNativeAttemptCannotPublishVerifiedCompensation(): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:expire-native', true); + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $snapshot = (new ScopeRepository())->inspect($scope); + $this->assertSame('requested', $snapshot['status']); + $this->assertNotContains('verified', array_column($snapshot['obligations'], 'compensation_status')); + } + + public function testPossibleLateForwardEffectCannotBecomeVerifiedAfterRetryExhaustion(): void + { + [$scope] = $this->completedOrigin(); + Cache::put('test:scope:late-forward-possible', true); + $recovery = app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->driveNativeRetries($recovery['runId']); + $snapshot = (new ScopeRepository())->inspect($scope); + $this->assertSame('requested', $snapshot['status']); + $this->assertNotContains('verified', array_column($snapshot['obligations'], 'compensation_status')); + $this->assertSame('failed', WorkflowRun::query()->findOrFail($recovery['runId'])->status->value); + } + + public function testRecordedAuthorizationIncludesNativeAttemptAndPolicyVersion(): void + { + [$scope] = $this->completedOrigin(); + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $events = array_values( + array_filter((new ScopeRepository())->journal( + $scope + ), static fn (array $row): bool => $row['kind'] === 'authorized') + ); + $this->assertNotEmpty($events); + foreach ($events as $row) { + $record = json_decode($row['data'], true, flags: JSON_THROW_ON_ERROR); + $this->assertSame('test-policy-1', $record['policyVersion']); + $this->assertNotEmpty($record['attemptId']); + $this->assertNotEmpty($record['runId']); + } + } + + public function testCompletedChildScopeIsVerifiedBeforeEarlierParentCompensation(): void + { + WorkflowStub::fake(); + $child = new ScopeIdentity('shop', 'client-a', 'shipment-42', '1'); + $childOrigin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'child-origin', 'client-a'); + $childOrigin->start($child->toArray(), 'complete'); + $this->assertTrue($childOrigin->refresh()->completed()); + $parent = $this->identity(); + $parentOrigin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'parent-origin', 'client-a'); + $parentOrigin->start($parent->toArray(), 'with-child', $child->toArray()); + $this->assertTrue($parentOrigin->refresh()->completed()); + app(ScopeRecovery::class)->request($parent, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('verified', (new ScopeRepository())->inspect($parent)['status']); + $this->assertSame('verified', (new ScopeRepository())->inspect($child)['status']); + $this->assertSame( + ['order-42/second', 'shipment-42/second', 'shipment-42/first', 'order-42/first'], + Cache::get('test:scope:undo-scoped-log') + ); + } + + public function testChildParticipationRecordsItsOwnAuthorizationPolicyVersion(): void + { + WorkflowStub::fake(); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + return $operation === 'participate' ? 'child-policy-7' : 'parent-policy-2'; + } + }); + $parent = $this->identity(); + $child = new ScopeIdentity('shop', 'client-a', 'shipment-42', '1'); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'parent-origin', 'client-a'); + $origin->start($parent->toArray(), 'with-child', $child->toArray()); + $this->assertTrue($origin->refresh()->completed()); + $records = array_values(array_filter( + (new ScopeRepository())->journal($parent), + static fn (array $row): bool => $row['kind'] === 'child_authorized' + )); + $this->assertCount(1, $records); + $record = json_decode($records[0]['data'], true, flags: JSON_THROW_ON_ERROR); + $this->assertSame('child-policy-7', $record['policyVersion']); + $this->assertSame($child->id(), $record['childScopeId']); + $this->assertSame('operator-a', $record['principal']); + $this->assertNotEmpty($record['attemptId']); + } + + public function testInvalidChildParticipationPolicyCannotRegisterAChild(): void + { + WorkflowStub::fake(); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + return $operation === 'participate' ? '' : 'parent-policy-2'; + } + }); + $parent = $this->identity(); + $child = new ScopeIdentity('shop', 'client-a', 'shipment-42', '1'); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'parent-origin', 'client-a'); + $origin->start($parent->toArray(), 'with-child', $child->toArray()); + $this->assertCount(1, (new ScopeRepository())->inspect($parent)['obligations']); + $this->assertSame(0, \Workflow\V2\Support\ConfiguredV2Models::connection() + ->table('workflow_compensation_children') + ->count()); + $this->assertSame(1, (int) Cache::get('test:scope:effect-count')); + } + + public function testUnavailableChildCannotAdvanceEarlierParentCompensation(): void + { + WorkflowStub::fake(); + $parent = $this->identity(); + $child = new ScopeIdentity('shop', 'client-a', 'missing-shipment', '1'); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'parent-origin', 'client-a'); + $origin->start($parent->toArray(), 'with-child', $child->toArray()); + $this->assertTrue($origin->refresh()->completed()); + $result = app(ScopeRecovery::class)->request($parent, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->driveNativeRetries($result['runId']); + $this->assertSame('requested', (new ScopeRepository())->inspect($parent)['status']); + $this->assertSame(['second'], Cache::get('test:scope:undo-log')); + } + + public function testOrdinaryRunPruningPreservesRetainedScopeRecovery(): void + { + [$scope, $origin] = $this->completedOrigin(); + \Workflow\V2\Support\WorkflowRunRetentionCleanup::pruneRun($origin->runId()); + $this->assertSame(0, WorkflowHistoryEvent::query()->where('workflow_run_id', $origin->runId())->count()); + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(['second', 'first'], Cache::get('test:scope:undo-log')); + } + + public function testTamperedRetainedDescriptorIsRefusedBeforeProviderCall(): void + { + [$scope] = $this->completedOrigin(); + $row = (new ScopeRepository())->inspect($scope)['obligations'][1]; + $descriptor = $row['descriptor']; + $descriptor['input']['changed'] = true; + \Workflow\V2\Support\ConfiguredV2Models::connection()->table('workflow_compensation_obligations')->where( + 'id', + $row['id'] + )->update([ + 'descriptor' => json_encode($descriptor, JSON_THROW_ON_ERROR), + ]); + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('requested', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(Cache::get('test:scope:undo-log')); + } + + public function testTamperedRegistrationCannotReachTheForwardProvider(): void + { + WorkflowStub::fake(); + $scope = $this->identity(); + $tampered = false; + \Workflow\V2\Support\ConfiguredV2Models::connection()->listen( + static function (QueryExecuted $query) use ($scope, &$tampered): void { + if ($tampered || ! str_starts_with($query->sql, 'insert') + || ! str_contains($query->sql, 'workflow_compensation_obligations')) { + return; + } + $tampered = true; + $row = $query->connection->table('workflow_compensation_obligations') + ->where('scope_id', $scope->id()) + ->first(); + $descriptor = json_decode($row->descriptor, true, flags: JSON_THROW_ON_ERROR); + $descriptor['input']['changed'] = true; + $query->connection->table('workflow_compensation_obligations') + ->where('id', $row->id) + ->update([ + 'descriptor' => json_encode($descriptor, JSON_THROW_ON_ERROR), + ]); + } + ); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'tampered-forward-origin', 'client-a'); + $origin->start($scope->toArray(), 'complete'); + $this->assertTrue($tampered); + $this->assertNull(Cache::get('test:scope:effect-count')); + $this->assertNotContains( + 'observed', + array_column((new ScopeRepository())->inspect($scope)['obligations'], 'forward_status') + ); + } + + public function testReleaseRefusesUnresolvedWorkThenRetainsAnUnreusableTombstone(): void + { + [$scope] = $this->completedOrigin(); + try { + app(\Workflow\V2\Compensation\Recovery\ScopeRetention::class)->release($scope, 'operator-a'); + $this->fail('Unresolved obligations were released.'); + } catch (\LogicException) { + $this->assertSame('open', (new ScopeRepository())->inspect($scope)['status']); + } + app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'order-cancelled'); + $before = (new ScopeRepository())->inspect($scope); + app(\Workflow\V2\Compensation\Recovery\ScopeRetention::class)->release($scope, 'operator-a'); + $after = (new ScopeRepository())->inspect($scope); + $this->assertSame('released', $after['status']); + $this->assertSame( + array_column($before['obligations'], 'fingerprint'), + array_column($after['obligations'], 'fingerprint') + ); + $this->assertArrayNotHasKey('input', $after['obligations'][0]['descriptor']); + $this->expectException(\LogicException::class); + app(ScopeRecovery::class)->request($scope, 'cancel-2', 'operator-a', 'order-cancelled'); + } + + public function testDefaultEmbeddedNamespaceIsPreservedWithoutInventingAnAlias(): void + { + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== null || $principal !== 'operator-a') { + throw new \LogicException('Default namespace access refused.'); + } + return 'default-policy-1'; + } + }); + WorkflowStub::fake(); + $scope = new ScopeIdentity('shop', null, 'order-42', '1'); + $origin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'default-origin'); + $origin->start($scope->toArray(), 'complete'); + $this->assertTrue($origin->refresh()->completed()); + $result = app(ScopeRecovery::class)->request($scope, 'cancel-1', 'operator-a', 'cancel'); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(WorkflowRun::query()->findOrFail($result['runId'])->namespace); + } + + public function testRetainedRecoveryRunsInAFreshPhpProcess(): void + { + [$scope] = $this->completedOrigin(); + $input = tempnam(sys_get_temp_dir(), 'scope-recovery-input-'); + $report = tempnam(sys_get_temp_dir(), 'scope-recovery-report-'); + chmod($input, 0600); + try { + file_put_contents($input, json_encode([ + 'scope' => $scope->toArray(), + 'database' => config('database.connections.' . config('database.default')), + 'redis' => config('database.redis'), + 'cache' => config('cache.stores.redis'), + 'cache_prefix' => config('cache.prefix'), + 'report' => $report, + ], JSON_THROW_ON_ERROR)); + $process = new \Symfony\Component\Process\Process([ + PHP_BINARY, 'vendor/bin/phpunit', 'tests/Support/CompensationScopeRecoveryProcessTest.php', + ], dirname(__DIR__, 3), [ + 'DWF_SCOPE_RECOVERY_INPUT' => $input, + ], timeout: 30); + $process->mustRun(); + $result = json_decode(file_get_contents($report), true, flags: JSON_THROW_ON_ERROR); + $this->assertNotSame(getmypid(), $result['pid']); + $this->assertSame('verified', $result['status']); + $this->assertSame('verified', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame(['second', 'first'], Cache::get('test:scope:undo-log')); + } finally { + unlink($input); + unlink($report); + } + } + + public function testCancellationWhileChildWaitsResolvesItsSagaBeforeEarlierParentWork(): void + { + WorkflowStub::fake(); + $child = new ScopeIdentity('shop', 'client-a', 'shipment-42', '1'); + $childOrigin = WorkflowStub::make( + \Tests\Fixtures\V2\TestWaitingScopeWorkflow::class, + 'waiting-child', + 'client-a' + ); + $childOrigin->start($child->toArray()); + $this->assertSame('waiting', $childOrigin->refresh()->status()); + $parent = $this->identity(); + $parentOrigin = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'parent-origin', 'client-a'); + $parentOrigin->start($parent->toArray(), 'with-child', $child->toArray()); + app(ScopeRecovery::class)->request($parent, 'cancel-1', 'operator-a', 'order-cancelled'); + $this->assertSame('verified', (new ScopeRepository())->inspect($child)['status']); + $this->assertSame('verified', (new ScopeRepository())->inspect($parent)['status']); + $this->assertSame( + ['order-42/second', 'shipment-42/second', 'shipment-42/first', 'order-42/first'], + Cache::get('test:scope:undo-scoped-log') + ); + } + + private function driveNativeRetries(string $runId): void + { + for ($step = 0; $step < 20; $step++) { + $run = WorkflowRun::query()->findOrFail($runId); + if ($run->status->isTerminal()) { + return; + } + $task = \Workflow\V2\Models\WorkflowTask::query()->where('workflow_run_id', $runId)->where( + 'status', + 'ready' + )->orderBy('available_at') + ->first(); + $this->assertNotNull($task, 'A nonterminal recovery must retain a native retry task.'); + if ($task->available_at !== null && $task->available_at->isFuture()) { + \Illuminate\Support\Carbon::setTestNow($task->available_at->copy()->addMicrosecond()); + } + if ($task->task_type->value === 'activity') { + $this->app->call([new \Workflow\V2\Jobs\RunActivityTask($task->id), 'handle']); + } else { + $this->app->call([new \Workflow\V2\Jobs\RunWorkflowTask($task->id), 'handle']); + } + } + $this->fail('Native retry budget did not terminate.'); + } + + private function completedOrigin(): array + { + WorkflowStub::fake(); + $scope = $this->identity(); + $workflow = WorkflowStub::make(TestRetainedScopeWorkflow::class, 'retained-origin', 'client-a'); + $workflow->start($scope->toArray(), 'complete'); + $this->assertTrue($workflow->refresh()->completed()); + return [$scope, $workflow]; + } + + private function identity(): ScopeIdentity + { + return new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + } +} diff --git a/tests/Fixtures/V2/ReplayRegression/redrive-native-active-start-adoption.json b/tests/Fixtures/V2/ReplayRegression/redrive-native-active-start-adoption.json new file mode 100644 index 000000000..c59213735 --- /dev/null +++ b/tests/Fixtures/V2/ReplayRegression/redrive-native-active-start-adoption.json @@ -0,0 +1,229 @@ +{ + "$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json", + "fixture_schema": "durable-workflow.replay-regression/v1", + "id": "redrive-native-active-start-adoption", + "protocol_version": "1.0", + "bindings": [ + "php" + ], + "workflow": { + "type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "arguments": [ + "Taylor", + false + ], + "payload_codec": "avro" + }, + "history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "StartAccepted", + "recorded_at": "2026-11-12T00:00:01+00:00", + "workflow_run_id": "source-run", + "workflow_command_id": "adoption-command", + "payload": { + "workflow_instance_id": "source-instance", + "workflow_run_id": "source-run", + "workflow_class": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "workflow_type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "workflow_command_id": "adoption-command", + "outcome": "returned_existing_active" + } + }, + { + "sequence": 3, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + } + ], + "redrive_plan": { + "source_history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "StartAccepted", + "recorded_at": "2026-11-12T00:00:01+00:00", + "workflow_run_id": "source-run", + "workflow_command_id": "adoption-command", + "payload": { + "workflow_instance_id": "source-instance", + "workflow_run_id": "source-run", + "workflow_class": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "workflow_type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "workflow_command_id": "adoption-command", + "outcome": "returned_existing_active", + "command": { + "id": "adoption-command", + "type": "start", + "target_scope": "instance", + "resolved_run_id": "source-run", + "status": "accepted", + "outcome": "returned_existing_active" + } + } + }, + { + "sequence": 3, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityFailed", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "message": "provider unavailable", + "exception_class": "RuntimeException" + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + }, + { + "sequence": 7, + "event_type": "WorkflowFailed", + "payload": { + "failed_step_kind": "activity", + "failed_step_sequence": 2 + }, + "recorded_at": "2026-11-12T00:00:06+00:00" + } + ], + "expected": { + "eligible": true, + "resume_step_sequence": 2, + "completed_count": 1, + "reused_event_types": [ + "ActivityCompleted" + ] + }, + "refusals": [ + { + "event_index": 1, + "field": "outcome", + "value": "started_new", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "workflow_instance_id", + "value": "other-instance", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "workflow_run_id", + "value": "other-run", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "workflow_class", + "value": "OtherWorkflow", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "workflow_type", + "value": "other-type", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "workflow_command_id", + "value": "other-command", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "rejection_reason", + "value": "refused", + "reason": "unexpected_start" + }, + { + "event_index": 1, + "field": "command", + "value": null, + "reason": "unexpected_start" + } + ], + "source_run": { + "id": "source-run", + "workflow_instance_id": "source-instance", + "workflow_class": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "workflow_type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow" + } + }, + "expected": { + "completed": false, + "result": null, + "commands": [ + { + "type": "schedule_activity", + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "payload_codec": "avro" + } + ] + } +} diff --git a/tests/Fixtures/V2/ReplayRegression/redrive-retried-activity-after-completed-child.json b/tests/Fixtures/V2/ReplayRegression/redrive-retried-activity-after-completed-child.json new file mode 100644 index 000000000..7c0d4baae --- /dev/null +++ b/tests/Fixtures/V2/ReplayRegression/redrive-retried-activity-after-completed-child.json @@ -0,0 +1,441 @@ +{ + "$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json", + "fixture_schema": "durable-workflow.replay-regression/v1", + "id": "redrive-retried-activity-after-completed-child", + "protocol_version": "1.0", + "bindings": [ + "php" + ], + "workflow": { + "type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "arguments": [ + "Taylor", + true + ], + "payload_codec": "avro" + }, + "history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ChildWorkflowScheduled", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "parent_close_policy": "abandon", + "cancellation_policy": "abandon" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ChildRunStarted", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "child_run_number": 1, + "parent_close_policy": "abandon", + "cancellation_policy": "abandon" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ChildRunCompleted", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "child_run_number": 1, + "child_status": "completed", + "closed_reason": "completed", + "closed_at": "2026-11-12T00:00:01.000000Z", + "output": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "attempt_number": 1, + "workflow_task_id": "task-2-1" + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + }, + { + "sequence": 7, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "retry_after_attempt_id": "attempt-2-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-2-1", + "retry_task_id": "task-2-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:06+00:00" + }, + { + "sequence": 8, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "workflow_task_id": "task-2-2" + }, + "recorded_at": "2026-11-12T00:00:07+00:00" + }, + { + "sequence": 9, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:08+00:00" + } + ], + "redrive_plan": { + "source_history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ChildWorkflowScheduled", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "parent_close_policy": "abandon", + "cancellation_policy": "abandon" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ChildRunStarted", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "child_run_number": 1, + "parent_close_policy": "abandon", + "cancellation_policy": "abandon" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ChildRunCompleted", + "payload": { + "sequence": 1, + "workflow_link_id": "01JCREDRIVECHILDLINK000000", + "child_call_id": "01JCREDRIVECHILDLINK000000", + "child_workflow_instance_id": "01jcredrivechildinstance00", + "child_workflow_run_id": "01jcredrivechildrun0000000", + "child_workflow_class": "Tests\\Fixtures\\V2\\TestRedriveGreetingChildWorkflow", + "child_workflow_type": "test-redrive-greeting-child-workflow", + "child_run_number": 1, + "child_status": "completed", + "closed_reason": "completed", + "closed_at": "2026-11-12T00:00:01.000000Z", + "output": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "attempt_number": 1, + "workflow_task_id": "task-2-1" + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + }, + { + "sequence": 7, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "retry_after_attempt_id": "attempt-2-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-2-1", + "retry_task_id": "task-2-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:06+00:00" + }, + { + "sequence": 8, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "workflow_task_id": "task-2-2" + }, + "recorded_at": "2026-11-12T00:00:07+00:00" + }, + { + "sequence": 9, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:08+00:00" + }, + { + "sequence": 10, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 3, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-3" + }, + "recorded_at": "2026-11-12T00:00:09+00:00" + }, + { + "sequence": 11, + "event_type": "ActivityStarted", + "payload": { + "sequence": 3, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-3", + "activity_attempt_id": "attempt-3-1", + "attempt_number": 1, + "workflow_task_id": "task-3-1" + }, + "recorded_at": "2026-11-12T00:00:10+00:00" + }, + { + "sequence": 12, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 3, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-3", + "activity_attempt_id": "attempt-3-1", + "retry_after_attempt_id": "attempt-3-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-3-1", + "retry_task_id": "task-3-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:11+00:00" + }, + { + "sequence": 13, + "event_type": "ActivityStarted", + "payload": { + "sequence": 3, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-3", + "activity_attempt_id": "attempt-3-2", + "attempt_number": 2, + "workflow_task_id": "task-3-2" + }, + "recorded_at": "2026-11-12T00:00:12+00:00" + }, + { + "sequence": 14, + "event_type": "ActivityFailed", + "payload": { + "sequence": 3, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-3", + "activity_attempt_id": "attempt-3-2", + "attempt_number": 2, + "message": "provider unavailable", + "exception_class": "RuntimeException" + }, + "recorded_at": "2026-11-12T00:00:13+00:00" + }, + { + "sequence": 15, + "event_type": "WorkflowFailed", + "payload": { + "failed_step_kind": "activity", + "failed_step_sequence": 3 + }, + "recorded_at": "2026-11-12T00:00:14+00:00" + } + ], + "expected": { + "eligible": true, + "resume_step_sequence": 3, + "completed_count": 1, + "reused_event_types": [ + "ChildRunCompleted", + "ActivityCompleted" + ] + }, + "refusals": [ + { + "event_index": 6, + "field": "activity_execution_id", + "value": "substituted-execution", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 6, + "field": "activity_attempt_id", + "value": "substituted-attempt", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 6, + "field": "retry_task_id", + "value": "substituted-task", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 6, + "field": "retry_after_attempt", + "value": 9, + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 11, + "field": "activity_execution_id", + "value": "substituted-execution", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 11, + "field": "activity_attempt_id", + "value": "substituted-attempt", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 11, + "field": "retry_task_id", + "value": "substituted-task", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 11, + "field": "retry_after_attempt", + "value": 9, + "reason": "invalid_activity_retry_history" + } + ] + }, + "expected": { + "completed": false, + "result": null, + "commands": [ + { + "type": "schedule_activity", + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "payload_codec": "avro" + } + ] + } +} diff --git a/tests/Fixtures/V2/ReplayRegression/redrive-retried-completed-activity.json b/tests/Fixtures/V2/ReplayRegression/redrive-retried-completed-activity.json new file mode 100644 index 000000000..a63c15a0a --- /dev/null +++ b/tests/Fixtures/V2/ReplayRegression/redrive-retried-completed-activity.json @@ -0,0 +1,332 @@ +{ + "$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json", + "fixture_schema": "durable-workflow.replay-regression/v1", + "id": "redrive-retried-completed-activity", + "protocol_version": "1.0", + "bindings": [ + "php" + ], + "workflow": { + "type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "arguments": [ + "Taylor", + false + ], + "payload_codec": "avro" + }, + "history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ActivityStarted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-1", + "attempt_number": 1, + "workflow_task_id": "task-1-1" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-1", + "retry_after_attempt_id": "attempt-1-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-1-1", + "retry_task_id": "task-1-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityStarted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-2", + "attempt_number": 2, + "workflow_task_id": "task-1-2" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-2", + "attempt_number": 2, + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + } + ], + "redrive_plan": { + "source_history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ActivityStarted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-1", + "attempt_number": 1, + "workflow_task_id": "task-1-1" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-1", + "retry_after_attempt_id": "attempt-1-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-1-1", + "retry_task_id": "task-1-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityStarted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-2", + "attempt_number": 2, + "workflow_task_id": "task-1-2" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "activity_attempt_id": "attempt-1-2", + "attempt_number": 2, + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + }, + { + "sequence": 7, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2" + }, + "recorded_at": "2026-11-12T00:00:06+00:00" + }, + { + "sequence": 8, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "attempt_number": 1, + "workflow_task_id": "task-2-1" + }, + "recorded_at": "2026-11-12T00:00:07+00:00" + }, + { + "sequence": 9, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "retry_after_attempt_id": "attempt-2-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-2-1", + "retry_task_id": "task-2-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:08+00:00" + }, + { + "sequence": 10, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "workflow_task_id": "task-2-2" + }, + "recorded_at": "2026-11-12T00:00:09+00:00" + }, + { + "sequence": 11, + "event_type": "ActivityFailed", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "message": "provider unavailable", + "exception_class": "RuntimeException" + }, + "recorded_at": "2026-11-12T00:00:10+00:00" + }, + { + "sequence": 12, + "event_type": "WorkflowFailed", + "payload": { + "failed_step_kind": "activity", + "failed_step_sequence": 2 + }, + "recorded_at": "2026-11-12T00:00:11+00:00" + } + ], + "expected": { + "eligible": true, + "resume_step_sequence": 2, + "completed_count": 1, + "reused_event_types": [ + "ActivityCompleted" + ] + }, + "refusals": [ + { + "event_index": 3, + "field": "activity_execution_id", + "value": "substituted-execution", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 3, + "field": "activity_attempt_id", + "value": "substituted-attempt", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 3, + "field": "retry_task_id", + "value": "substituted-task", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 3, + "field": "retry_after_attempt", + "value": 9, + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 8, + "field": "activity_execution_id", + "value": "substituted-execution", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 8, + "field": "activity_attempt_id", + "value": "substituted-attempt", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 8, + "field": "retry_task_id", + "value": "substituted-task", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 8, + "field": "retry_after_attempt", + "value": 9, + "reason": "invalid_activity_retry_history" + } + ] + }, + "expected": { + "completed": false, + "result": null, + "commands": [ + { + "type": "schedule_activity", + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "payload_codec": "avro" + } + ] + } +} diff --git a/tests/Fixtures/V2/ReplayRegression/redrive-retried-failed-activity.json b/tests/Fixtures/V2/ReplayRegression/redrive-retried-failed-activity.json new file mode 100644 index 000000000..c448c16be --- /dev/null +++ b/tests/Fixtures/V2/ReplayRegression/redrive-retried-failed-activity.json @@ -0,0 +1,210 @@ +{ + "$schema": "https://raw.githubusercontent.com/durable-workflow/.github/main/regression-corpus/evidence-schema.json", + "fixture_schema": "durable-workflow.replay-regression/v1", + "id": "redrive-retried-failed-activity", + "protocol_version": "1.0", + "bindings": [ + "php" + ], + "workflow": { + "type": "Tests\\Fixtures\\V2\\TestRedriveRetryWorkflow", + "arguments": [ + "Taylor", + false + ], + "payload_codec": "avro" + }, + "history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + } + ], + "redrive_plan": { + "source_history": [ + { + "sequence": 1, + "event_type": "WorkflowStarted", + "payload": {}, + "recorded_at": "2026-11-12T00:00:00+00:00" + }, + { + "sequence": 2, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1" + }, + "recorded_at": "2026-11-12T00:00:01+00:00" + }, + { + "sequence": 3, + "event_type": "ActivityCompleted", + "payload": { + "sequence": 1, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveFirstActivity", + "activity_execution_id": "execution-1", + "result": "wwHioz3/VYAiNwocSGVsbG8sIFRheWxvciE=", + "payload_codec": "avro" + }, + "recorded_at": "2026-11-12T00:00:02+00:00" + }, + { + "sequence": 4, + "event_type": "ActivityScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2" + }, + "recorded_at": "2026-11-12T00:00:03+00:00" + }, + { + "sequence": 5, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "attempt_number": 1, + "workflow_task_id": "task-2-1" + }, + "recorded_at": "2026-11-12T00:00:04+00:00" + }, + { + "sequence": 6, + "event_type": "ActivityRetryScheduled", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-1", + "retry_after_attempt_id": "attempt-2-1", + "retry_after_attempt": 1, + "retry_of_task_id": "task-2-1", + "retry_task_id": "task-2-2", + "max_attempts": 2, + "retry_reason": "failure", + "retry_backoff_seconds": 0 + }, + "recorded_at": "2026-11-12T00:00:05+00:00" + }, + { + "sequence": 7, + "event_type": "ActivityStarted", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "workflow_task_id": "task-2-2" + }, + "recorded_at": "2026-11-12T00:00:06+00:00" + }, + { + "sequence": 8, + "event_type": "ActivityFailed", + "payload": { + "sequence": 2, + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_class": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "activity_execution_id": "execution-2", + "activity_attempt_id": "attempt-2-2", + "attempt_number": 2, + "message": "provider unavailable", + "exception_class": "RuntimeException" + }, + "recorded_at": "2026-11-12T00:00:07+00:00" + }, + { + "sequence": 9, + "event_type": "WorkflowFailed", + "payload": { + "failed_step_kind": "activity", + "failed_step_sequence": 2 + }, + "recorded_at": "2026-11-12T00:00:08+00:00" + } + ], + "expected": { + "eligible": true, + "resume_step_sequence": 2, + "completed_count": 1, + "reused_event_types": [ + "ActivityCompleted" + ] + }, + "refusals": [ + { + "event_index": 5, + "field": "activity_execution_id", + "value": "substituted-execution", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 5, + "field": "activity_attempt_id", + "value": "substituted-attempt", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 5, + "field": "retry_task_id", + "value": "substituted-task", + "reason": "invalid_activity_retry_history" + }, + { + "event_index": 5, + "field": "retry_after_attempt", + "value": 9, + "reason": "invalid_activity_retry_history" + } + ] + }, + "expected": { + "completed": false, + "result": null, + "commands": [ + { + "type": "schedule_activity", + "activity_type": "Tests\\Fixtures\\V2\\TestRedriveSecondActivity", + "payload_codec": "avro" + } + ] + } +} diff --git a/tests/Fixtures/V2/TestForwardAdmissionWorkflow.php b/tests/Fixtures/V2/TestForwardAdmissionWorkflow.php new file mode 100644 index 000000000..f371be17a --- /dev/null +++ b/tests/Fixtures/V2/TestForwardAdmissionWorkflow.php @@ -0,0 +1,21 @@ +execute(new Obligation('first', 'test.operation', '1', [ + 'mode' => $mode, + ])); + } +} diff --git a/tests/Fixtures/V2/TestRedriveRetryWorkflow.php b/tests/Fixtures/V2/TestRedriveRetryWorkflow.php new file mode 100644 index 000000000..ea08c1a69 --- /dev/null +++ b/tests/Fixtures/V2/TestRedriveRetryWorkflow.php @@ -0,0 +1,20 @@ +execute(new Obligation('first', 'test.operation', '1', [ + 'mode' => $mode, + ]), new ActivityOptions(maxAttempts: 1)); + if ($child !== null) { + $scope->participate('delivery', ScopeIdentity::fromArray($child)); + } + $scope->execute(new Obligation('second', 'test.operation', '1', [])); + return [ + 'status' => 'done', + ]; + } catch (Throwable) { + $this->compensate(); + return $scope->verifyCompensation(); + } + } +} diff --git a/tests/Fixtures/V2/TestScopeBoundaryChildWorkflow.php b/tests/Fixtures/V2/TestScopeBoundaryChildWorkflow.php new file mode 100644 index 000000000..22da1a69e --- /dev/null +++ b/tests/Fixtures/V2/TestScopeBoundaryChildWorkflow.php @@ -0,0 +1,32 @@ +execute(new Obligation('child', 'test.operation', '1', [])); + signal('finish'); + return [ + 'status' => 'done', + ]; + } catch (Throwable) { + $this->compensate(); + return $scope->verifyCompensation(); + } + } +} diff --git a/tests/Fixtures/V2/TestScopeRunBoundaryWorkflow.php b/tests/Fixtures/V2/TestScopeRunBoundaryWorkflow.php new file mode 100644 index 000000000..a74b22812 --- /dev/null +++ b/tests/Fixtures/V2/TestScopeRunBoundaryWorkflow.php @@ -0,0 +1,51 @@ +execute(new Obligation('first', 'test.operation', '1', [ + 'mode' => $mode === 'unknown' ? 'lost-ack' : 'complete', + ]), new ActivityOptions(maxAttempts: 1)); + if ($mode === 'continue') { + if ($childIdentity !== null) { + $scope->participate('delivery', ScopeIdentity::fromArray($childIdentity)); + self::select([ + 'child' => static fn () => self::child( + TestScopeBoundaryChildWorkflow::class, + new ChildWorkflowOptions(parentClosePolicy: ParentClosePolicy::Abandon), + $childIdentity + ), + 'ready' => static fn () => self::timer(0), + ]); + } + return self::continueAsNew($identity, $mode, 1, $childIdentity); + } + self::activity(TestRedriveSecondActivity::class, new ActivityOptions(maxAttempts: 1), 'boundary'); + } else { + signal('resume'); + } + // This is new work, so redrive's reused activity prefix cannot authorize it in the old scope. + $scope->execute(new Obligation('second', 'test.operation', '1', [])); + return [ + 'status' => 'done', + ]; + } +} diff --git a/tests/Fixtures/V2/TestScopedOperation.php b/tests/Fixtures/V2/TestScopedOperation.php new file mode 100644 index 000000000..ef5c94a0f --- /dev/null +++ b/tests/Fixtures/V2/TestScopedOperation.php @@ -0,0 +1,54 @@ +operationId; + if (! Cache::has($key)) { + Cache::forever($key, $context->obligation->key); + Cache::increment('test:scope:effect-count'); + } + if (($context->obligation->input['mode'] ?? null) === 'lost-ack') { + return ForwardResult::unknown('ack-lost'); + } + return ForwardResult::observed([ + 'reference' => $context->operationId, + ], ['provider:' . $context->operationId]); + } + + public function compensate(OperationContext $context): Resolution + { + $scopedLog = Cache::get('test:scope:undo-scoped-log', []); + $scopedLog[] = $context->scope->key . '/' . $context->obligation->key; + Cache::forever('test:scope:undo-scoped-log', $scopedLog); + $log = Cache::get('test:scope:undo-log', []); + $log[] = $context->obligation->key; + Cache::forever('test:scope:undo-log', $log); + if ($context->obligation->key === 'first' && Cache::get('test:scope:unknown-first', false)) { + return Resolution::unknown('provider-unavailable'); + } + if (Cache::pull('test:scope:expire-native', false)) { + \Illuminate\Support\Carbon::setTestNow(now()->addMinutes(10)); + } + Cache::forget('test:scope:effect:' . $context->operationId); + return Resolution::verified( + ['provider-undo:' . $context->operationId], + ! Cache::get('test:scope:late-forward-possible', false) + ); + } +} diff --git a/tests/Fixtures/V2/TestWaitingScopeWorkflow.php b/tests/Fixtures/V2/TestWaitingScopeWorkflow.php new file mode 100644 index 000000000..b2eee45ea --- /dev/null +++ b/tests/Fixtures/V2/TestWaitingScopeWorkflow.php @@ -0,0 +1,30 @@ +execute(new Obligation('first', 'test.operation', '1', [])); + $scope->execute(new Obligation('second', 'test.operation', '1', [])); + self::await(static fn (): bool => false, conditionKey: 'business-result'); + return [ + 'status' => 'unexpected-completion', + ]; + } catch (Throwable) { + $this->compensate(); + return $scope->verifyCompensation(); + } + } +} diff --git a/tests/Support/CompensationScopeRecoveryProcessTest.php b/tests/Support/CompensationScopeRecoveryProcessTest.php new file mode 100644 index 000000000..055041699 --- /dev/null +++ b/tests/Support/CompensationScopeRecoveryProcessTest.php @@ -0,0 +1,91 @@ +input(); + $this->app->instance(HandlerRegistry::class, new HandlerRegistry([ + 'test.operation' => [ + '1' => new TestScopedOperation(), + ], + ])); + $this->app->instance(ScopeAuthorization::class, new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if ($scope->namespace !== 'client-a' || $principal !== 'operator-a') { + throw new \LogicException('Cold scope access refused.'); + } + return 'cold-policy-1'; + } + }); + $scope = ScopeIdentity::fromArray($input['scope']); + app(ScopeRecovery::class)->request($scope, 'cold-request', 'operator-a', 'cancel'); + for ($step = 0; $step < 20 && app(ScopeRepository::class)->inspect($scope)['status'] !== 'verified'; $step++) { + $this->artisan('queue:work', [ + '--once' => true, + '--queue' => 'scope-recovery-test', + '--sleep' => 0, + '--tries' => 1, + ])->assertExitCode(0); + } + $status = app(ScopeRepository::class)->inspect($scope)['status']; + $this->assertSame('verified', $status); + file_put_contents($input['report'], json_encode([ + 'pid' => getmypid(), + 'status' => $status, + ], JSON_THROW_ON_ERROR)); + } + + protected function getPackageProviders($app): array + { + return [WorkflowServiceProvider::class]; + } + + protected function defineEnvironment($app): void + { + $input = $this->input(); + $app['config']->set([ + 'database.default' => 'scope_process', + 'database.connections.scope_process' => $input['database'], + 'database.redis' => $input['redis'], + 'workflows.storage.connection' => 'scope_process', + 'workflows.v2.namespaces' => ['default', 'client-a'], + 'queue.default' => 'database', + 'queue.connections.database' => [ + 'driver' => 'database', + 'connection' => 'scope_process', + 'table' => 'jobs', + 'queue' => 'scope-recovery-test', + 'retry_after' => 90, + ], + 'cache.default' => 'redis', + 'cache.stores.redis' => $input['cache'], + 'cache.prefix' => $input['cache_prefix'], + 'logging.default' => 'null', + ]); + } + + private function input(): array + { + $path = getenv('DWF_SCOPE_RECOVERY_INPUT'); + if (! is_string($path) || $path === '') { + $this->markTestSkipped('Invoked only by the retained-scope integration control.'); + } + return json_decode(file_get_contents($path), true, flags: JSON_THROW_ON_ERROR); + } +} diff --git a/tests/Unit/V2/Compensation/ChildScopeTest.php b/tests/Unit/V2/Compensation/ChildScopeTest.php new file mode 100644 index 000000000..88eabd68f --- /dev/null +++ b/tests/Unit/V2/Compensation/ChildScopeTest.php @@ -0,0 +1,93 @@ +scope('order'); + $child = $this->scope('shipment'); + $children = new ChildScopes(); + $id = $children->register($parent, 'origin', 'delivery', $child); + $this->assertSame($id, $children->register($parent, 'origin', 'delivery', $child)); + $this->expectException(LogicException::class); + $children->register($parent, 'origin', 'delivery', new ScopeIdentity('shop', 'client-a', 'shipment', '2')); + } + + public function testExistingGraphIsLockedBeforeAnyDuplicateInsert(): void + { + $parent = $this->scope('order'); + $child = $this->scope('shipment'); + $children = new ChildScopes(); + $children->register($parent, 'origin', 'delivery', $child); + $queries = []; + ConfiguredV2Models::connection()->listen(static function (QueryExecuted $query) use (&$queries): void { + if (str_contains($query->sql, 'workflow_compensation_graphs')) { + $queries[] = strtolower($query->sql); + } + }); + $children->register($parent, 'origin', 'delivery', $child); + $this->assertStringStartsWith('select', $queries[0]); + $this->assertSame([], array_filter($queries, static fn (string $sql): bool => str_starts_with($sql, 'insert'))); + } + + public function testCycleAndSecondNestingLevelAreRefused(): void + { + $children = new ChildScopes(); + $children->register($this->scope('order'), 'order-run', 'delivery', $this->scope('shipment')); + foreach ([[$this->scope('shipment'), $this->scope('order')], + [$this->scope('shipment'), $this->scope('return')]] as [$parent, $child]) { + try { + $children->register($parent, 'shipment-run', 'child', $child); + $this->fail('Unsupported or cyclic scope nesting was admitted.'); + } catch (LogicException) { + $this->assertSame(1, count((new ScopeRepository())->inspect($this->scope('order'))['obligations'])); + } + } + } + + public function testForeignNamespaceCannotBeAChild(): void + { + $this->expectException(LogicException::class); + (new ChildScopes())->register( + $this->scope('order'), + 'order-run', + 'delivery', + new ScopeIdentity('shop', 'client-b', 'shipment', '1') + ); + } + + public function testChildFanOutLimitCannotLeaveAPartialParticipant(): void + { + $children = new ChildScopes(); + $parent = $this->scope('order'); + for ($index = 0; $index < ChildScopes::MAX_CHILDREN; ++$index) { + $children->register($parent, 'order-run', 'delivery-' . $index, $this->scope('shipment-' . $index)); + } + try { + $children->register($parent, 'order-run', 'overflow', $this->scope('overflow')); + $this->fail('The child fan-out limit was exceeded.'); + } catch (LogicException) { + $this->assertCount(ChildScopes::MAX_CHILDREN, (new ScopeRepository())->inspect($parent)['obligations']); + $this->assertSame(ChildScopes::MAX_CHILDREN, \Workflow\V2\Support\ConfiguredV2Models::connection() + ->table('workflow_compensation_children') + ->count()); + } + } + + private function scope(string $key): ScopeIdentity + { + return new ScopeIdentity('shop', 'client-a', $key, '1'); + } +} diff --git a/tests/Unit/V2/Compensation/ScopeDescriptorTest.php b/tests/Unit/V2/Compensation/ScopeDescriptorTest.php new file mode 100644 index 000000000..eff937127 --- /dev/null +++ b/tests/Unit/V2/Compensation/ScopeDescriptorTest.php @@ -0,0 +1,142 @@ +assertSame($original->id(), (new ScopeIdentity('shop', 'client-a', 'order-42', '1'))->id()); + foreach ([['other', 'client-a', 'order-42', '1'], + ['shop', 'client-b', 'order-42', '1'], + ['shop', 'client-a', 'order-42', '2'], + ] as $parts) { + $this->assertNotSame($original->id(), (new ScopeIdentity(...$parts))->id()); + } + } + + public function testDescriptorCanonicalizesMapsWithoutReorderingLists(): void + { + $a = new Obligation('book', 'carrier.cancel', '1', [ + 'b' => 2, + 'a' => [ + '01' => 1, + '001' => 2, + ], + ]); + $b = new Obligation('book', 'carrier.cancel', '1', [ + 'a' => [ + '001' => 2, + '01' => 1, + ], + 'b' => 2, + ]); + $this->assertSame($a->fingerprint(), $b->fingerprint()); + $this->assertNotSame( + (new Obligation('book', 'carrier.cancel', '1', [1, 2]))->fingerprint(), + (new Obligation('book', 'carrier.cancel', '1', [2, 1]))->fingerprint() + ); + $this->assertSame($a->fingerprint(), Obligation::fromArray($a->toArray())->fingerprint()); + } + + public function testPersistedVersionOneSchemasHaveStableIdentityAndFingerprintVectors(): void + { + $scope = new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + $obligation = new Obligation('book', 'carrier.cancel', '1', []); + $this->assertSame('d0f6748c0bcd9166d88ca96e6e60c5d118108ee7ec1e9a13339ceb67977a92bb', $scope->id()); + $this->assertSame( + '78c1a87680c6374f612fad9e4ae34f9db0e139868bc9f7df3e4605b78129d260', + $obligation->fingerprint() + ); + $this->assertSame( + ['schemaVersion', 'application', 'namespace', 'key', 'generation'], + array_keys($scope->toArray()) + ); + $this->assertSame( + ['schemaVersion', 'key', 'handlerType', 'handlerVersion', 'input'], + array_keys($obligation->toArray()) + ); + $this->assertSame([ + 'schemaVersion' => 1, + 'observed' => false, + 'value' => [], + 'evidence' => [], + 'reason' => 'pending', + ], ForwardResult::unknown('pending')->toArray()); + $this->assertSame([ + 'schemaVersion' => 1, + 'verified' => false, + 'reason' => 'pending', + 'evidence' => [], + 'forwardSettled' => false, + ], Resolution::unknown('pending')->toArray()); + foreach ([0, 2, '1', null] as $version) { + foreach ([$scope, $obligation] as $value) { + $payload = $value->toArray(); + $payload['schemaVersion'] = $version; + try { + $value::fromArray($payload); + $this->fail('Unsupported schema was accepted.'); + } catch (InvalidArgumentException) { + $this->addToAssertionCount(1); + } + } + } + } + + public function testExecutableClassNameCannotBeAHandlerIdentifier(): void + { + $this->expectException(InvalidArgumentException::class); + new Obligation('book', 'App\\DeleteEverything', '1', []); + } + + public function testMutableObjectsCannotEnterTheDescriptor(): void + { + $this->expectException(InvalidArgumentException::class); + new Obligation('book', 'carrier.cancel', '1', [ + 'input' => new \stdClass(), + ]); + } + + public function testEvidenceReferenceLimitIsValidatedBeforeAcceptingProviderResults(): void + { + $references = array_fill(0, 33, 'proof'); + foreach ([ + static fn () => ForwardResult::observed([], $references), + static fn () => Resolution::verified($references, true), + ] as $result) { + try { + $result(); + $this->fail('The result admitted more evidence than its bounded journal envelope supports.'); + } catch (InvalidArgumentException $error) { + $this->assertStringContainsString('evidence reference limit', $error->getMessage()); + } + } + } + + public function testEvidenceReferencesCannotHideUnboundedMetadataInMapKeys(): void + { + $this->expectException(InvalidArgumentException::class); + ForwardResult::observed([], [ + 'provider-metadata' => 'proof', + ]); + } + + public function testOversizedDescriptorIsRejectedBeforeStorage(): void + { + $this->expectException(InvalidArgumentException::class); + new Obligation('book', 'carrier.cancel', '1', [ + 'input' => str_repeat('x', 16385), + ]); + } +} diff --git a/tests/Unit/V2/Compensation/ScopeProgressTest.php b/tests/Unit/V2/Compensation/ScopeProgressTest.php new file mode 100644 index 000000000..6acf7846e --- /dev/null +++ b/tests/Unit/V2/Compensation/ScopeProgressTest.php @@ -0,0 +1,104 @@ +registered(); + $progress = new ScopeProgress(); + $this->assertFalse($progress->admitForward($scope, $id, 'native-attempt-1')['uncertain']); + $progress->forwardResult($scope, $id, 'native-attempt-1', ForwardResult::unknown('ack-missing')); + $this->assertTrue($progress->admitForward($scope, $id, 'native-attempt-2')['uncertain']); + (new ScopeRepository())->request($scope, 'request', 'operator', 'cancel'); + $this->expectException(LogicException::class); + $progress->admitForward($scope, $id, 'native-attempt-3'); + } + + public function testReverseProgressStopsAtUnknownAndDoesNotRepeatVerifiedWork(): void + { + [$scope, $first] = $this->registered(); + $repository = new ScopeRepository(); + $second = $repository->register($scope, new Obligation('second', 'carrier.cancel', '1', [])); + $repository->request($scope, 'request', 'operator', 'cancel'); + $progress = new ScopeProgress(); + $progress->beginCompensation($scope, $second, 'native-undo-1'); + $progress->resolve($scope, $second, 'native-undo-1', Resolution::unknown('provider-unavailable')); + try { + $progress->beginCompensation($scope, $first, 'native-undo-2'); + $this->fail('Earlier compensation advanced past unresolved work.'); + } catch (LogicException) { + $this->assertSame('requested', $repository->inspect($scope)['status']); + } + $progress->beginCompensation($scope, $second, 'native-undo-3'); + $progress->resolve($scope, $second, 'native-undo-3', Resolution::verified(['proof-second'], true)); + $this->assertNull($progress->beginCompensation($scope, $second, 'native-undo-4')); + $progress->beginCompensation($scope, $first, 'native-undo-5'); + $progress->resolve($scope, $first, 'native-undo-5', Resolution::verified(['proof-first'], true)); + $this->assertSame('verified', $progress->finish($scope)['status']); + } + + public function testUnknownForwardCannotBecomeVerifiedWithoutSettledEvidence(): void + { + [$scope, $id] = $this->registered(); + $progress = new ScopeProgress(); + $progress->admitForward($scope, $id, 'native-forward'); + (new ScopeRepository())->request($scope, 'request', 'operator', 'cancel'); + $progress->beginCompensation($scope, $id, 'native-undo'); + $this->expectException(LogicException::class); + $progress->resolve($scope, $id, 'native-undo', Resolution::verified(['undo-ack'], false)); + } + + public function testCompensationCannotOverwriteAnotherNativeAttempt(): void + { + [$scope, $id] = $this->registered(); + (new ScopeRepository())->request($scope, 'request', 'operator', 'cancel'); + $progress = new ScopeProgress(); + $progress->beginCompensation($scope, $id, 'native-undo-1'); + $progress->beginCompensation($scope, $id, 'native-undo-2'); + $this->expectException(LogicException::class); + $progress->resolve($scope, $id, 'native-undo-1', Resolution::verified(['proof'], true)); + } + + public function testMaximumEvidenceReferencesFitForwardAndCompensationJournals(): void + { + [$scope, $id] = $this->registered(); + $references = array_fill(0, 32, 'p' . str_repeat('/', 127)); + $progress = new ScopeProgress(); + $repository = new ScopeRepository(); + $progress->admitForward($scope, $id, str_repeat('f', 26)); + $progress->forwardResult($scope, $id, str_repeat('f', 26), ForwardResult::observed([], $references)); + $repository->request($scope, 'request', 'operator', 'cancel'); + $progress->beginCompensation($scope, $id, str_repeat('c', 26)); + $progress->resolve($scope, $id, str_repeat('c', 26), Resolution::verified($references, true)); + $records = array_values(array_filter( + $repository->journal($scope), + static fn (array $row): bool => in_array($row['kind'], ['forward_outcome', 'compensation_outcome'], true) + )); + $this->assertCount(2, $records); + foreach ($records as $row) { + $this->assertSame($references, json_decode($row['data'], true, flags: JSON_THROW_ON_ERROR)['evidence']); + } + $this->assertSame('verified', $progress->finish($scope)['status']); + } + + private function registered(): array + { + $scope = new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + $repository = new ScopeRepository(); + $repository->open($scope, 'original-run'); + return [$scope, $repository->register($scope, new Obligation('book', 'carrier.cancel', '1', []))]; + } +} diff --git a/tests/Unit/V2/Compensation/ScopeRecoveryTest.php b/tests/Unit/V2/Compensation/ScopeRecoveryTest.php new file mode 100644 index 000000000..40dedc801 --- /dev/null +++ b/tests/Unit/V2/Compensation/ScopeRecoveryTest.php @@ -0,0 +1,111 @@ +assertSame($a->recoveryInstanceId(), ScopeIdentity::fromArray($a->toArray())->recoveryInstanceId()); + $this->assertNotSame( + $a->recoveryInstanceId(), + (new ScopeIdentity('shop', 'client-b', 'order-42', '1'))->recoveryInstanceId() + ); + $this->assertNotSame( + $a->recoveryInstanceId(), + (new ScopeIdentity('shop', 'client-a', 'order-42', '2'))->recoveryInstanceId() + ); + } + + public function testRequestIsBoundToItsOriginalCallerAndContent(): void + { + [$scope, $router] = $this->open(); + $first = $router->request($scope, 'request-1', 'operator-a', 'cancel-order'); + $again = $router->request($scope, 'request-1', 'operator-a', 'cancel-order'); + $this->assertSame($first, $again); + $this->expectException(\LogicException::class); + $router->request($scope, 'request-1', 'operator-b', 'cancel-order'); + } + + public function testAnActiveOriginUsesNativeCooperativeCancellation(): void + { + [$scope, $router, $runId] = $this->open(); + $result = $router->request($scope, 'request-1', 'operator-a', 'cancel-order'); + $this->assertSame('origin', $result['route']); + $this->assertSame($runId, $result['runId']); + $this->assertSame('requested', (new ScopeRepository())->inspect($scope)['status']); + $this->assertNull(\Workflow\V2\Models\WorkflowInstance::query()->find($scope->recoveryInstanceId())); + } + + public function testClosedOriginStartsOneCanonicalRecoveryForDifferentRequests(): void + { + [$scope, $router, $runId] = $this->open(); + \Workflow\V2\Models\WorkflowRun::query()->findOrFail($runId)->forceFill([ + 'status' => 'completed', + 'closed_at' => now(), + ])->save(); + $first = $router->request($scope, 'request-1', 'operator-a', 'cancel-order'); + $second = $router->request($scope, 'request-2', 'operator-a', 'cancel-order'); + $this->assertSame('recovery', $first['route']); + $this->assertSame($first['runId'], $second['runId']); + $this->assertSame( + 1, + \Workflow\V2\Models\WorkflowRun::query()->where( + 'workflow_instance_id', + $scope->recoveryInstanceId() + )->count() + ); + } + + public function testDeniedRequestCannotCloseTheScopeOrCreateRecovery(): void + { + [$scope, $router] = $this->open(); + try { + $router->request($scope, 'request-1', 'intruder', 'cancel-order'); + $this->fail('An unauthorized request was accepted.'); + } catch (\LogicException) { + $this->assertSame('open', (new ScopeRepository())->inspect($scope)['status']); + $this->assertSame( + 0, + \Workflow\V2\Support\ConfiguredV2Models::connection()->table('workflow_compensation_requests')->count() + ); + } + } + + private function open(): array + { + config([ + 'workflows.v2.task_dispatch_mode' => 'poll', + ]); + $scope = new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + $origin = WorkflowStub::make(TestSignalWorkflow::class, 'origin-workflow', 'client-a'); + $origin->start(); + (new ScopeRepository())->open($scope, $origin->runId()); + $authorization = new class() implements ScopeAuthorization { + public function assertAllowed(ScopeIdentity $scope, string $principal, string $operation): string + { + if (! in_array($principal, ['operator-a', 'operator-b'], true) || $scope->namespace !== 'client-a') { + throw new \LogicException('Not authorized for this scope.'); + } + return 'test-policy-1'; + } + }; + return [ + $scope, + new ScopeRecovery(new ScopeRepository(), $authorization, app(WorkflowControlPlane::class)), + $origin->runId(), + ]; + } +} diff --git a/tests/Unit/V2/Compensation/ScopeRepositoryTest.php b/tests/Unit/V2/Compensation/ScopeRepositoryTest.php new file mode 100644 index 000000000..01892289b --- /dev/null +++ b/tests/Unit/V2/Compensation/ScopeRepositoryTest.php @@ -0,0 +1,168 @@ +scope(); + $repository = new ScopeRepository(); + $repository->open($scope, 'original-run'); + $first = $repository->register($scope, new Obligation('book', 'carrier.cancel', '1', [ + 'reference' => 'booking-1', + ])); + $again = (new ScopeRepository())->register($scope, new Obligation('book', 'carrier.cancel', '1', [ + 'reference' => 'booking-1', + ])); + $this->assertSame($first, $again); + $this->assertCount(1, $repository->inspect($scope)['obligations']); + $this->assertSame(['opened', 'registered'], array_column($repository->journal($scope), 'kind')); + } + + public function testAnExistingScopeIsLockedBeforeAnyDuplicateInsert(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'original-run'); + $queries = []; + ConfiguredV2Models::connection()->listen(static function (QueryExecuted $query) use (&$queries): void { + if (str_contains($query->sql, 'workflow_compensation_scopes')) { + $queries[] = strtolower($query->sql); + } + }); + $repository->open($this->scope(), 'original-run'); + $this->assertStringStartsWith('select', $queries[0]); + $this->assertSame([], array_filter($queries, static fn (string $sql): bool => str_starts_with($sql, 'insert'))); + } + + public function testConflictingDuplicateDoesNotReplaceOriginalDescriptor(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'original-run'); + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '1', [])); + try { + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '2', [])); + $this->fail('Conflicting descriptor was adopted.'); + } catch (LogicException) { + $this->assertSame( + '1', + $repository->inspect($this->scope())['obligations'][0]['descriptor']['handlerVersion'] + ); + } + } + + public function testScopeGenerationAndNamespaceCannotShareObligations(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'run-a'); + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '1', [])); + foreach ([ + new ScopeIdentity('shop', 'client-b', 'order-42', '1'), + new ScopeIdentity('shop', 'client-a', 'order-42', '2'), + ] as $other) { + $repository->open($other, 'run-b'); + $this->assertSame([], $repository->inspect($other)['obligations']); + } + } + + public function testRegistrationAndItsJournalRollBackTogether(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'original-run'); + $connection = ConfiguredV2Models::connection(); + $connection->beginTransaction(); + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '1', [])); + $connection->rollBack(); + $this->assertSame([], $repository->inspect($this->scope())['obligations']); + $this->assertSame(['opened'], array_column($repository->journal($this->scope()), 'kind')); + } + + public function testOriginalRunCannotBeSilentlyReplaced(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'original-run'); + $this->expectException(\Workflow\Exceptions\NonRetryableException::class); + $this->expectExceptionMessage('scope_owned_by_another_run'); + $repository->open($this->scope(), 'replacement-run'); + } + + public function testJournalWriteFailureCannotLeaveAnUnjournaledObligation(): void + { + $repository = new ScopeRepository(); + $repository->open($this->scope(), 'original-run'); + $fail = true; + ConfiguredV2Models::connection()->beforeExecuting(static function (string $sql) use (&$fail): void { + if ($fail && str_starts_with($sql, 'insert') && str_contains($sql, 'workflow_compensation_journal')) { + throw new \RuntimeException('injected journal failure'); + } + }); + try { + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '1', [])); + $this->fail('Expected the injected failure.'); + } catch (\RuntimeException $error) { + $this->assertSame('injected journal failure', $error->getMessage()); + } finally { + $fail = false; + } + $this->assertSame([], $repository->inspect($this->scope())['obligations']); + $this->assertCount(1, $repository->journal($this->scope())); + $repository->register($this->scope(), new Obligation('book', 'carrier.cancel', '1', [])); + $this->assertSame(1, $repository->inspect($this->scope())['obligations'][0]['position']); + } + + public function testOpeningLiveStorageFromWorkflowReplayIsRefusedBeforeWriting(): void + { + $scope = $this->scope(); + $fiber = new Fiber(static function () use ($scope): void { + WorkflowFiberContext::enter(); + try { + (new ScopeRepository())->open($scope, 'original-run'); + } finally { + WorkflowFiberContext::leave(); + } + }); + try { + $fiber->start(); + $this->fail('Workflow replay opened live compensation storage.'); + } catch (LogicException $error) { + $this->assertStringContainsString('never in workflow replay', $error->getMessage()); + } + $this->assertSame(0, ConfiguredV2Models::connection()->table('workflow_compensation_scopes')->count()); + $this->assertSame(0, ConfiguredV2Models::connection()->table('workflow_compensation_journal')->count()); + } + + public function testObligationLimitCannotLeaveAPartialRegistration(): void + { + $repository = new ScopeRepository(); + $scope = $this->scope(); + $repository->open($scope, 'original-run'); + for ($index = 0; $index < ScopeRepository::MAX_OBLIGATIONS; ++$index) { + $repository->register($scope, new Obligation('work-' . $index, 'carrier.cancel', '1', [])); + } + $journal = $repository->journal($scope); + try { + $repository->register($scope, new Obligation('overflow', 'carrier.cancel', '1', [])); + $this->fail('The obligation limit was exceeded.'); + } catch (LogicException) { + $this->assertCount(ScopeRepository::MAX_OBLIGATIONS, $repository->inspect($scope)['obligations']); + $this->assertSame($journal, $repository->journal($scope)); + } + } + + private function scope(): ScopeIdentity + { + return new ScopeIdentity('shop', 'client-a', 'order-42', '1'); + } +} diff --git a/tests/Unit/V2/FailedRunRedrivePlanTest.php b/tests/Unit/V2/FailedRunRedrivePlanTest.php index 4cef87267..76f2b1049 100644 --- a/tests/Unit/V2/FailedRunRedrivePlanTest.php +++ b/tests/Unit/V2/FailedRunRedrivePlanTest.php @@ -202,6 +202,168 @@ public function testRefusesNonFailedRun(): void /** * @return list}> */ + public function testNativeRetriesCanPrecedeTheRecordedFailedActivity(): void + { + $plan = FailedRunRedrivePlan::forRun($this->runWithHistory($this->retryHistory())); + self::assertTrue($plan['eligible'], (string) $plan['reason']); + self::assertSame(2, $plan['resume_step_sequence']); + } + + public function testRetryIdentitySubstitutionIsRefused(): void + { + foreach (['activity_execution_id', 'activity_attempt_id', 'retry_after_attempt', 'retry_task_id'] as $field) { + $history = $this->retryHistory(); + $history[6]['payload'][$field] = $field === 'retry_after_attempt' ? 9 : 'substituted'; + $plan = FailedRunRedrivePlan::forRun($this->runWithHistory($history)); + self::assertFalse($plan['eligible']); + self::assertSame('invalid_activity_retry_history', $plan['reason']); + } + } + + public function testNativeActiveAdoptionIsHarmlessOnlyWhenBoundToTheSameRunAndCommand(): void + { + $plan = FailedRunRedrivePlan::forRun($this->adoptionRun()); + self::assertTrue($plan['eligible'], (string) $plan['reason']); + self::assertSame(2, $plan['resume_step_sequence']); + } + + public function testSubstitutedActiveStartAdoptionCannotAuthorizeRedrive(): void + { + foreach ([ + ['outcome', 'started_new'], + ['workflow_instance_id', 'other-instance'], + ['workflow_run_id', 'other-run'], + ['workflow_class', 'OtherWorkflow'], + ['workflow_type', 'other.type'], + ['workflow_command_id', 'other-command'], + ['rejection_reason', 'refused'], + ['command', null], + ['command.id', 'other-command'], + ['command.type', 'signal'], + ['command.status', 'rejected'], + ['command.outcome', 'started_new'], + ['command.target_scope', 'run'], + ['command.resolved_run_id', 'other-run'], + ] as [$field, $replacement]) { + $run = $this->adoptionRun(); + $event = $run->historyEvents[2]; + $payload = $event->payload; + $keys = explode('.', $field); + if (count($keys) === 2) { + $payload[$keys[0]][$keys[1]] = $replacement; + } else { + $payload[$field] = $replacement; + } + $event->payload = $payload; + $plan = FailedRunRedrivePlan::forRun($run); + self::assertFalse($plan['eligible'], $field); + self::assertSame('unexpected_start', $plan['reason'], $field); + } + foreach (['workflow_run_id', 'workflow_command_id'] as $field) { + $run = $this->adoptionRun(); + $run->historyEvents[2]->setAttribute($field, 'substituted'); + self::assertSame('unexpected_start', FailedRunRedrivePlan::forRun($run)['reason']); + } + } + + private function adoptionRun(): WorkflowRun + { + $history = $this->linearFailureHistory(); + array_splice($history, 2, 0, [[ + 'event_type' => HistoryEventType::StartAccepted, + 'payload' => [ + 'workflow_instance_id' => 'instance-a', + 'workflow_run_id' => 'run-a', + 'workflow_class' => 'TestWorkflow', + 'workflow_type' => 'test.workflow', + 'workflow_command_id' => 'adoption-command', + 'outcome' => 'returned_existing_active', + 'command' => [ + 'id' => 'adoption-command', + 'type' => 'start', + 'target_scope' => 'instance', + 'resolved_run_id' => 'run-a', + 'status' => 'accepted', + 'outcome' => 'returned_existing_active', + ], + ], + ]]); + $run = $this->runWithHistory($history); + $run->forceFill([ + 'id' => 'run-a', + 'workflow_instance_id' => 'instance-a', + 'workflow_class' => 'TestWorkflow', + 'workflow_type' => 'test.workflow', + ]); + $run->historyEvents[2]->forceFill([ + 'workflow_run_id' => 'run-a', + 'workflow_command_id' => 'adoption-command', + ]); + return $run; + } + + private function retryHistory(): array + { + $history = $this->linearFailureHistory(); + array_splice($history, 4, 1, [ + [ + 'event_type' => HistoryEventType::ActivityScheduled, + 'payload' => [ + 'sequence' => 2, + 'activity_type' => 'second', + 'activity_execution_id' => 'execution-2', + ], + ], + [ + 'event_type' => HistoryEventType::ActivityStarted, + 'payload' => [ + 'sequence' => 2, + 'activity_type' => 'second', + 'activity_execution_id' => 'execution-2', + 'activity_attempt_id' => 'attempt-1', + 'attempt_number' => 1, + 'workflow_task_id' => 'task-1', + ], + ], + [ + 'event_type' => HistoryEventType::ActivityRetryScheduled, + 'payload' => [ + 'sequence' => 2, + 'activity_type' => 'second', + 'activity_execution_id' => 'execution-2', + 'activity_attempt_id' => 'attempt-1', + 'retry_after_attempt_id' => 'attempt-1', + 'retry_after_attempt' => 1, + 'retry_of_task_id' => 'task-1', + 'retry_task_id' => 'task-2', + 'max_attempts' => 2, + ], + ], + [ + 'event_type' => HistoryEventType::ActivityStarted, + 'payload' => [ + 'sequence' => 2, + 'activity_type' => 'second', + 'activity_execution_id' => 'execution-2', + 'activity_attempt_id' => 'attempt-2', + 'attempt_number' => 2, + 'workflow_task_id' => 'task-2', + ], + ], + [ + 'event_type' => HistoryEventType::ActivityFailed, + 'payload' => [ + 'sequence' => 2, + 'activity_type' => 'second', + 'activity_execution_id' => 'execution-2', + 'activity_attempt_id' => 'attempt-2', + 'attempt_number' => 2, + ], + ], + ]); + return $history; + } + private function linearFailureHistory(): array { return [ diff --git a/tests/Unit/V2/ReplayRegressionCorpusTest.php b/tests/Unit/V2/ReplayRegressionCorpusTest.php index 8f7d514fa..af16aabca 100644 --- a/tests/Unit/V2/ReplayRegressionCorpusTest.php +++ b/tests/Unit/V2/ReplayRegressionCorpusTest.php @@ -24,12 +24,14 @@ use Throwable; use Workflow\Serializers\Serializer; use Workflow\V2\Enums\HistoryEventType; +use Workflow\V2\Enums\RunStatus; use Workflow\V2\Models\WorkflowHistoryEvent; use Workflow\V2\Models\WorkflowRun; use Workflow\V2\Models\WorkflowServiceCall; use Workflow\V2\Support\ActivitySnapshot; use Workflow\V2\Support\CooperativeCancellationDelivery; use Workflow\V2\Support\ExecutionTracing; +use Workflow\V2\Support\FailedRunRedrivePlan; use Workflow\V2\Support\HistoryEventPayloadContract; use Workflow\V2\Support\PortableLocalActivityPreparation; use Workflow\V2\Support\PreparedLocalRetryPolicy; @@ -81,6 +83,9 @@ protected function tearDown(): void public function testFixtureExecutesThroughColdReplayRunner(array $fixture): void { $this->assertHistoryPayloadContract($fixture); + if (isset($fixture['redrive_plan'])) { + $this->assertRetriedActivityRedrivePlan($fixture); + } if (in_array( $fixture['id'], @@ -409,6 +414,40 @@ public static function replayRegressionFixtures(): array return $fixtures; } + private function assertRetriedActivityRedrivePlan(array $fixture): void + { + $scenario = $fixture['redrive_plan']; + $runWithHistory = static function (array $history) use ($scenario): WorkflowRun { + $run = new WorkflowRun([ + ...($scenario['source_run'] ?? []), + 'status' => RunStatus::Failed, + 'payload_codec' => 'avro', + ]); + $run->setRelation('historyEvents', collect($history)->map( + static fn (array $event): WorkflowHistoryEvent => new WorkflowHistoryEvent($event) + )); + return $run; + }; + $plan = FailedRunRedrivePlan::forRun($runWithHistory($scenario['source_history'])); + $expected = $scenario['expected']; + $this->assertSame($expected['eligible'], $plan['eligible'], $fixture['id'] . ': ' . $plan['reason']); + $this->assertNull($plan['reason']); + $this->assertSame($expected['resume_step_sequence'], $plan['resume_step_sequence']); + $this->assertCount($expected['completed_count'], $plan['completed']); + $this->assertSame($expected['reused_event_types'], array_map( + static fn (array $step): string => $step['event_type']->value, + $plan['reused_steps'] + )); + foreach ($scenario['refusals'] as $case) { + $history = $scenario['source_history']; + $history[$case['event_index']]['payload'][$case['field']] = $case['value']; + $rejected = FailedRunRedrivePlan::forRun($runWithHistory($history)); + $this->assertFalse($rejected['eligible'], $fixture['id'] . ': ' . $case['field']); + $this->assertSame($case['reason'], $rejected['reason']); + $this->assertSame([], $rejected['reused_steps']); + } + } + private function assertPreparedV1RefusesTheVersionedPolicy(array $fixture): void { $policy = $fixture['unqualified_preparation_policy'];