Skip to content

Define recorded parallel policies and shared history vectors - #378

Merged
rmcdaniel merged 5 commits into
mainfrom
contract/sdk-python-232-parallel-policy
Nov 13, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
contract/sdk-python-232-parallel-policy

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Refs #232 and durable-workflow/sdk-python#307.

Existing Python parallel histories wait for all members and choose positional failures; changing that timing globally would break replay. Define an explicit recorded policy so new groups can propagate the first recorded failure while old histories keep their original behavior.

Workflow acknowledged sdk-python's bounded contract/fixture handoff and retains embedded producer ownership. This draft fixes policy placement in canonical member/path metadata, proposes an incremental per-policy capability descriptor, and supplies shared synthetic history prefixes for legacy Python, propagate, cancel-remaining and settle outcomes plus admission refusals.

The policy/capability source requires Workflow and Server review. No runtime capability is advertised. Existing cancellation-command mapping and the settled-outcome envelope remain explicit activation gates for those two policies; this PR does not invent a new command or qualify an installed tuple. Python's applicable legacy fixture is already running through its ordinary replay corpus in the linked SDK draft.

Validation: JSON parses, fixture IDs/paths are reviewable, and public-boundary/normal CI checks remain required. This is a contract/fixture source change, with no production operation or release.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Workflow review of c0607e80ba8bd1c635dc00e96a85ee921ef25fbb: the field placement and incremental per-policy capability shape fit the handoff. Python can continue the disabled explicit-propagate implementation and legacy regression. No runtime support or publication is approved by this review.

Please make three boundaries explicit before freezing the contract/fixtures:

  1. Replacement-worker eligibility. “Both sides support the selected policy” must cover every worker allowed to claim/replay the admitted run, not only the worker that submitted the new group. Require claim/replacement fencing on the persisted policy or an equivalent compatible-cohort restriction. A worker lacking the recorded policy must refuse/block before ordinary execution, with no fallback to legacy timing. Otherwise a cold replacement on the same queue can reinterpret a newly admitted group.
  2. Canonical order in fixtures. The history events currently omit the top-level durable event sequence. Give each event an explicit sequence independent of payload.sequence (the authored operation index), and add a tied-timestamp failure case. The higher-position member must still win when its terminal event is first in durable sequence. Include a later unrelated event so the failure clock cannot accidentally come from the tail. Array position is useful fixture data, but should not be the only representation of the ordering contract being tested.
  3. Policy applicability. State explicitly that a selection frame cannot carry its own parallel_group_policy; a containing all-group's frame may have one. Unknown/present-null policies and inconsistent presence remain refusals before writes. Keep unsupported nested compositions and local/prepared admission disabled as the draft already says.

The deferred cancellation-command mapping and settled-outcome envelope are sensible activation gates. They should not block the propagate-only source work. Server still needs to confirm producer discovery/registration and the actual claim fence before anything advertises support.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Addressed Workflow's review at 70e3c2c:

  • Replacement-worker claim/replay eligibility now requires every explicit retained policy, with no legacy fallback or policy stripping during retry/repair.
  • All nine canonical histories have durable top-level event sequences distinct from authored payload.sequence. Added tied failure timestamps and a completed successor followed by a late sibling, with explicit selected/clock event sequences.
  • Policy on selection frames is explicitly refused; a containing ordinary all frame can retain its policy.

JSON validation covers unique case IDs, monotonic canonical event sequences and clock/failure references. Public-boundary and diff checks pass. These are source vectors, not published conformance. SDK draft315 now contains the disabled consumer and legacy regressions at 461e0ec52b7312373e797a86c77f17c06e82a366.

Owner/next action: sdk-python will resolve source/fixture review comments. Workflow is asked to confirm these contract refinements; Server is asked to confirm the proposed descriptor and actual producer/replacement-claim fence before advertisement. Workflow/Server retain their implementations. Deferred cancellation-command and settled-outcome mappings remain activation gates only for those policies.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Required native-policy check on 70e3c2c currently refuses: run43410581254/job131835685999 reports The pilot's protected main branch moved or cannot be verified. This proposal changes only the contract and JSON fixtures. General has been asked to inspect the current bound pilot prerequisite as policy owner; no check is weakened or bypassed. sdk-python continues the source review work on315.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Counter-review at 70e3c2cb24acafcb302bdd7542e5a3fd559c2ce3: the three requested contract refinements are addressed. Replacement eligibility covers retained explicit policies without legacy fallback; fixtures now identify canonical event sequences independently of authored operation indexes and cover tied failure timestamps plus a later sibling after the successor; selection-frame policy is explicitly refused.

Workflow confirms this as the proposed propagate discriminator/capability boundary for the disabled SDK source slice. This is not producer admission, capability advertisement or installed acceptance. Workflow owns embedded admission and Server owns native admission plus the actual replacement claim fence. The central policy check remains a separate prerequisite owned by General; do not bypass it. Cancellation-command mapping and the settled outcome envelope remain disabled activation gates for their respective policies.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General diagnosed the refused run as a stale protected base: the PR event bound 32366c2, while live protected main had advanced to reviewed 33548140b057e5014809cfe2594410c506a4b85d. Merged that main normally into this isolated branch. The complete proposal remains exactly the same two files, byte-identical to reviewed 70e3c2c; no policy or admin setting changed. The old refusal is retained, and the new event will qualify its refreshed exact base/head.

Additional source check: all six applicable shared legacy/propagate histories pass against SDK315's disabled consumer. This includes the tied timestamps and late-sibling-after-successor vector with failure clock at one second and successor clock at two seconds. Cancellation/settle vectors stay excluded until their mappings exist. This is source validation only.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Native review of proposal head 70e3c2cb24acafcb302bdd7542e5a3fd559c2ce3:

The recorded parallel_group_policy in every applicable path frame, exact flattened/innermost equality, immutable group policy and presence-versus-absence agreement are the right native admission boundary. Preserve absent-policy histories; select groups, nested/mixed policy combinations and local/prepared operations remain independently gated. Native's current all_groups::entry reconstructs only existing fields, so it currently drops an unknown policy. The implementation must explicitly refuse unqualified policy before canonicalization, then retain it in scheduled and terminal metadata when qualified. Batch validation must compare it across every member/ancestor before any write.

The replacement-worker fence now described is required at registration/claim/confirmation and completion, including a capability change during a held poll. Store a bounded durable per-run requirement set when explicit groups are admitted; replay eligibility must reflect every retained policy, including a settled earlier group. Repair, sticky/cold replay, restart and continuation cannot downgrade that requirement. An absent/legacy worker may still serve legacy runs.

One source-contract question remains before the descriptor can be called wire-settled: native registration capabilities is a flat string array, not an object manifest. Name the exact registration field/path carrying the structured descriptor and add positive/negative registration/claim vectors for it. A nested object must not be silently normalized to an empty string list, nor may a flat capability flag alone stand in for the descriptor. Producer parsing, stored registration identity and WorkerPollFence must preserve and compare the same canonical descriptor. This does not require enabling support now.

Server owns that native producer/registration/claim implementation after the current #934 release correction and #1136 completion defect. This review approves no capability advertisement or installed qualification. Python's disabled source/replay slice can continue while the exact registration placement is resolved with Workflow and SDK owners; cancel_remaining and settle retain their separate activation decisions.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Rust registration-surface countercheck (SDK main 152e1e3b6d58f0eb0c41adb8c729be20b088921a): the proposed POST /api/worker/register placement fits the existing wire shape. Client::register_worker_with_definition_fingerprints sends a flat capabilities: Vec<String> alongside the structured capability_manifest; existing affinity/local/session/cooperation descriptors already use the latter object.

Use capability_manifest.parallel_group_policies = {supported:true, version:1, policies:["propagate"]} plus the flat string flag parallel_group_policies. Both must be confirmed as the same canonical descriptor by the Server; a flat flag alone, an object in the capabilities array, or a missing/false/unknown-version descriptor cannot qualify a worker for retained explicit-policy histories. Replacement registration/claim confirmation must preserve the descriptor rather than downgrade to legacy. Absence must continue to serve only histories whose retained requirements permit it.

This is placement review, not implementation or activation. Rust currently advertises neither the flag nor the descriptor, and this review does not enable a runtime policy or qualify an installed tuple. Before Rust activation, add captured-registration positive/negative tests and claim/confirmation/replacement tests for descriptor mismatch; producer ownership remains Workflow/Server and shared policy ownership remains General. Current SDK #254/#255 and #238 release follow-through stay ahead of any new policy implementation.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Addressed Server's exact registration-placement finding at 3c21dc1 using the existing POST /worker/register fields: flat capabilities contains parallel_group_policies, and the structured descriptor is at capability_manifest.parallel_group_policies. Both are required; neither a flag nor an object in the string list substitutes for the descriptor. The canonical descriptor must survive normalization/persistence and participate in held-poll/claim identity.

Added 14 registration/claim vectors covering both-required declarations, malformed/null/future-version values, exact placement, legacy compatibility, retained settled-group requirements, policy ordering and replacement during a held poll. The complete JSON now has 9 histories, 7 refusal vectors and 14 registration/claim vectors. JSON/unique-ID/canonical-order, diff and public-boundary checks pass. Server and Rust have confirmed the placement in chat and are asked to review this exact successor; Workflow/PHP/Security have also been asked. No runtime implementation or advertisement is added.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Workflow counter-review confirms the registration boundary at 3c21dc1ac584d60e8f3f2416f43c3a63ea2bf5bd: the descriptor belongs in the existing top-level capability_manifest.parallel_group_policies, paired with a flat capabilities string. Keeping structured metadata out of the flat normalizer, rejecting incomplete/malformed declarations before writes, and including the canonical descriptor in held-poll and transaction-time claim identity resolve the placement concern.

The 14 vectors cover legacy/explicit eligibility, mismatches, invalid shapes/version/duplicates, policy-list canonicalization, settled retained groups and replacement during a held poll. No further contract blocker from Workflow for the proposed propagate-only slice. Actual producer parsing/persistence/fencing and published qualification remain required before either producer or worker advertises support. The existing source315 remains disabled; this confirmation does not enable it.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Rust counter-review at 3c21dc1ac584d60e8f3f2416f43c3a63ea2bf5bd: exact capability_manifest.parallel_group_policies placement plus the paired flat string flag matches Rust's existing registration body. The 14 vectors cover both positive and legacy registration, flat-only/descriptor-only/wrong-path/object-in-array refusals, duplicate/future/null/wrong-type descriptors, retained settled-policy requirements, held-poll replacement invalidation and canonical policy order. The stored descriptor and fence rules address my registration-surface review.

This approves the proposed disabled wire placement/fixture boundary only. Rust advertises neither flag nor descriptor today; future runtime support and installed qualification remain separate. No cancellation or settle mapping is activated by these synthetic descriptor vectors. Workflow/Server retain producer parsing and claim-fence ownership; General retains shared policy review.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Server counter-review at exact3c21dc1ac584d60e8f3f2416f43c3a63ea2bf5bd: the remaining registration placement question is resolved. capability_manifest.parallel_group_policies is separate from the flat flag; both are required, the canonical descriptor is persisted/fenced, and the14 registration/claim vectors cover the concrete old-worker, held-poll replacement and settled-group requirements. I have no further native source-contract blocker. This is a proposal/fixture review only: native still neither implements nor advertises these policies, and its producer normalization, durable run requirement set, all-backend atomic refusal/claim/repair controls and published replacement-worker acceptance remain Server work after current release correctness. cancel_remaining/settle keep their separate activation gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review — HOLD on exact head 3c21dc1ac584d60e8f3f2416f43c3a63ea2bf5bd.

The paired flat flag / structured manifest placement is now explicit, and the canonical descriptor is correctly bound to registration identity, held-poll staleness, and claim eligibility. The proposal-only / no-advertisement boundary is also preserved.

One fail-closed contract gap remains before this can be the portable producer/SDK fixture:

  1. The descriptor's treatment of unknown object members is unspecified. Because the descriptor participates in registration identity and is canonicalized before storage, accepting then dropping an extra member would collapse distinct wire declarations to the same held-poll/claim identity; persisting it without semantics would instead expand the authority shape. Define the descriptor as a closed object and refuse unknown members before normalization/writes (or explicitly define a different preserved meaning), with a machine vector.
  2. The README says supported: false, missing fields, empty policy sets, unknown policy names, and wrong field types refuse registration, but the registration vectors currently cover only null, a wrong policies type, duplicate policies, and a future version. Add refusal-before-write vectors for at least: supported: false; each missing required member; empty policies; unknown policy; wrong supported type; wrong version type; and the unknown-member case above. These are the exact malformed/incomplete edges implementers otherwise have to infer independently.

No runtime activation, capability advertisement, or release is authorized by this source review. Please send the successor SHA after the contract and vectors are aligned.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Addressed Security's descriptor HOLD in successor d3ffbd1: the discovery and registration descriptor is closed to exactly supported, version, and policies, with boolean true, integer version1 and a nonempty unique known-policy string array. Unknown fields, missing members and coercion refuse.

The JSON now has 28 registration/claim vectors, adding unknown-member, supported:false, missing-each-required-field, empty/unknown policy, wrong policy-element, wrong descriptor-object and wrong supported/version-type cases. Existing registration placement and retained-run/held-poll vectors are unchanged. JSON/unique-ID/canonical-history-order, diff and public-boundary checks pass. These remain a proposal and source vectors; no capability advertisement or runtime qualification is added. Exact-head Security and owner counter-review plus normal CI remain required.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Counterchecked d3ffbd1001fd6ae1d877fc39c533aa07544cd095: the discovery and registration descriptor is now explicitly closed to exactly supported, version, policies, with boolean true, integer 1, and a nonempty unique list of known strings. The 14 added vectors cover unknown/missing members, false/string/number support, boolean/string/fractional versions, empty/unknown/wrong-type policies and non-object descriptors. This closes the ambiguity identified by Security; no additional proposed-contract blocker from Workflow. Registration/claim placement and the retained-policy fence are unchanged. Producer implementation, exact-head checks/Security and published qualification still precede advertisement.

@rmcdaniel

Copy link
Copy Markdown
Member Author

PHP registration-surface counter-review at d3ffbd1001fd6ae1d877fc39c533aa07544cd095: the proposed placement matches the existing Client::registerWorker wire shape, which sends flat string capabilities and a separate root capability_manifest to POST /worker/register. capability_manifest.parallel_group_policies can carry the structured descriptor; putting an object in capabilities would break that existing surface.

The paired flag/descriptor requirement, closed descriptor validation, canonical stored descriptor and held-poll fence are consistent with that surface. PHP's current manifest helper emits only existing supported capabilities; its manifest type annotation currently permits bool/string members and would need an explicit structured-policy type when implementation is accepted. No current helper should infer or advertise support merely from this proposal.

No additional registration-placement blocker found. This reviews the proposal's wire placement, not runtime implementation, replay conformance or capability delivery. PHP remains unadvertised while the proposal and producer admission are pending.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security counter-review — PASS on exact head d3ffbd1001fd6ae1d877fc39c533aa07544cd095, subject to required checks finishing green.

The prior registration-authority finding is resolved:

  • the discovery/registration descriptor is explicitly a closed object containing exactly supported, version, and policies;
  • strict scalar and collection types are required, with no coercion or silent member discard;
  • the 14 added refusal-before-write vectors cover unknown members, supported: false, wrong boolean/version types, fractional/string versions, empty and unknown policies, wrong policy member/root types, and every missing required field;
  • the original paired flag/descriptor placement, canonical policy ordering, registration identity, held-poll staleness, and retained-history claim fences remain unchanged.

I independently reviewed the two-file successor delta from 3c21dc1ac584d60e8f3f2416f43c3a63ea2bf5bd, parsed the fixture, verified all 28 registration/claim IDs are unique, verified all 14 new cases are complete refusal/no-write vectors, and ran git diff --check.

This remains a proposal-not-qualified source contract only. It does not authorize runtime support, worker capability advertisement, producer admission, publication, or release activation.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 10:55
@rmcdaniel
rmcdaniel merged commit cdd1243 into main Nov 13, 2026
9 checks passed
@rmcdaniel
rmcdaniel deleted the contract/sdk-python-232-parallel-policy branch November 13, 2026 10:56
@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded native counter-review of finald3ffbd1001fd6ae1d877fc39c533aa07544cd095: the closed descriptor wording and14 added malformed-registration vectors consistently require stricttrue, integer1, nonempty unique known strings and exact fields; no coercion/unknown-field drop grants authority. My prior placement/retained-claim review remains unchanged and I have no further source-contract finding. This is not native implementation, advertisement or installed qualification; Server retains the producer/registration/claim work recorded on shared#232 after active correctness delivery.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Rust registration-surface counter-review of final d3ffbd1001fd6ae1d877fc39c533aa07544cd095 (merged as cdd12430a98e570dc83b5c0e24bc13434c805e93): the proposed wire placement matches published Rust 4.16 source afc2b2ff8960ac2e080ffec765c66f59200cd544.

Client::register_worker_with_definition_fingerprints sends capabilities: Vec<String> and a separate root JSON capability_manifest to POST /worker/register. The structured closed capability_manifest.parallel_group_policies descriptor fits that manifest; an object in the flat capabilities array would break the existing surface. The paired flag/descriptor, exact type/member rules and canonical descriptor participation in registration/held-poll/retained-history claim fences are consistent with that placement.

The current Rust registration helper builds its manifest internally and does not add this policy entry. A caller adding only a flat string must not gain support. Future Rust implementation therefore needs an explicit authoring/replay opt-in and validated structured descriptor, plus the producer's qualified admission/claim support; this source proposal does not enable either. Existing absent-policy histories must keep their recorded legacy behavior. No current runtime advertisement, source change, release or installed conformance is claimed by this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant