Skip to content

Add bounded checkpointed batch example - #334

Merged
rmcdaniel merged 2 commits into
mainfrom
sdk-python/218-checkpointed-batch
Nov 14, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
sdk-python/218-checkpointed-batch

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Add the runnable bounded-batch source example required by durable-workflow/.github#218. It reads the new acknowledged checkpoint accessor, binds its cursor to the complete business input and reconciles a separate synthetic provider receipt before acknowledging progress. Provider receipts bind the full input even when no checkpoint was received, so changed append/truncate/reorder requests cannot silently reuse a business ID.

Three explicit failure points cover before commit, after commit and after acknowledgement. Twenty-two source tests also cover both lost-response outcomes, invalid inputs/cursors, partial receipts, preservation of existing provider state and the actual importable handler through cooperative preflight/callback serialization. Blocking filesystem work uses asyncio.to_thread; the activity retains fixed retry and timeout budgets. The guide distinguishes the synthetic provider from real idempotency/lookup/fencing and does not promise atomic or exactly-once external effects.

Validation: source tests, Ruff, mypy and diff checks pass. No SDK runtime implementation changes. The example uses an unreleased API; no example, Worker, Server, package build/publication or installed qualification was selected. Actual published-package cold replacement remains separate unfinished acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security/source review PASS for exact 65a5b28e8ef8ddf0dd5b8ffb218eacbccd06352a.

The four-file example preserves the required separation between Server checkpoint authority and external business-effect authority:

  • a cursor is accepted only when its exact full-input fingerprint matches and its integer position is in range;
  • the provider key is stable across attempts and each receipt independently binds business ID/index, complete batch fingerprint and item;
  • commit or exact reconciliation happens before heartbeat acknowledgement;
  • an uncertain acknowledgement response is handled in both possible Server outcomes without treating the external effect as atomic with the checkpoint;
  • changed/append/truncate/reordered inputs, partial/conflicting receipts, invalid attempts/cursors and all three first-attempt failure boundaries refuse or reconcile without overwriting prior provider state;
  • the example states its synthetic/Linux/source-only limits and explicitly disclaims exactly-once effects.

Blocking filesystem work is kept out of the event loop, all activity/retry/deadline bounds are fixed, and the runtime SDK implementation is unchanged from accepted 333. The complete current-head source/package/quality checks are green; the still-running integration job remains a separate gate.

This verdict does not qualify a published SDK, real provider, Server persistence, cold-worker execution or example invocation.

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ordinary source/API review: HOLD for exact 65a5b28e8ef8ddf0dd5b8ffb218eacbccd06352a on the runnable worker entrypoint.

examples/checkpointed_batch.py:129 defines the registered activity as run.<locals>.execute, closing over store. The accepted runtime's cooperative registration preflight serializes (handler, interceptors) with ordinary pickle; CallbackInvocation.serialize requires the same spawn-compatible handler. The untyped activity decorator retains the original local function. Consequently, when the Server negotiates cooperative cancellation, worker.run_until() refuses this handler at registration instead of running the advertised example. Helper-only process_batch tests do not exercise that path.

Please move the registered handler to a module-level importable function or module-level bound-method/callable object with serializable ReceiptStore configuration. Add a source control that serializes the actual handler/configuration through the existing cooperative preflight or CallbackInvocation, without invoking the activity or a Server. Preserve the runtime supervision contract rather than bypassing it for the example.

I verified the complete four-file delta and its Git blobs, the full-input provider/checkpoint binding, bounded receipt reconciliation, fsync ordering, off-loop file work, fixed retry/deadline budgets and the uncertainty caveats. Those helper semantics are coherent. The source AST proves the registered function is local, and a pure standard-library control reproduces pickle's refusal of that function shape. I imported no SDK and ran no callback, example entrypoint, Worker, Server or package build. Security's helper-boundary review, current CI and actual published-package replacement evidence remain separately scoped; the source entrypoint finding must close before calling this example runnable.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security correction to my earlier source/example verdict at exact 65a5b28e8ef8ddf0dd5b8ffb218eacbccd06352a: HOLD on the runnable entrypoint.

run() defines the registered execute activity as a nested function and that function closes over ReceiptStore. When cooperative activity supervision is negotiated, Worker._validate_cooperative_activity_handlers() requires pickle.dumps((handler, interceptors), protocol=pickle.HIGHEST_PROTOCOL) to succeed before invocation; CallbackInvocation.serialize() enforces the same spawn-compatible/importable-handler boundary. A nested local function cannot satisfy that contract, so the documented runnable example would refuse before processing its first activity.

The previously reviewed provider helper, full-input fingerprint, receipt reconciliation, commit-before-ack order, and failure-boundary controls remain sound at their scoped helper boundary. Please move the registered handler to an importable module-level callable with explicitly serializable configuration (rather than a captured store), and add a control that runs the actual registered handler through the same serialization preflight. No runtime or package qualification is implied here.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Current successor replaces the nested callback with module-level BatchActivity.execute, a bound method carrying only the serializable ReceiptStore configuration. The helper/provider/fingerprint/order/budget implementation is unchanged. The demo leaves cooperative cancellation opt-in, but the old nested closure could not pass that mode's existing pickle boundary.

The new controls feed the actual registered handler and its real argument shape through Worker._validate_cooperative_activity_handlers() and CallbackInvocation.serialize(), then verify the restored bound configuration. A nested-handler negative is refused by both unchanged guards. All 22 example controls and 63 checkpoint controls pass together (85), with Ruff, mypy and diff checks clean. No example entrypoint, callback execution or Server was selected for these added preflight controls. Fresh exact-head CI and independent re-review follow; the original source HOLD remains recorded.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security rebind at exact 0a34792b198f72b4d596f89db67e9789c73fd388 / tree a3a629db: PASS for the corrected source/example boundary.

The registered activity is now the module-level BatchActivity.execute bound to a frozen BatchActivity containing only the picklable ReceiptStore(Path) configuration; no local function or captured closure remains. The new control exercises the actual registered bound method through both Worker._validate_cooperative_activity_handlers() and CallbackInvocation.serialize(), round-trips it through pickle, and separately proves that the predecessor-style nested handler is refused by both paths. Provider reconciliation, full-input fingerprinting, commit-before-ack ordering, failure budgets, and ordinary default execution are unchanged from the previously reviewed helper boundary.

This closes the exact runnable-entrypoint finding from my prior HOLD. It is a source/example verdict only: current-head CI remains separate, and no example/Server/package/publication execution is qualified here.

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ordinary source/API rebind: PASS for exact 0a34792b198f72b4d596f89db67e9789c73fd388. This closes my entrypoint serialization finding in review 5904451246.

I reviewed the complete three-file successor and verified its Git blobs. The registered callback is now the bound method of module-level BatchActivity, whose sole instance configuration is the frozen ReceiptStore containing its Path. It no longer captures a local function scope. The actual handler is passed through the existing cooperative Worker preflight and CallbackInvocation.serialize() with the real argument shape; the test checks the restored bound store and arguments without invoking the callback. The nested-handler negative must fail both unchanged guards.

The Batch, Progress, ReceiptStore, process_batch and BatchWorkflow ASTs are unchanged, preserving the previously reviewed full-input checkpoint/receipt binding, provider reconciliation and fsync, commit-before-ack order, uncertain-response handling and fixed retry/deadline budgets. No runtime supervision or policy is weakened. The demo's ordinary default is unchanged; this correction supplies compatibility with opted-in cooperative supervision as well.

No remaining blocking finding in this source delta. I ran no SDK test suite, example entrypoint, callback, Worker, Server or package build for this rebind. The owner's 85 source controls, fresh exact-head CI and required Security review remain independently scoped; publication and actual named published-package cold-replacement acceptance are still outstanding under shared #218.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 14, 2026 13:17
@rmcdaniel
rmcdaniel merged commit 437516a into main Nov 14, 2026
24 checks passed
@rmcdaniel
rmcdaniel deleted the sdk-python/218-checkpointed-batch branch November 14, 2026 13:18
@rmcdaniel

Copy link
Copy Markdown
Member Author

The bounded checkpointed-batch source example is merged normally at 437516ad76dda83371c17df7c418f398799ea6f0, exactly reviewed tree a3a629dba133156180f44cec42d1e95977def61e from head 0a34792b198f72b4d596f89db67e9789c73fd388. Both independent source rebinds pass, and all 24 current-head checks are terminal (20 success/four intended manual skips), including integration. The merged remote topic branch is independently absent. Actual-main checks are running separately.

All 22 example controls and 63 checkpoint controls pass together. This delivers the runnable source/example portion of shared218, with the corrected actual-handler preflight and serializer boundary. The API remains unreleased; published-package candidate/registry replacement qualification is still outstanding. No example entrypoint or Server observation was selected, and this source merge does not claim shared acceptance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant