Skip to content

Report open-run legacy version usage in native Server - #1221

Merged
rmcdaniel merged 17 commits into
mainfrom
feat/server-220-version-usage
Nov 14, 2026
Merged

rmcdaniel merged 17 commits into
mainfrom
feat/server-220-version-usage

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Open runs without a version marker currently have no fleet read that keeps them distinct from runs which recorded a current decision. This adds GET /api/workflow-version-usage and capabilities.workflow_version_usage, using Workflow #1146's v1 contract for namespace-scoped legacy/current/unknown counts and paged records.

The query reads existing run/history rows in one read-only repeatable snapshot. It preserves missing, pruned, incomplete, unsupported, malformed and conflicting evidence as unknown, validates signed 32-bit decisions and ranges, and uses exact identity comparisons on all three backends. Counts precede state filtering; opaque query-bound cursors traverse ascending run IDs under the first page's fixed upper bound. Each page remains a live observation. No workflow code, argument/result/memo or external payload fetch is involved.

Regression coverage includes recorded boundary decisions, missing-marker safety, invalid ranges, history gaps and marker budget, open/closed filtering, all-state counts, named namespaces, case-sensitive type identity, query/cursor validation and keyset admission bounds. OpenAPI and native usage documentation describe the capability, errors and live paging limits.

Validation at final e1729e222fd56e7d8b3181663969df76a49d38f9: all 63 current-head checks succeed, with one declared skip, including CodeQL, feature qualification and the complete native/backend/corpus matrix. Focused validation passes all 17 reader/cursor unit cases and 12 HTTP cases on SQLite, MariaDB and PostgreSQL 17. The endpoint cases include malformed reuse refusal, an actual native redrive writer/query positive, unknown ownership/history, and authenticated pagination. Cursor tests use fresh OS-random keys and retain wrong-key, unsigned and tampered-bound refusal. Independent exact-source Security review and Workflow shared-contract parity review pass. Published endpoint availability and CLI/Waterline acceptance remain separately attributable.

Implements the native source portion of #1220 and durable-workflow/.github#220. The shared contract is Workflow #1146 at 30341b139323c4bf3fa26a34c2c562f88c075bbe. No marker producer, storage schema, replay behavior, #980 qualification, #1204 capture, release alias or promotion changes are included. #1220 remains open through publication and consumer follow-through.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Concrete native compatibility finding: the proposed GET /api/workflows/version-usage occupies the existing GET /api/workflows/{workflow_id} URL for the valid workflow ID version-usage. It would turn that existing describe request into a 422 query-validation failure. Reserving an existing customer ID on an additive API change is not acceptable.

The native route is therefore GET /api/workflow-version-usage (relative route /workflow-version-usage). Query fields, v1 response shape, workflow_version_usage capability, error behavior and namespace semantics stay the same. No conflicting alias is added. I am adding a regression that creates version-usage, then verifies its original describe URL still works. Workflow and CLI have the same concrete correction for their contract/transport references before adoption. The draft is not published yet.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security source review: HOLD on exact 1a77d085d9c2fd0f3c9906d3175fbc5dd7263d1a.

The distinct GET /api/workflow-version-usage route correctly preserves the existing valid workflow ID version-usage. I also verified the revised repeated-decision rule: one version may retain distinct positive logical sequences while its individually valid range evolves; conflicting versions and reused logical sequences remain unknown.

Two concrete blockers remain:

  1. The supposedly opaque continuation has no integrity protection. Query::validate at rust/src/runtime/version_usage.rs:72-84 accepts a base64url JSON body directly, and lines 311-317 emit the same unsigned body. A caller can decode a returned token and change after to skip records or through to shrink/expand the bounded cohort; the modified token still passes every identity check. That violates the contract that invalid/mismatched cursors return 422 and that continuations retain the first page bound. It can produce incomplete counts during exactly the rollout/removal decision this endpoint is meant to inform. Bind the serialized body with a domain-separated HMAC using the runtime cursor authority, verify the tag before deserialization/field use, and add mutation controls for body/tag, after, and through. The existing observation and schedule cursors provide repository patterns.

  2. The Server docs still call Workflow 60c426... the normative shared v1 contract, but that exact contract still specifies relative /workflows/version-usage. The collision repair therefore leaves the cross-repository contract and future consumers inconsistent. Update Workflow Prepare Native Server 3.5.0 release notes and Compose default #1146 to the distinct route and rebind this PR to the corrected exact contract commit before adoption.

The namespace predicates are closed on both run and instance namespace; MySQL uses binary comparisons, PostgreSQL uses deterministic exact equality plus C ordering, and SQLite uses binary/default ordering. Missing/pruned/gapped/unsupported histories remain unknown, counts precede state filtering, and application payload fields are not selected. I found no separate tenant-leakage issue in the reviewed query.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded #220 source parity review at exact 1a77d085d9c2fd0f3c9906d3175fbc5dd7263d1a (no moving-head/whole-source verdict):

  1. Unknown history kinds can still report current. Evidence::event advances the physical sequence for an arbitrary future event kind, while record ignores it. A complete sequence of WorkflowStarted, valid current VersionMarkerRecorded, then FutureHistoryEvent therefore reaches Current. The shared draft requires unsupported retained formats to remain history_unavailable; please add this causal control and a closed supported-kind check.
  2. The marker reader is looser than the embedded reader. After a valid requested current marker, an unsupported field/version on a marker or malformed range/logical sequence on a different change ID is ignored. Embedded VersionUsageDecision conservatively marks these histories unknown. Please align the strict format/decision preconditions or obtain an explicit shared contract decision before adoption; do not silently claim parity from valid-marker-only cases. Valid repeated versions with independently valid differing bounds are allowed, as already clarified.
  3. Missing parent provenance can disappear from counts. Both upper and row queries inner-join workflow_instances. A matching open workflow_runs row whose instance is unavailable is silently absent rather than represented as unknown. The query's safety requirement is to retain unavailable provenance as unknown, not report zero usage. Please retain the namespace-scoped run cohort and classify unavailable ownership/provenance conservatively without looking up foreign instance detail. I am adding the corresponding explicit embedded ownership/source-provenance controls on Workflow Prepare Native Server 3.5.0 release notes and Compose default #1146.

The revised top-level /workflow-version-usage route avoids the existing workflow-ID collision; namespace echo/capability/422 shape is compatible. These observations concern source classification/counting only. Current native tests, backend/cursor/authority checks and independent Security still gate #1221; no producer/runtime/publication acceptance is inferred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact source 32928fa07f303bf09683114cc7a72f1631319481 / tree ae56703479991597a41c6d454cd51a8d9cfb9f51 passed the captured focused Rust check: 11 classifier/cursor controls and all 8 SQLite HTTP cases, exit 0/no OOM. The captured source file hashes still equal the branch. No new-code compiler warning was introduced; the pre-existing dependency/runtime warnings remain visible. The pinned image lacks rustfmt, so no component installation or formatting claim is made.

The native prefix correction recognizes StartAccepted before WorkflowStarted, as shown by the retained failing predecessor; children/continued runs can begin with WorkflowStarted. Gaps, missing starts, unsupported start data and missing tails remain unavailable. Repeated valid versions may expand ranges. HMAC-authenticated cursors reject modified bounds and unsigned input; legal escaped query values round-trip. The distinct route preserves describing workflow ID version-usage. The normative reference now pins Workflow 4ae0b68f with the corrected route.

The source check ran November 14, 11:19:45–11:20:38 UTC, using the cached pinned Rust 1.99 image, 2 CPUs / 8 GiB / 512 PIDs, offline/read-only captured source and public registry, private target/temp/logs, with two finite 900-second commands. Captured full container d34957cb4ec73f3ba7e85c0376f84031465dbe5cf0d0b8ebd0288e7f2e2554b0 and its exact task-label inventory are independently absent. Source logs remain temporary diagnostics, retained seven days. All predecessor nonpasses are distinct from this pass.

Current-head full backend/corpus/required CI and renewed exact Security/Workflow source review still gate merge. Security's earlier HOLD is not cleared by this local result. This establishes neither published endpoint availability nor downstream CLI/Waterline acceptance and changes no #980/#1204 or promotion gate.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 14, 2026 11:27
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security whole-head HOLD for exact 32928fa07f303bf09683114cc7a72f1631319481 / tree ae56703479991597a41c6d454cd51a8d9cfb9f51.

The prior cursor and route findings are fixed: the cursor now has a domain-separated HMAC verified before JSON decoding, and the distinct /api/workflow-version-usage route preserves workflow ID version-usage. Two current-source findings remain:

  1. PostgreSQL/SQLite scope equality is not actually collation-independent. POSTGRES_ROWS and POSTGRES_UPPER apply COLLATE "C" only to run-ID ordering/bounds. The joins and predicates for instance ID, instance namespace, run namespace, workflow type, history run ID, and status still use the column/default collation; the SQLite templates likewise omit explicit COLLATE BINARY. A nondeterministic PostgreSQL collation can therefore make a case/accent-equivalent foreign namespace or workflow type satisfy the authenticated predicate, contrary to the source comment and tenant boundary. Apply exact C/BINARY comparison to every identity/scope equality in both rows and upper-bound queries, and add backend controls using collation-equivalent but byte-distinct namespace/type/ID values.

  2. Malformed start prefixes can still produce a positive decision. Evidence::event constrains only the first event and duplicate WorkflowStarted. It accepts, for example, WorkflowStarted, StartAccepted, VersionMarkerRecorded, StartAccepted, StartAccepted, WorkflowStarted, VersionMarkerRecorded, and an arbitrary event between StartAccepted and WorkflowStarted; all can classify current. The accepted shapes are exactly WorkflowStarted or StartAccepted, WorkflowStarted before subsequent evidence. Track the prefix state explicitly and make any other placement/repetition return history-unavailable, with negative controls for each malformed shape.

The HMAC/route/native ordinary-prefix corrections remain accepted. Backend/corpus/CI gates remain separate; no runtime or release action is requested.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security whole-source PASS for exact b8ccffb16513ca2ccd03f29c8130aa2987167376 / tree c91397c8488692b14be68a7fc395aec6803b816e.

Both findings from 6876578856 are closed:

  • SQLite and PostgreSQL now apply explicit BINARY / "C" collation to every namespace, workflow type, instance/run/history identity, status, keyset bound, and ordering comparison in both upper-bound and row queries. MySQL remains explicitly binary.
  • The history classifier now accepts only WorkflowStarted or the immediate StartAccepted, WorkflowStarted prefix. Late/repeated StartAccepted, duplicate starts, intervening events, and any other first event permanently make the history unavailable; the five synthetic controls plus the contiguous late-start stored-history regression cover the prior positive-decision gap.

The previously accepted domain-separated cursor HMAC, safe route, exact response/count bounds, and Workflow 4ae0b68f normative contract remain unchanged. Current-head focused/backend/corpus/CI results are still required before integration. This verdict is source-only and grants no runtime, #980, #1204, release, or promotion action.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The three bounded parity findings are corrected at 6e99c522: a closed retained-v2 event-kind registry rejects future history formats; every marker is validated before filtering change ID, with native's known task snapshot field allowed; and namespace-qualified LEFT JOINs retain runs with unavailable/foreign owner metadata as unknown instead of dropping them from counts. The normative contract now pins Workflow 0f8f30f2.

All 15 unit controls and 10 SQLite HTTP cases pass on that captured source. A private counterfactual using the earlier b8ccffb1 reader with the new controls fails exactly the three new marker/event tests and the missing/foreign-owner HTTP test, with no compilation or fixture errors. This discriminates the source correction from a changed expectation. The earlier strict-prefix/collation corrections are retained.

Current-head backend/corpus/required CI and renewed exact Security/Workflow source review remain the merge gates. No published availability, installed consumer result or promotion is inferred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded producer parity review PASS at exact 6e99c5221703a463bcf4741849936549051c8e0e. I re-read the classifier and all three SQL templates against Workflow’s version-usage contract. The three source findings in the earlier review are resolved:

  • Unknown event kinds are checked against the retained-v2 registry before classification. A valid current marker followed by an unsupported event remains history_unavailable.
  • Every marker is checked for signed-32-bit bounds, positive unique logical position and closed supported fields before its change ID can be ignored. The native task object is a known runtime-specific field; unknown extensions remain unavailable. Evolving individually valid ranges still work.
  • Namespace-matching owners are left-joined. Missing or foreign owners retain the open run in counts/pages as unknown, with exact binary/C identity comparisons and keyset order.

Native StartAccepted prefix handling remains deliberately native. I also read the new individual negative controls and missing/foreign owner HTTP accounting controls; their execution and the predecessor counterfactual are attributed to the Server owner’s recorded results. This is a source parity disposition for these seams, not full native/backend/published qualification. Security, current complete CI/backend/corpus and normal integration/readback remain required. No producer, replay, persistence schema or release authority is inferred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

At exact 33c4a8df5efffca474dc83331d87a82a7f417f61 / tree 1aee7302b83bcd192379d336288b77a535311102, the complete library suite passes 530 tests / 96 declared backend/helper ignores, followed by all 10 focused HTTP cases. This includes the newly advertised capability assertion; the prior CI inventory-count failure is corrected. The runtime implementation is unchanged from the parity-reviewed 6e99c522; remaining changes are that test correction and normative references to Workflow 9af1eb53 (same route/request/v1 response).

The earlier reader's four causal failures and the bounded Workflow parity PASS remain attributable to their exact source. Full current-head native backend/corpus/required CI and renewed Security review still gate merge. No skipped backend-only library case is counted as a pass, and no publication/consumer acceptance is claimed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review of current exact c37dc693975a6cb8067d4e2d526f3d321c13d582 is HOLD on one remaining shared marker-contract defect.

Evidence::record() allowlists reused_from_run_id and reused_recorded_at (lines 185–188) but validates neither pairing nor values. An otherwise-valid target marker containing only "reused_from_run_id": [], or a blank/malformed reused_recorded_at, proceeds through lines 193–200 and is classified current. The pinned shared v1 contract says malformed decision data remains unknown and presently names only the producer task snapshot as a known extension; Workflow #1146 has the same concrete defect under review.

Either reject replay-provenance fields as unsupported under v1, or add them to the normative contract and require both-or-neither with a nonempty exact run ID and the producer's canonical valid recorded clock. Add direct unit and HTTP/storage controls for lone, wrong-type, blank, and malformed provenance, including a valid redriven-marker positive if the pair is admitted.

The safe distinct route, operator/admin namespace admission, HMAC-before-parse cursor, exact binary/C joins and keyset comparisons, strict prefix/event-kind/history-contiguity handling, marker budget, ownership/import fail-closed behavior, and the capability-corpus successor otherwise match the reviewed boundary. Current backend/corpus/CI remain separate gates and do not clear this source finding.

Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
Comment thread rust/src/runtime/version_usage.rs Fixed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security HOLD on exact cb4e29f, narrowly scoped to the normative contract pin.

The implementation-side findings are closed: the route is distinct; operator/admin and namespace middleware applies; cursor HMAC is domain-separated and verified before body parsing; namespace/type/run joins and keyset ordering are exact-collation bound; unsupported owner/history/event/marker formats remain unknown; and the complete reused-marker pair is checked on every marker before classification.

However, both the Server guide and OpenAPI still call Workflow 9af1eb53f698f8dd764dfd7841c1851d79555b9d the shared normative v1 contract. That exact document does not define reused_from_run_id or reused_recorded_at. Its known marker-field list ends at task, and it says unknown marker extensions require a reader upgrade and remain unavailable. cb4 instead treats those two fields as known and can classify the actual redriven marker current. Thus the exact source currently cites a normative schema that says its positive is unsupported.

Rebind the links to the final reviewed Workflow successor that defines the complete pair, exact source-ID constraints and explicit-offset 1..9-digit RFC3339 clock, and add the corresponding fields to the shared schema if it owns the closed response/marker shape. Preserve a causal assertion that the predecessor pinned document lacks the fields and the successor contains the exact admitted pair. No backend/CI/runtime PASS is inferred; current full gates remain separate.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security source PASS for exact c1ebf5a. The only cb4 successor delta is the two immutable normative links, both now pinned to Workflow 30341b139323c4bf3fa26a34c2c562f88c075bbe. I independently read that exact blob: it defines both-or-neither reused_from_run_id/reused_recorded_at, a nonempty bounded source run identity, an explicit-offset RFC3339 instant, runtime-specific precision, history_invalid for malformed/incomplete pairs, and no retained-string rewrite. That aligns with native c1 validation and its 1..9 fractional-digit producer format. The shared response schema blob remains bf87b18853363219c325cbae11f4fa4583ab68df and does not own marker payload internals.

This closes the prior normative-pin HOLD and the earlier implementation findings. Full current-head library/backend/corpus/CI completion remains an independent integration gate; no runtime or promotion verdict is included.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security source rebind PASS for exact 1e927eedc5c36dc837adbecd3b17b5bbe6133767 / tree 54f24229cc009501a854bc768982a693132e919e.

The version-usage implementation, tests, HMAC/cursor behavior, exact-collation queries and Workflow 30341b13 normative pins are byte-identical to reviewed c1ebf5a1. The only composed delta is docs/storage-admission.md and rust/src/runtime/storage_admission.rs; I compared both complete blobs and they are byte-identical to separately reviewed Server 1223 exact 63cea4854ec6ebe56ab1603b014e246b907aedc0 (SHA-256 b0bf0ed4... / 06447e41...). There is no manual conflict resolution or combined semantic delta.

Fresh current-head backend/corpus/CI remains the integration gate. No runtime, release or promotion scope is inferred.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security test-only rebind at exact f944e74d131075d8847de5be196b1947706c13df / tree aa2b44870418e7ca4629cf253b5ed4806a275656: PASS.

The production prefix is byte-identical to reviewed 1e927eed. The sole delta replaces seven fixed cursor-test key strings with fresh 32-byte OS-random keys encoded through the same URL-safe representation accepted by the existing cursor code. The wrong-key check now uses a separately generated key and explicitly proves the two keys differ; unsigned body, mutated after/through, maximum-value round trip, and all production HMAC/domain-separation assertions remain unchanged. No suppression, scanner exemption, deterministic production secret, or weakened authentication path is introduced.

Focused/current-head test and complete CodeQL/backend integration remain separate merge gates; no runtime or promotion scope is added.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded Workflow v1 source-parity rebind PASS at f944e74d131075d8847de5be196b1947706c13df.

I read the reader successor to the previously reviewed 6e99c522 seams. Reuse provenance now requires both fields or neither and validates them before filtering the change ID. Native bounded UTF-8 run identities and explicit-offset RFC 3339 clocks retain their supported nanosecond precision without rewriting strings. The malformed-pair cases include other-change markers, and the HTTP control exercises an actual native redrive writer followed by a read-only current classification. This agrees with Workflow's shared contract at 30341b13 (unchanged in current 04d86304).

The closed retained-kind registry, exact owner/namespace/history joins, unknown accounting, signed query-bound cursors and collision-free route retain the earlier parity fixes. I also read the fresh-random-key cursor controls. No new shared-interface finding. This is source review; I did not execute the native tests, and current whole-head CI/backend/Security, publication and consumer acceptance remain the Server owner's gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Current-head integration HOLD on exact f944e74 is a concrete test-fixture defect, not a new production-reader finding.

The new pruned_missing_prefix_and_conflicting_history_remain_unknown control inserts history ID version-usage-late-admission, which is 28 bytes, into workflow_history_events.id VARCHAR(26). The current MariaDB job fails at tests/execution/version_usage.rs:22 with SQLSTATE 22001 / error 1406 “Data too long for column id”; both PostgreSQL profiles fail at line 17 with SQLSTATE 22001 “value too long for type character varying(26)”. The dedicated SQLite/focused result did not exercise this schema constraint.

Use a valid <=26-byte unique fixture ID, preserve the late StartAccepted placement and last_history_sequence increment, then rerun the exact current whole-head backend matrix. The current artifacts do not execute the intended classifier assertion on those backends, so they remain NONPASS. The production source/test-key Security PASS at 6877840351 is unchanged; this finding is scoped to complete integration evidence.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded fixture finding CLOSED / source rebind PASS for exact e1729e222fd56e7d8b3181663969df76a49d38f9.

The complete delta from reviewed f944e74d is one literal in rust/tests/execution/version_usage.rs: the malformed late-StartAccepted fixture ID is shortened from 28 bytes to version-late-admission (22 bytes), within the real VARCHAR(26) schema. Production code, the corruption shape, query assertions, HMAC/collation/prefix logic, and test-key correction are byte-identical. This directly closes the MariaDB/MySQL/PostgreSQL 22001 seam reported in the backend finding.

I did not rerun the backend matrix locally; the reported cached MariaDB and PostgreSQL 17 endpoint passes and current exact-head full CI/CodeQL remain separate integration evidence. No runtime, promotion, or release scope is added.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit c407077 Nov 14, 2026
64 checks passed
@rmcdaniel
rmcdaniel deleted the feat/server-220-version-usage branch November 14, 2026 14:18
@rmcdaniel

Copy link
Copy Markdown
Member Author

Final exact-head Security/integration disposition: PASS for e1729e222fd56e7d8b3181663969df76a49d38f9 / tree cb7a571b399a141d97a0893cacdfa3a6b24dd14b.

Independent current readback confirms 64 terminal contexts: 63 SUCCESS, one declared SKIPPED, no failed or active context. This includes CodeQL, native build, repository/policy contracts, the complete SQLite/MariaDB/MySQL/PostgreSQL endpoint and corpus coverage, corrected-marker cases, and feature source qualification.

The final successor remains exactly the reviewed source: the only f944e74d… → e1729e22… change is the malformed late-StartAccepted fixture ID shortening from 28 bytes to 22 bytes, which preserves the intended corruption and now fits the real VARCHAR(26) constraint. Production tenant/namespace joins, exact collations, unavailable-history accounting, closed marker grammar, domain-separated HMAC cursor verification, route/capability contract, and Workflow 30341b13… normative pin are unchanged from the prior Security/parity PASS.

The earlier backend failures remain correctly NONPASS and are superseded only by these fresh exact-head backend results. This closes the source and current integration review for PR #1221. Published endpoint availability, CLI/Waterline consumer acceptance, release/promotion, #980 and #1204 remain separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants