Rust Server train: operator metrics redispatch (#708), query acknowledgement wait (#720), route coverage

docs/rust-server-port-checkpoints/2026-10-23-operator-metrics-redispatch-states.md

@@ -0,0 +1,81 @@
+# October 23, operator metrics after PHP redispatches an overdue task
+
+- **Found by.** On PR #673's parity run 39995620421, the Shared Server
+  fixtures failed on PostgreSQL 17 and MySQL 8.0. The diff came before
+  any PHP pass line, so published PHP 2.11.1 failed the
+  `operator-metrics` contract, not native. The failing stage was
+  `waiting_after_redispatch`.
+- **The race in published PHP.** PHP's repair pass redispatches the
+  overdue ready task in two steps:
+  - `TaskWatchdog::recoverExistingTask` raises the task's
+    `repair_count` and projects the run summary, in one transaction. The
+    run is marked `repair_needed`.
+  - `TaskDispatcher::markDispatched` then stamps the dispatch, and
+    `refreshRunSummary` clears `repair_needed` in a separate step.
+
+  `OperatorMetrics::snapshot` reads runs, then tasks, then activities,
+  then the backlog, with no shared snapshot. `HealthCheck` takes its own
+  snapshot after the metrics request. So one response can mix states.
+- **The four PHP states.** In order, the overdue task is:
+  0. overdue and unrepaired;
+  1. overdue, repaired, run marked `repair_needed`;
+  2. dispatched, run still marked `repair_needed`;
+  3. dispatched, run summary refreshed.
+
+  The old contract accepted states 0, 1 and 3 read consistently. It
+  refused state 2 (the MySQL 8.0 shape) and a backlog read after the
+  redispatch while the tasks were read before it (the PostgreSQL 17
+  shape).
+- **First model, refused in review.** The first correction (head
+  `3fa5cd5288`) let each section of the response show one state, with
+  sections never going backwards. Security's review (#708 comment
+  6373392835) showed this was not enough. Published Workflow `aca7928484`
+  reads each value with its own query, even inside one section. So a
+  forward-only change can split the `tasks` section: `dispatch_overdue`
+  at state 0, `max_repair_count` at state 1, and `unhealthy` at state 2.
+  The timestamps are read before the counts, so a timestamp can also
+  outlive its count.
+- **Contract decision.** For published PHP only, the
+  `waiting_after_redispatch` stage follows PHP's actual query order. The
+  reads that depend on the repair state are, in order:
+  - runs: `oldest_repair_needed_at`, then `repair_needed`;
+  - tasks: `oldest_dispatch_overdue_since` (with `oldest_unhealthy_at`,
+    which PHP builds from the same timestamps), then `dispatch_overdue`,
+    then `max_repair_count`, then `unhealthy`, counted again;
+  - backlog: `unhealthy_tasks`, then `repair_needed_runs`.
+
+  The response passes when some sequence of states that never goes
+  backwards in that order produces exactly its counts and timestamps. The
+  health request reads its own snapshot afterwards, starting at or after
+  the response's last read. Each health warning follows the field it
+  reads:
+  - `task_transport` follows `tasks.unhealthy`;
+  - `routing_health` follows `tasks.dispatch_overdue`;
+  - `durable_resume_paths` follows `backlog.repair_needed_runs`.
+
+  Native runs no repair loop, so native must still report the overdue
+  task consistently, with each timestamp present exactly when its count
+  is. A native response that shows a redispatch must show it as one
+  settled state.
+- **How it was checked.**
+  - Security's counterexample passes for PHP and fails for native. So
+    does the timestamp kept from before a redispatch.
+  - New tests refuse each backward step: a recount that is overdue after
+    the dispatch count was not; a backlog that is overdue after the
+    recount was not; an overdue count after its timestamp showed the
+    redispatch; a run timestamp from a repair the later reads have not
+    reached; and health whose recount is overdue after its dispatch count
+    was not.
+  - An exhaustive test covers all 256 combinations of the eight values
+    that depend on the state. The 37 combinations that some forward-only
+    order produces pass; the other 219 fail.
+  - Two earlier tests assumed one read per section. They now accept
+    health whose backlog was read after the run was projected again, and
+    whose dispatch count was read before the redispatch. A native
+    response still fails both.
+  - The original CI shapes (MySQL 8.0 state 2, PostgreSQL 17 backlog
+    after the redispatch) still pass. The server-parity unit tests pass:
+    5,502 of 5,504, with one intended skip and one test that needs Docker
+    inside its container.
+- **Not changed.** The fixture, its stages and the other stages' expected
+  values. Native's behaviour is unchanged.

docs/rust-server-port.md

@@ -14,87 +14,83 @@ qualify a later tree.
 
 ## Current state
 
-Updated October 22, 2026.
-
-- **Merged since October 20.**
-  - **Backup-first takeover** on SQLite, PostgreSQL and MySQL/MariaDB:
-    [#523](https://github.com/durable-workflow/server/pull/523),
-    [#529](https://github.com/durable-workflow/server/pull/529),
-    [#532](https://github.com/durable-workflow/server/pull/532),
-    [#533](https://github.com/durable-workflow/server/pull/533),
-    [#536](https://github.com/durable-workflow/server/pull/536),
-    [#538](https://github.com/durable-workflow/server/pull/538) and
-    [#539](https://github.com/durable-workflow/server/pull/539). This covers the
-    fingerprint, the layouts and WAL preparation. It also covers a PHP-accepted
-    signal and an expired PHP lease resumed under native, and the refusal of
-    unsupported features.
-  - **Multi-node and API parity.**
-    - [#549](https://github.com/durable-workflow/server/pull/549): the
-      multi-node kill and restart qualification.
-    - [#548](https://github.com/durable-workflow/server/pull/548): ownership
-      guard refusals.
-    - [#550](https://github.com/durable-workflow/server/pull/550) and
-      [#551](https://github.com/durable-workflow/server/pull/551): the search
-      attribute registry and namespace update.
-    - [#594](https://github.com/durable-workflow/server/pull/594): history
-      export.
-    - [#595](https://github.com/durable-workflow/server/pull/595) and
-      [#596](https://github.com/durable-workflow/server/pull/596): retention and
-      activity timeouts.
-  - **The October 21-22 trains.**
-    [#629](https://github.com/durable-workflow/server/pull/629),
-    [#638](https://github.com/durable-workflow/server/pull/638),
-    [#641](https://github.com/durable-workflow/server/pull/641),
-    [#649](https://github.com/durable-workflow/server/pull/649),
-    [#659](https://github.com/durable-workflow/server/pull/659) and
-    [#665](https://github.com/durable-workflow/server/pull/665) covered:
-    - input envelopes, run repair, legacy settings, activity deadlines,
-      runtime credentials and principals;
-    - query and update admission, signals rejected after a run closes, start
-      fingerprints and all-of barriers;
-    - cooperative waits, child cancellation and parent-close policies,
-      continue-as-new messages, history pages, declared signals and build IDs.
-  - **GHSA-6q48-6c7w-9gw2 trust remediation.**
-    [#671](https://github.com/durable-workflow/server/pull/671) moves the Rust
-    locks to the fixed webpki roots.
-    [#672](https://github.com/durable-workflow/server/pull/672) pins the image's
-    TLS toolchains and checks every trust store.
-- **Open.**
-  - [#675](https://github.com/durable-workflow/server/pull/675): train of
-    [#667](https://github.com/durable-workflow/server/pull/667) (definition
-    routing), [#669](https://github.com/durable-workflow/server/pull/669) (poll
-    waits), [#670](https://github.com/durable-workflow/server/pull/670) (worker
-    admission) and [#674](https://github.com/durable-workflow/server/pull/674)
-    (schedule fixture probe).
-  - [#676](https://github.com/durable-workflow/server/pull/676): accept
-    published PHP's health read during or after its own task repair.
-  - [#673](https://github.com/durable-workflow/server/pull/673): pip, Composer
-    and Cargo trust in CI and conformance, in Security review.
-    [#677](https://github.com/durable-workflow/server/issues/677) owns the
-    remaining clients, with an explicit non-completion boundary.
-- **Shared Server fixtures on main.** Embedded passes. Native differs on two
-  fixtures:
-  - **`signals-before-worker`:** native records `SignalRejected` after a
-    terminal event.
-  - **`worker-listing`:** native reports `_worker_protocol_version` in
-    `process_metrics`.
-
-  In both, native follows published Server 2.10.0 while the fixtures still
-  expect the frozen 2.5.14 reference. Separately, the published PHP reference
-  sometimes differs from itself on `operator-metrics` (postgres-17) because of
-  its own repair window, which #676 handles. Moving the reference needs a
-  recorded contract decision, not an edited expectation.
-- **Parity surface.** Native serves 97 of published PHP's 175 API routes. That
-  count compares `php artisan route:list --path=api` on this tree with the
-  native router. The missing routes are listed below.
-- **Next action.**
-  1. Verify the Rust suites on #675 and #676, then merge them.
-  2. Record the reference decision for Server 2.10.0 and re-record the affected
-     fixtures from that published release, so main's Shared Server fixtures run
-     green.
-  3. Port the missing families below.
-
-  Packaging the native binary into the server image remains an open gate.
+Updated October 24, 2026.
+
+- **Merged October 24.** Train
+  [#729](https://github.com/durable-workflow/server/pull/729) (merge
+  `61219609b`):
+  - MySQL statement closes are sent with the next command
+    ([#717](https://github.com/durable-workflow/server/pull/717)), ending the
+    MySQL-family stalls of [#714](https://github.com/durable-workflow/server/issues/714);
+  - policy maps that are not JSON objects are refused, like PHP #705, and a
+    namespace quota map that is not UTF-8 is refused under either name
+    ([#719](https://github.com/durable-workflow/server/pull/719));
+  - worker history shows consumed signal sequences, hydrating only each page's
+    own signals, plus full task snapshots
+    ([#710](https://github.com/durable-workflow/server/pull/710));
+  - a newer push cancels a superseded Shared Server fixtures run
+    ([#724](https://github.com/durable-workflow/server/pull/724)).
+- **Merged October 20-23.** These are summarized in the checkpoint history:
+  - backup-first takeover on every database;
+  - multi-node kill and restart;
+  - ownership guards, search attributes, namespace update, history export,
+    retention and activity timeouts;
+  - the October 21-23 trains;
+  - the move to the published 2.11.1 reference;
+  - the GHSA-6q48-6c7w-9gw2 fixes.
+
+  Each train's PR holds its evidence.
+- **In review or qualifying, in merge order.**
+  1. CI speed and the merge queue:
+     - [#741](https://github.com/durable-workflow/server/pull/741) runs the
+       SDK direction profiles in parallel lanes, about 14 minutes off every
+       database job, and fixes a timestamp-comparison test flake;
+     - [#742](https://github.com/durable-workflow/server/pull/742) adds the
+       merge-queue prerequisites (`merge_group` triggers, one fixtures
+       result, and fixture cells skipped when a change cannot affect them).
+  2. This train: operator metrics read across PHP's redispatch steps
+     ([#708](https://github.com/durable-workflow/server/pull/708)),
+     query-authority clients that wait for the acknowledgement
+     ([#720](https://github.com/durable-workflow/server/pull/720)), and the
+     machine-checked route list below.
+  3. Condition waits without a key, task snapshots, the SQLite patch cohort and
+     message streams (#713, #718, #722, #709).
+  4. Update validation with PHP's poll request ID contract
+     ([#712](https://github.com/durable-workflow/server/pull/712)), then the
+     held-poll registration fence
+     ([#739](https://github.com/durable-workflow/server/pull/739), #734) and
+     worker sessions
+     ([#740](https://github.com/durable-workflow/server/pull/740), #736),
+     stacked in that order.
+  5. Activity cancellation acknowledgement, and pip, Composer and Cargo trust
+     in CI ([#673](https://github.com/durable-workflow/server/pull/673)).
+- **Shared Server fixtures.** Native, embedded and published PHP 2.11.1 are
+  compared on SQLite, MySQL 8.0, MariaDB 10.11 and PostgreSQL 17. Where the
+  published reference itself misbehaves, an explicit boundary separates it
+  from native, and the PHP owner has an issue for each case:
+  - `operator-metrics` across PHP's redispatch window (#708);
+  - the published-SDK patch cohort's SQLite poll refusal
+    ([#721](https://github.com/durable-workflow/server/issues/721), #722);
+  - update validation under SQLite write locks
+    ([#726](https://github.com/durable-workflow/server/issues/726), fixed in
+    PHP by #730);
+  - `worker-sdk-persistent-pressure`'s shutdown window
+    ([#728](https://github.com/durable-workflow/server/issues/728));
+  - a retried activity's cooperative cancellation
+    ([Workflow #923](https://github.com/durable-workflow/workflow/issues/923));
+  - a reacquired worker session keeping its old failure reason
+    ([#737](https://github.com/durable-workflow/server/issues/737)).
+
+  No expectation is edited to make native pass.
+- **Parity surface.** Native serves 126 of the 175 API route methods in
+  published 2.11.1's route table. An execution test sends each one through
+  the native router, and the ones still answering
+  `501 rust_capability_not_implemented` must match
+  [`unserved-php-routes.v2.11.1.txt`](../rust/tests/fixtures/unserved-php-routes.v2.11.1.txt).
+  The work in review serves 12 more: message streams (3), update validation
+  (3), worker sessions (5) and activity cancellation acknowledgement (1).
+- **Next action.** Land the items above in order, keep porting the unserved
+  routes fixture first, and package the native binary into the server image.
 
 ## API surface still missing
 
@@ -601,7 +597,9 @@ October 21, operator metrics and health:
   - PHP's maintenance loop redispatches an overdue ready task, after which
     PHP reports it repaired, not overdue. Native runs no repair loop, so the
     task stays overdue until a worker claims it. The fixture accepts either
-    PHP state.
+    PHP state. Since October 23 it also accepts PHP's reads that straddle
+    the redispatch, provided they never go backwards; see
+    [the redispatch-states checkpoint](rust-server-port-checkpoints/2026-10-23-operator-metrics-redispatch-states.md).
   - The operator dashboards, capacity evidence, and the repair, retention,
     activity-timeout and payload-cleanup pass endpoints remain.
 

rust/tests/execution.rs

@@ -287,6 +287,9 @@ mod external_payloads;
 #[path = "execution/trace_carriers.rs"]
 mod trace_carriers;
 
+#[path = "execution/route_coverage.rs"]
+mod route_coverage;
+
 enum TestDatabase {
     Sqlite(tempfile::TempDir),
     Postgres {

rust/tests/execution/activity_deadlines.rs

@@ -483,9 +483,14 @@ async fn heartbeat_renews_only_progress_deadline_and_cannot_resurrect_elapsed_au
     assert_eq!(after.0, initial.0, "Start-to-close deadline remains fixed");
     assert!(after.1 > initial.1, "Heartbeat deadline advances");
     assert_eq!(after.2, initial.2, "Lifecycle deadline remains fixed");
+    // Compare instants, not text: a lease on a whole second is written
+    // without a fraction ("...:26Z"), which sorts after "...:26.045Z".
+    let instant = |value: &Value| chrono::DateTime::parse_from_rfc3339(value.as_str().unwrap()).unwrap();
     assert!(
-        renewed.1["lease_expires_at"].as_str().unwrap()
-            > activity["lease_expires_at"].as_str().unwrap()
+        instant(&renewed.1["lease_expires_at"]) > instant(&activity["lease_expires_at"]),
+        "{} renews {}",
+        renewed.1["lease_expires_at"],
+        activity["lease_expires_at"]
     );
     let prefix = run_history(&app, &started["workflow_id"], &started["run_id"]).await;
     database

rust/tests/execution/route_coverage.rs

@@ -0,0 +1,76 @@
+use super::*;
+
+// The published Server 2.11.1 API routes native does not serve yet. A port
+// that serves a route removes it from this list in the same change, so the
+// list in docs/rust-server-port.md cannot drift from the code.
+const UNSERVED: &str = include_str!("../fixtures/unserved-php-routes.v2.11.1.txt");
+
+/// Sends every method of every published PHP API route through the native
+/// router, with each path parameter filled in, and lists those that answer
+/// 501 rust_capability_not_implemented.
+#[tokio::test]
+async fn unserved_published_routes_match_the_recorded_list() {
+    let database = TestDatabase::new().await;
+    let runtime = database.open().await.unwrap();
+    let app = router(runtime.clone());
+    let table: Value = serde_json::from_str(include_str!("../../src/runtime/php-routes.v2.11.1.json")).unwrap();
+    let mut unserved = Vec::new();
+    let mut checked = 0;
+    for route in table["routes"].as_array().unwrap() {
+        let (methods, uri) = (route[0].as_str().unwrap(), route[1].as_str().unwrap());
+        if !uri.starts_with("api/") {
+            continue;
+        }
+        for method in methods.split('|').filter(|method| *method != "HEAD") {
+            checked += 1;
+            let path = format!("/{}", fill_parameters(uri));
+            let worker = path.starts_with("/api/worker/");
+            let body = if matches!(method, "GET" | "DELETE") { Body::empty() } else { Body::from("{}") };
+            let request = Request::builder()
+                .method(method)
+                .uri(&path)
+                .header("authorization", "Bearer test-token")
+                .header("accept", "application/json")
+                .header("content-type", "application/json")
+                .header(
+                    if worker { "x-durable-workflow-protocol-version" } else { "x-durable-workflow-control-plane-version" },
+                    if worker { "1.20" } else { "2" },
+                )
+                .body(body)
+                .unwrap();
+            let response = app.clone().oneshot(request).await.unwrap();
+            let status = response.status();
+            let bytes = to_bytes(response.into_body(), 8 * 1024 * 1024).await.unwrap();
+            if status == StatusCode::NOT_IMPLEMENTED
+                && String::from_utf8_lossy(&bytes).contains("rust_capability_not_implemented")
+            {
+                unserved.push(format!("{method} /{uri}"));
+            }
+        }
+    }
+    assert!(checked > 170, "the published route table lists {checked} API route methods");
+    let recorded: Vec<&str> = UNSERVED.lines().filter(|line| !line.is_empty() && !line.starts_with('#')).collect();
+    let now_served: Vec<&&str> = recorded.iter().filter(|route| !unserved.iter().any(|found| found == **route)).collect();
+    let now_unserved: Vec<&String> = unserved.iter().filter(|found| !recorded.contains(&found.as_str())).collect();
+    assert!(
+        now_served.is_empty() && now_unserved.is_empty(),
+        "update rust/tests/fixtures/unserved-php-routes.v2.11.1.txt and the port log. Now served: {now_served:?}. Now unserved: {now_unserved:?}"
+    );
+    assert_eq!(unserved, recorded, "keep the recorded list in route-table order");
+    runtime.close().await;
+    database.remove().await;
+}
+
+/// Replaces each `{parameter}` in a route with a placeholder value.
+fn fill_parameters(uri: &str) -> String {
+    let mut path = String::new();
+    let mut rest = uri;
+    while let Some(open) = rest.find('{') {
+        let close = open + rest[open..].find('}').expect("closed route parameter");
+        path.push_str(&rest[..open]);
+        path.push_str("probe-x");
+        rest = &rest[close + 1..];
+    }
+    path.push_str(rest);
+    path
+}