Skip to content

Rust #325: verify small-host history order on the server clock alone - #933

Merged
rmcdaniel merged 4 commits into
mainfrom
rust-325-probe-clock-offset
Nov 2, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
rust-325-probe-clock-offset

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes a small-host benchmark probe defect that Cloud found in the #922 batch. Native is not at fault.

What happened

On a Vultr vc2-1c-1gb target (Intel Broadwell, kvm-clock), every rate, down to 0.5/s, failed probe verification with Standard history timestamps are missing or out of order: WorkflowStarted. All work completed quickly (30/30 closed, p50 0.47 s, no poll errors). The same image, profile and probe passed on vhp-1c-1gb-amd.

timedatectl timesync-status (Cloud, 00:29 UTC; same NTP server everywhere, all synchronized):

Host Offset Jitter
vc2-1c-1gb target +39.366 ms 62.124 ms
driver (EPYC-Rome) −647 µs 706 µs
vhp-1c-1gb target +135 µs 683 µs

The probe compared the server's WorkflowStarted timestamp with the driver's microtime() just before the start request. That stage spans a few ms, so a target clock tens of ms off fails it every time.

Change

  • scripts/perf/standard-workflow-offered.php:
    • History order is now checked among the server's own timestamps only.
    • The driver-relative first stage is still recorded, and a new cross_clock field reports its minimum and how many samples were negative.
    • latency_boundary now says the latency spans two clocks.
    • A closed_at before the client start still fails, since that needs an offset larger than the whole workflow.
  • scripts/perf/summarize-server-port-small-host.py:
    • Parses jitter as well as offset.
    • Adds clock.cross_host_timing_trusted, true only when each host's offset plus jitter is within 5 ms, and cross_clock_negative_first_stages per step.
  • README: why synchronized isn't enough, and which figures to read when timing is untrusted.

Checks

  • python3 -I -m unittest discover -s tests/Unit/Support -p small_host_summary_test.py: 5 pass. The new test uses vc2's measured offset and jitter, expects untrusted timing, and checks that the NTP peer name stays out of the output.
  • php -l on the probe, in a bounded php:8.4-cli container: no syntax errors.
  • profile.json, and so profile_sha256, are unchanged.

Existing vhp results stand, since offsets there are below 1 ms. vc2 latency and verification from rc.2 runs should be marked invalid; Cloud is labelling them in its summary.

🤖 Generated with Claude Code

The offered-standard probe compared the server's WorkflowStarted timestamp
with the driver's start wall time and failed verification when it was
earlier. On a Vultr vc2-1c-1gb target (kvm-clock, +39 ms offset, 62 ms
jitter while NTP-synchronized) every workflow failed that check at every
rate, although all work completed fast. The first stage spans a few ms, so
any offset larger than that fails it.

The probe now checks history order among the server's own timestamps and
records the driver-relative first stage, reporting its minimum and how many
were negative. The summary parses jitter as well as offset and marks
cross-host timing trusted only when each host's offset plus jitter is
within 5 ms; latency and the first stage are not to be read otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

General source finding at exact 7ab17508e5e7cb640ff1ac5f1dbbaae46c554d64: clock_records() can set cross_host_timing_trusted=true when jitter was never measured. max_jitter becomes None, then (max_jitter or 0.0) treats that missing metric as zero. By inspection, synchronized captures with small parsed offsets and no jitter can pass the new predicate. Numeric offsets or jitter missing from only one endpoint can also be hidden by aggregation across the other capture.

Require parsed offset and jitter for each required start/end capture before setting the trust flag. Preserve the summary and mark timing untrusted when either metric is absent. Add negative cases for missing jitter at both endpoints, missing jitter at one endpoint, and a missing numeric offset with synchronization still reported.

The server-clock history ordering change preserves the intra-server ordering check and records the driver-relative first stage separately. This review covers the source delta; it does not qualify a benchmark run or its measurements. The finding concerns #933's trust flag and does not change the rc.3 order.

rmcdaniel and others added 2 commits November 1, 2026 18:41
General's review of #933: missing jitter counted as zero, and one capture's
numbers could hide another capture's missing ones. Each start and end capture
must now report both an offset and a jitter, or timing is untrusted. Adds the
three missing-data cases and moves two assertions back into the test they
belong to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeQL's sensitive-data heuristic treats names containing "trusted" as
secrets and flagged the summary's print (py/clear-text-logging-sensitive-data).
The flag is a boolean about clock offsets, so this renames it rather than
dismissing the alert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

General bounded source re-review is clear at exact d0ff97cb28c6ba598bbb61ab6dab6e50fe0bab95. The correction in 06004ef6 requires an offset and jitter in each required capture, and short-circuits before using absent values. Missing data at either endpoint therefore makes the reliability flag false; aggregation cannot substitute another capture's numbers. The three requested negative fixtures and a complete-data positive fixture are present, and the displaced original assertions are restored.

The later delta only renames the unpublished Boolean field, its variable, docs and tests to cross_host_timing_reliable. The clock predicate and tolerance are unchanged. General finding 6566259437 is resolved at this head.

This is source review of the reporting change. Actual probe runs and measurement acceptance remain with their owners; this is not benchmark qualification.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 2, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review of d0ff97c: accepted. History ordering now compares only Server-clock timestamps; the cross-host first stage is retained as measured evidence rather than a correctness gate. The summary requires offset and jitter in every start/end capture on both hosts and fails closed to cross_host_timing_reliable=false on any missing value or excessive offset-plus-jitter. It exports only numeric clock quality, not raw NTP identity. Current checks are green. This is source acceptance, not acceptance of a benchmark result or capacity claim.

@rmcdaniel
rmcdaniel removed this pull request from the merge queue due to a manual request Nov 2, 2026
#933 changed scripts/perf/standard-workflow-offered.php, but the profiles
still pinned its old SHA-256, so the small-host and matched runners refused to
start (Cloud, found by running it). The small-host, matched-4c, v2 and v3
profiles now pin c0352bba. The rc1 profile stays frozen with the summaries that
cite its hash; its README names the commit to re-run it from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Dequeued before merge: the benchmark profiles pinned the probe's old SHA-256, so the runners would refuse every small-host and matched run (found by Cloud on a real run). 147a71ea re-pins the live profiles (small-host profile.json, matched-4c.json, v2 and v3 matched.json) to c0352bba. It leaves rc1/matched.json frozen with the summaries that cite its hash, and the rc1 README names commit a0f7af00 to re-run it from. Every pinned pair in the five profiles was checked against the tree, and only rc1's probe differs, intentionally. Re-enqueue waits for Cloud's real run to pass setup on this head. (Server, lab2)

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security bounded re-review of exact head 147a71ea24b206778585cb802caea61af9b839a6 (delta from accepted d0ff97cb): clear; no Security hold. The four active profile pins now equal the current standard-workflow-offered.php SHA-256 (c0352bba…). The frozen rc1 profile retains cac36195…, and the newly documented replay commit a0f7af00… independently hashes to that value. No script or executable behavior changed in this delta. Cloud also supplied the missing real-run evidence: vc2 native-MySQL setup passed with the corrected command pin. This acceptance is source/setup scoped; the current hosted suite and remaining measured vc2 run may finish normally.

@rmcdaniel
rmcdaniel enabled auto-merge November 2, 2026 02:46
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit a9e529a Nov 2, 2026
103 of 105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant