Skip to content

moqtest: per-second latency histogram in the perf client - #630

Open
gmarzot wants to merge 1 commit into
mainfrom
feature/perf-client-latency-hist
Open

gmarzot wants to merge 1 commit into
mainfrom
feature/perf-client-latency-hist

Conversation

@gmarzot

@gmarzot gmarzot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

The perf client only reports latency as a run average plus each second's min/avg/max. The moqx nightly perf run needs p95/p99 to see changes in tail latency (openmoq/moqx#886). The existing Prometheus histogram can't give p99: its buckets double in width, and it covers the whole run, including ramp-up.

  • New LatencyQuantiles.h: a log-linear histogram with 1 ms buckets below 128 ms, then buckets at most 1/32 of their value wide up to 16 s, plus an overflow bucket.
  • moqperf_test_client records every object into it. Each second it prints that second's non-empty buckets as [LATENCY_HIST] [42s] 7:12 8:40 ... 203:3. Each key is the largest latency the bucket holds, so the moqx scripts can add up the seconds in their steady-state window without knowing the bucket layout.
  • The final summary adds whole-run Object Latency p50/p90/p95/p99/p99.9/max.
  • The [AGGREGATE] line and the Prometheus export are unchanged.
  • New LatencyQuantilesTests: bucket boundaries, nearest-rank percentiles, overflow, merge, and the per-second reset.

The moqx side (parsing, comparison, dashboard) is a separate PR. It works with or without this change.


This change is Reviewable

Summary by CodeRabbit

  • New Features
    • Performance test output now includes latency quantiles for individual intervals and the full run, including p50, p90, p95, p99, p99.9, and maximum latency.
    • Latency measurements are grouped into ranges, with high-latency values reported in an overflow bucket.

moqperf_test_client reports latency as a run average plus each second's
min/avg/max. That is not enough to see p95/p99 shifts. The existing
Prometheus histogram doubles its bucket width, so it can't resolve p99,
and it covers the whole run including ramp-up.

The client now also records each object in a log-linear histogram: 1 ms
buckets below 128 ms, then buckets at most 1/32 of their value wide up
to 16 s. Each second it prints that second's non-empty buckets:

  [LATENCY_HIST] [42s] 7:12 8:40 ... 203:3

Each key is the largest latency the bucket holds. A harness can add up
the seconds inside its own steady-state window and read any percentile.
The final summary adds whole-run p50/p90/p95/p99/p99.9/max.

The [AGGREGATE] line and the Prometheus export are unchanged.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds bucketed latency quantile collection to the performance test client. It transfers interval snapshots into aggregation and logs interval and run-level latency statistics.

Changes: Latency quantile metrics

Layer / File(s) Summary
Bucketed quantile collection
moxygen/moqtest/LatencyQuantiles.h, moxygen/moqtest/test/LatencyQuantilesTest.cpp, moxygen/moqtest/test/CMakeLists.txt
LatencyQuantiles maps latency values to exact, logarithmic, and overflow buckets. It supports count accumulation, merging, nearest-rank quantiles, and bucket formatting. AtomicLatencyQuantiles records samples and drains interval snapshots. Tests cover these operations and the CMake file registers the test target.
Client latency interval snapshots
moxygen/moqtest/MoQPerfTestClient.h, moxygen/moqtest/MoQPerfTestClient.cpp
The client records available latency samples in its quantile collector. getResults drains interval counts into TestResults::intervalQuantiles.
Run aggregation and latency reporting
moxygen/moqtest/MoQPerfTestClientMain.cpp
Totals merges interval quantiles, and aggregateStats logs populated interval buckets and accumulates run data. After aggregation, main logs p50, p90, p95, p99, p99.9, and maximum latency when samples exist.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant MoQPerfTestClient
  participant TestResults
  participant aggregateStats
  participant main
  MoQPerfTestClient->>TestResults: provide interval quantiles
  TestResults->>aggregateStats: contribute interval totals
  aggregateStats->>main: accumulate run latency
  main->>main: log run quantiles and maximum
Loading

Suggested reviewers: afrind

Merge Risk: Minimal · up to 227c5

This change adds per-second latency histograms and whole-run percentiles to the perf test client. It does not change existing outputs. No actionable merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "moqtest: per-second latency histogram in the perf client" matches the main change. The PR adds LatencyQuantiles and AtomicLatencyQuantiles, records per-object latency into them, and logs pe…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title "moqtest: per-second latency histogram in the perf client" matches the main change. The PR adds LatencyQuantiles and AtomicLatencyQuantiles, records per-object latency into them, and logs per-second interval histograms plus whole-run p50/p90/p95/p99/p99.9/max in the perf client. The title is short, specific, and uses the module prefix convention. It describes the histogram aspect, while the whole-run percentile summary is a secondary part of the change.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between af08bf4 and 227c54a.

Files selected for processing: 6

  • moxygen/moqtest/LatencyQuantiles.h
  • moxygen/moqtest/MoQPerfTestClient.cpp
  • moxygen/moqtest/MoQPerfTestClient.h
  • moxygen/moqtest/MoQPerfTestClientMain.cpp
  • moxygen/moqtest/test/CMakeLists.txt
  • moxygen/moqtest/test/LatencyQuantilesTest.cpp

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@gmarzot gmarzot self-assigned this Nov 20, 2026
@gmarzot gmarzot added the gemini-review Ask Gemini to review this PR label Nov 20, 2026
@github-actions github-actions Bot removed the gemini-review Ask Gemini to review this PR label Nov 20, 2026
@gmarzot
gmarzot requested review from afrind and peterchave November 20, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant