Skip to content

perf: record and chart tail latency in the nightly run - #1002

Open
gmarzot wants to merge 1 commit into
mainfrom
feature/perf-tail-latency
Open

gmarzot wants to merge 1 commit into
mainfrom
feature/perf-tail-latency

Conversation

@gmarzot

@gmarzot gmarzot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Part of #886 (the percentiles and max; jitter is not included).

The nightly perf run only records avg_latency_ms, and perf-compare.py doesn't track latency, so changes in tail latency go unnoticed.

  • perf-results-to-json.sh adds latency_p50/p90/p95/p99/p999_ms and latency_samples by adding up the client's per-second [LATENCY_HIST] buckets over the steady-state window. latency_max_ms is the largest per-second maximum from [AGGREGATE], which today's client already prints. Results format version 5.
  • perf-compare.py tracks average, p95 and p99 latency at 10%, 15% and 20%. Max is reported but not flagged: over the last seven nights the window max moved by up to 2x with no code change. The latency metrics ignore the workflow's --threshold 5 override.
  • Dashboard: a p99 chart for each profile in Overview, p95/p99/max charts in Single and Compare, and a p99 column in both latest-result tables. Older runs show as gaps.
  • docs/perf-tracking.md lists the new metrics and format version 5.

The percentiles need the client from openmoq/moxygen#630. Until a moxygen sync brings it in, they are null, latency_max_ms is filled, and each run prints a warning that the histogram lines are missing.

Checked so far:

  • The new parser and the one on main give the same existing fields on all 42 client logs from the last seven nightly runs. Max matches the per-second maxima.
  • A synthetic log with known buckets gives the expected percentiles, and seconds outside the window are left out.
  • With --threshold 5, a 30% p99 increase is flagged, a 10% p95 increase is not, and a 2x max increase is reported only.
  • The dashboard script, run against the published data plus six runs with percentiles, draws every new chart and column.

Still to do: a manual perf-test run with the comparison turned on, after the moxygen sync lands.


This change is Reviewable

The nightly run only recorded avg_latency_ms, and perf-compare.py did
not track latency at all, so a change in tail latency went unnoticed.

- perf-results-to-json.sh adds latency_p50/p90/p95/p99/p999_ms by
  summing the client's per-second [LATENCY_HIST] buckets over the
  steady-state window. latency_max_ms is the largest per-second interval
  max from [AGGREGATE], which today's client already prints. A client
  without the histogram lines gives null percentiles and a warning.
  Schema version 5.
- perf-compare.py tracks average, p95 and p99 latency at 10%, 15% and
  20%, and reports max without flagging it: over the last seven nights
  the window max moved by up to 2x with no code change. The latency
  metrics ignore the workflow's --threshold 5 override.
- The dashboard gets a p99 chart per profile, p95/p99/max charts per
  stack, and a p99 column in both latest-result tables.

Percentiles need moxygen with the per-second latency histogram
(openmoq/moxygen#630); they appear once a moxygen sync brings it in.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Automatic approval withheld

Assessment Current risk Maximum for approval
Merge risk Minimal Low
Security architecture risk Unavailable for this commit Low

Commit: 1e105fc046f1.

No actionable comments were generated in the recent review.

Walkthrough

The performance results script now records latency percentiles, maximum latency, and sample count. Comparison reports apply per-metric latency thresholds. The dashboard displays latency charts and p99 comparison values.

Changes: Latency metrics in performance tracking

Layer / File(s) Summary
Extract and publish latency results
scripts/perf/perf-results-to-json.sh, docs/perf-tracking.md, changelog.d/added/perf-tail-latency.md
The results script aggregates histogram buckets and interval maxima into schema-version-5 JSON fields. Documentation describes the fields, steady-state window, and thresholds.
Compare latency metrics
scripts/perf/perf-compare.py, docs/perf-tracking.md
Average, p95, and p99 latency use per-metric thresholds. Maximum latency has no regression threshold. The --threshold override does not replace latency thresholds.
Display latency metrics
status/index.html
The dashboard adds latency charts and p99 values to comparison summaries.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ResultsScript as perf-results-to-json.sh
  participant ResultsJSON as Results JSON
  participant CompareScript as perf-compare.py
  participant Dashboard as status/index.html
  ResultsScript->>ResultsJSON: write latency percentiles, maximum, and sample count
  ResultsJSON->>CompareScript: provide latency metrics for comparison
  ResultsJSON->>Dashboard: provide latency values for charts and summaries
Loading

Possibly related issues

  • Issue 886 requests p90, p99, and maximum latency in nightly performance graphs. This change adds those latency metrics to the results and dashboard; it does not cover the issue’s RFC 3550 jitter request.

Suggested reviewers: peterchave, afrind

Merge Risk: Minimal · up to 1e105

This change adds tail-latency metrics to nightly performance results, the comparison script and the dashboard. No concrete merge-blocking risk was found, and older runs without percentiles appear as gaps.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title states the main change: the nightly performance run now records tail latency and charts it. The changeset adds latency percentiles and maximum latency to the results script, comparison thres…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description states the purpose of the change: adding tail-latency percentiles and max latency to the nightly perf run, the comparison script, and the dashboard. It lists the affected files, the th…
Changelog Entry Passed The PR makes user-visible changes: nightly runs now record latency percentiles and max, comparison reports flag average, p95 and p99 increases, and the dashboard adds new charts and columns. The PR ad…
Full details: Title check

Explanation

The title states the main change: the nightly performance run now records tail latency and charts it. The changeset adds latency percentiles and maximum latency to the results script, comparison thresholds, docs, and dashboard, so the title matches the primary change.

Full details: Description check

Explanation

The description states the purpose of the change: adding tail-latency percentiles and max latency to the nightly perf run, the comparison script, and the dashboard. It lists the affected files, the thresholds, the format version bump to 5, the dependency on the moxygen client change, and the testing done. It matches the changeset, so the check passes.

Full details: Changelog Entry

Explanation

The PR makes user-visible changes: nightly runs now record latency percentiles and max, comparison reports flag average, p95 and p99 increases, and the dashboard adds new charts and columns. The PR adds a changelog entry at changelog.d/added/perf-tail-latency.md (status A in the diff inventory). The entry is one list item with no PR link, which matches the format in changelog.d/README.md: section added is valid, the file name has no dot, and the content is the list item as it will appear in CHANGELOG.md. The entry covers the user-visible change and links to the performance-tracking docs. The check passes.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Automatic approval policy

Commit: 1e105fc046f1.

Setting Effective value Source
Automatic approvals Enabled Global configuration
Merge risk Low Global configuration
Security risk Low Global configuration

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

  • Source mode: Auto
  • Repository: openmoq/moqx
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: 37e24fe
  • Layers: Repository YAML (base), Repository UI (inherited)
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e2ff13a7-35cc-4021-88cb-b2903805251d
Commits

Reviewing files that changed from the base of the PR and between 37e24fe and 1e105fc.

Files selected for processing: 5

  • changelog.d/added/perf-tail-latency.md
  • docs/perf-tracking.md
  • scripts/perf/perf-compare.py
  • scripts/perf/perf-results-to-json.sh
  • status/index.html

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@gmarzot gmarzot self-assigned this Nov 17, 2026
@gmarzot
gmarzot requested review from afrind and peterchave November 17, 2026 20:23
@gmarzot gmarzot added the gemini-review Ask Gemini to review this PR label Nov 17, 2026
@github-actions github-actions Bot removed the gemini-review Ask Gemini to review this PR label Nov 17, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

The PR correctly implements robust latency percentiles computation (p50/p90/p95/p99/p999 and max) from the client's per-second histogram and aggregate data, and wires these new metrics efficiently into the comparison script and the dashboard. The threshold override logic is carefully structured to bypass the specific latency percentiles, matching the intended behavior without causing false positive regressions on latency jitter.

🔍 General Feedback

  • Excellent floating-point truncation strategy (- 1e-9) in awk to prevent exact-integer rank boundary errors.
  • The use of asorti with @ind_num_asc correctly orders histogram buckets, and the data-fallback flows in both awk (for null outputs) and python (None and zero handling) are clean and bug-free.
  • Safe integration of status/index.html table headers and row generation.

}
/\[AGGREGATE\]/ {
if (!match($0, /\[([0-9]+)s\]/, t) || !inwin(t[1] + 0)) next
if (match($0, /interval min\/avg\/max\): [0-9]+\/[0-9.]+\/([0-9]+) ms/, x) && x[1] + 0 > max) max = x[1] + 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] If the client's [AGGREGATE] interval max latency is genuinely 0 ms (which is technically possible in loopback tests), max (which defaults to 0 in a numeric context) will not be updated because 0 > 0 is false. Consequently, the END block will emit "null" instead of 0.

Consider initializing max = -1 in a BEGIN block and checking max >= 0 to ensure a legitimate 0 ms max latency is correctly recorded.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gemini found nothing above LOW.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant