You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: record and chart tail latency in the nightly run - #1002
Part of #886 (the percentiles and max; jitter is not included).
The nightly perf run only records avg_latency_ms, and perf-compare.py doesn't track latency, so changes in tail latency go unnoticed.
perf-results-to-json.sh adds latency_p50/p90/p95/p99/p999_ms and latency_samples by adding up the client's per-second [LATENCY_HIST] buckets over the steady-state window. latency_max_ms is the largest per-second maximum from [AGGREGATE], which today's client already prints. Results format version 5.
perf-compare.py tracks average, p95 and p99 latency at 10%, 15% and 20%. Max is reported but not flagged: over the last seven nights the window max moved by up to 2x with no code change. The latency metrics ignore the workflow's --threshold 5 override.
Dashboard: a p99 chart for each profile in Overview, p95/p99/max charts in Single and Compare, and a p99 column in both latest-result tables. Older runs show as gaps.
docs/perf-tracking.md lists the new metrics and format version 5.
The percentiles need the client from openmoq/moxygen#630. Until a moxygen sync brings it in, they are null, latency_max_ms is filled, and each run prints a warning that the histogram lines are missing.
Checked so far:
The new parser and the one on main give the same existing fields on all 42 client logs from the last seven nightly runs. Max matches the per-second maxima.
A synthetic log with known buckets gives the expected percentiles, and seconds outside the window are left out.
With --threshold 5, a 30% p99 increase is flagged, a 10% p95 increase is not, and a 2x max increase is reported only.
The dashboard script, run against the published data plus six runs with percentiles, draws every new chart and column.
Still to do: a manual perf-test run with the comparison turned on, after the moxygen sync lands.
The nightly run only recorded avg_latency_ms, and perf-compare.py did
not track latency at all, so a change in tail latency went unnoticed.
- perf-results-to-json.sh adds latency_p50/p90/p95/p99/p999_ms by
summing the client's per-second [LATENCY_HIST] buckets over the
steady-state window. latency_max_ms is the largest per-second interval
max from [AGGREGATE], which today's client already prints. A client
without the histogram lines gives null percentiles and a warning.
Schema version 5.
- perf-compare.py tracks average, p95 and p99 latency at 10%, 15% and
20%, and reports max without flagging it: over the last seven nights
the window max moved by up to 2x with no code change. The latency
metrics ignore the workflow's --threshold 5 override.
- The dashboard gets a p99 chart per profile, p95/p99/max charts per
stack, and a p99 column in both latest-result tables.
Percentiles need moxygen with the per-second latency histogram
(openmoq/moxygen#630); they appear once a moxygen sync brings it in.
No actionable comments were generated in the recent review.
Walkthrough
The performance results script now records latency percentiles, maximum latency, and sample count. Comparison reports apply per-metric latency thresholds. The dashboard displays latency charts and p99 comparison values.
Changes: Latency metrics in performance tracking
Layer / File(s)
Summary
Extract and publish latency results scripts/perf/perf-results-to-json.sh, docs/perf-tracking.md, changelog.d/added/perf-tail-latency.md
The results script aggregates histogram buckets and interval maxima into schema-version-5 JSON fields. Documentation describes the fields, steady-state window, and thresholds.
Average, p95, and p99 latency use per-metric thresholds. Maximum latency has no regression threshold. The --threshold override does not replace latency thresholds.
Display latency metrics status/index.html
The dashboard adds latency charts and p99 values to comparison summaries.
sequenceDiagram
participant ResultsScript as perf-results-to-json.sh
participant ResultsJSON as Results JSON
participant CompareScript as perf-compare.py
participant Dashboard as status/index.html
ResultsScript->>ResultsJSON: write latency percentiles, maximum, and sample count
ResultsJSON->>CompareScript: provide latency metrics for comparison
ResultsJSON->>Dashboard: provide latency values for charts and summaries
Loading
Possibly related issues
Issue 886 requests p90, p99, and maximum latency in nightly performance graphs. This change adds those latency metrics to the results and dashboard; it does not cover the issue’s RFC 3550 jitter request.
Suggested reviewers:peterchave, afrind
Merge Risk: Minimal · up to 1e105
This change adds tail-latency metrics to nightly performance results, the comparison script and the dashboard. No concrete merge-blocking risk was found, and older runs without percentiles appear as gaps.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title states the main change: the nightly performance run now records tail latency and charts it. The changeset adds latency percentiles and maximum latency to the results script, comparison thres…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description states the purpose of the change: adding tail-latency percentiles and max latency to the nightly perf run, the comparison script, and the dashboard. It lists the affected files, the th…
Changelog Entry
The PR makes user-visible changes: nightly runs now record latency percentiles and max, comparison reports flag average, p95 and p99 increases, and the dashboard adds new charts and columns. The PR ad…
Full details: Title check
Explanation
The title states the main change: the nightly performance run now records tail latency and charts it. The changeset adds latency percentiles and maximum latency to the results script, comparison thresholds, docs, and dashboard, so the title matches the primary change.
Full details: Description check
Explanation
The description states the purpose of the change: adding tail-latency percentiles and max latency to the nightly perf run, the comparison script, and the dashboard. It lists the affected files, the thresholds, the format version bump to 5, the dependency on the moxygen client change, and the testing done. It matches the changeset, so the check passes.
Full details: Changelog Entry
Explanation
The PR makes user-visible changes: nightly runs now record latency percentiles and max, comparison reports flag average, p95 and p99 increases, and the dashboard adds new charts and columns. The PR adds a changelog entry at changelog.d/added/perf-tail-latency.md (status A in the diff inventory). The entry is one list item with no PR link, which matches the format in changelog.d/README.md: section added is valid, the file name has no dot, and the content is the list item as it will appear in CHANGELOG.md. The entry covers the user-visible change and links to the performance-tracking docs. The check passes.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Reviewing files that changed from the base of the PR and between 37e24fe and 1e105fc.
Files selected for processing: 5
changelog.d/added/perf-tail-latency.md
docs/perf-tracking.md
scripts/perf/perf-compare.py
scripts/perf/perf-results-to-json.sh
status/index.html
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
📋 Review Summary
The PR correctly implements robust latency percentiles computation (p50/p90/p95/p99/p999 and max) from the client's per-second histogram and aggregate data, and wires these new metrics efficiently into the comparison script and the dashboard. The threshold override logic is carefully structured to bypass the specific latency percentiles, matching the intended behavior without causing false positive regressions on latency jitter.
🔍 General Feedback
Excellent floating-point truncation strategy (- 1e-9) in awk to prevent exact-integer rank boundary errors.
The use of asorti with @ind_num_asc correctly orders histogram buckets, and the data-fallback flows in both awk (for null outputs) and python (None and zero handling) are clean and bug-free.
Safe integration of status/index.html table headers and row generation.
The reason will be displayed to describe this comment to others. Learn more.
[LOW] If the client's [AGGREGATE] interval max latency is genuinely 0 ms (which is technically possible in loopback tests), max (which defaults to 0 in a numeric context) will not be updated because 0 > 0 is false. Consequently, the END block will emit "null" instead of 0.
Consider initializing max = -1 in a BEGIN block and checking max >= 0 to ensure a legitimate 0 ms max latency is correctly recorded.
The reason will be displayed to describe this comment to others. Learn more.
Gemini found nothing above LOW.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #886 (the percentiles and max; jitter is not included).
The nightly perf run only records
avg_latency_ms, andperf-compare.pydoesn't track latency, so changes in tail latency go unnoticed.perf-results-to-json.shaddslatency_p50/p90/p95/p99/p999_msandlatency_samplesby adding up the client's per-second[LATENCY_HIST]buckets over the steady-state window.latency_max_msis the largest per-second maximum from[AGGREGATE], which today's client already prints. Results format version 5.perf-compare.pytracks average, p95 and p99 latency at 10%, 15% and 20%. Max is reported but not flagged: over the last seven nights the window max moved by up to 2x with no code change. The latency metrics ignore the workflow's--threshold 5override.docs/perf-tracking.mdlists the new metrics and format version 5.The percentiles need the client from openmoq/moxygen#630. Until a moxygen sync brings it in, they are null,
latency_max_msis filled, and each run prints a warning that the histogram lines are missing.Checked so far:
--threshold 5, a 30% p99 increase is flagged, a 10% p95 increase is not, and a 2x max increase is reported only.Still to do: a manual
perf-testrun with the comparison turned on, after the moxygen sync lands.This change is