Skip to content

MOD-19327: Log and export the INFO fields a benchmark asks for (info_… - #574

Merged
JoanFM merged 4 commits into
redis-performance:masterfrom
nonirosenfeldredis:sharon-MOD-19327-info-metrics
Nov 5, 2026
Merged

JoanFM merged 4 commits into
redis-performance:masterfrom
nonirosenfeldredis:sharon-MOD-19327-info-metrics

Conversation

@nonirosenfeldredis

@nonirosenfeldredis nonirosenfeldredis commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

A benchmark could not surface a module counter that explains its result: the runners only keep memory, CPU and the search memory/disk fields.

  • add an optional info_metrics list to the benchmark definition (section and fields), read once after the client tool finished
  • log the values as INFO metrics: {...} and export them with the other end of run metrics, in both run-remote and run-local
  • sum across shards, skip malformed entries and missing sections, so this never fails a run
  • add an ignore_shard_errors argument to collect_redis_metrics: a failing section of one shard is logged and skipped, so the other shards of a cluster still report. Without it only a ConnectionError is tolerated, as before
  • document the key and cover the collection helper, the shard error handling and the run-local wiring with tests (the run-remote wiring needs real infrastructure and is not covered)

🤖 Generated with Claude Code

…metrics)

A benchmark could not surface a module counter that explains its result:
the runners only keep memory, CPU and the search memory/disk fields.

- add an optional `info_metrics` list to the benchmark definition (section and
  fields), read once after the client tool finished
- log the values as `INFO metrics: {...}` and export them with the other end
  of run metrics, in both run-remote and run-local
- sum across shards, skip malformed entries and missing sections, so this
  never fails a run
- document the key and cover the collection helper with tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

This is a useful, well-scoped feature and the commit history already does most of the work. Moving the best-effort guard into collect_info_metrics so run-local and run-remote can't drift was the right call. ignore_shard_errors defaults to False, so existing callers (collect_search_and_bigredis_metrics, the default end-of-run collection) keep their behaviour, and test_collect_redis_metrics_ignore_shard_errors pins that. I checked the run-remote side: info_metrics is assigned in the same straight-line block as cpu_usage_summary before it's used in the export path, so there's no stale or undefined value across loop iterations. Two things, most important first:

  1. The run-local wiring test passes even without the new feature. test_run_local_exports_the_info_metrics_the_benchmark_asks_for requests memory / used_memory and then asserts seen[0]["memory_used_memory"] > 0. But run_local.py:667-676 already collects memory_used_memory by default (the ["memory"] / used_memory, used_memory_dataset call) before info_metrics runs. So if the collect_info_metrics(...) block were removed, the test would still pass. Asking for a field that isn't in the default set should fix it, e.g. memory / used_memory_peak, or stats / total_commands_processed. Then assert on that key.

  2. Partial sums on shard failure look like real values downstream. Not a blocker, just worth a decision. With ignore_shard_errors=True, if one of three shards times out, the summed counter drops by about a third and gets exported to RedisTimeSeries next to the full-sum datapoints. Nothing marks it as partial: the warning only goes to the log. A drop like that on a dashboard is easy to mistake for a regression. I can see the argument for "some data beats none" (the docs say so explicitly). The alternative is to skip the summed scalar for a section when any shard failed it, and still report dict-valued per-shard fields. This repo's history doesn't give me a precedent either way, so it may be worth a second look from whoever owns the RTS dashboards.

Everything else looks good: malformed or non-list config is skipped with a warning, missing sections are simply absent, and None/non-numeric values never get summed. That's the class of bug PR#534 had to fix, and this code avoids it. Docs are updated in docs/Readme.md. With (1) fixed I think this is good to go.

@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.66667% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.36%. Comparing base (ab8d879) to head (9925002).

Files with missing lines Patch % Lines
redisbench_admin/run_remote/run_remote.py 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #574      +/-   ##
==========================================
+ Coverage   50.20%   50.36%   +0.16%     
==========================================
  Files          74       74              
  Lines        9194     9224      +30     
==========================================
+ Hits         4616     4646      +30     
  Misses       4578     4578              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nonirosenfeldredis and others added 2 commits October 27, 2026 11:44
Review feedback: the "never fails a run" promise only held in run-remote.

- move the guard into collect_info_metrics, so run-remote and run-local
  cannot drift: INFO errors are logged and yield no metrics, and a value
  that is not a list is ignored with a warning
- drop the now redundant try/except at the run-remote call site
- document that numeric scalar values are summed across shards
- cover a non-list value and an INFO error with tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ics)

- collect_info_metrics wrapped every shard in one try, so a TimeoutError from
  a single slow shard dropped the values of all of them on a cluster
- keep the single call for the normal path; only when it raises and there is
  more than one connection, read the shards one by one, skip the failing ones
  with a warning and sum the rest, so partial results are still reported
- document the behaviour in docs/Readme.md
- test the partial and the all-shards-failing cases, and an entry list that is
  entirely malformed
- test the wiring in run_local: a benchmark with info_metrics set reaches the
  end of run metrics (run_remote needs real infrastructure and stays untested)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@nonirosenfeldredis

Copy link
Copy Markdown
Contributor Author

@JoanFM pls review when u can
thanks!

@JoanFM JoanFM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small change, I think this is a great feature!

Comment thread redisbench_admin/run/metrics.py Resolved
- review feedback: instead of retrying shard by shard in collect_info_metrics,
  give collect_redis_metrics an ignore_shard_errors argument so it skips a
  failing section of a shard itself and the other shards still report
- collect_info_metrics goes back to a single call, which also keeps the
  _shard_N suffix of dict-valued fields when a shard fails
- default behaviour is unchanged: without the flag only a ConnectionError is
  tolerated and any other error still raises
- test the flag on collect_redis_metrics directly

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants