Skip to content

MOD-19327: Read numbers out of the reply of a command at the end of a run (command_metrics) - #575

Closed
nonirosenfeldredis wants to merge 1 commit into
redis-performance:masterfrom
nonirosenfeldredis:sharon-MOD-19327-command-metrics
Closed

nonirosenfeldredis wants to merge 1 commit into
redis-performance:masterfrom
nonirosenfeldredis:sharon-MOD-19327-command-metrics

Conversation

@nonirosenfeldredis

Copy link
Copy Markdown
Contributor

Follow-up of #574.

info_metrics reads INFO fields. Some counters are not in INFO, only in the reply of a command, for example the debug info of a RediSearch vector index (FT.DEBUG VECSIM_INFO: id holes, next id, compaction moves and time of the disk vector index). A benchmark could not surface them.

  • add an optional command_metrics list to the benchmark definition: a name, the command (argument list) and the dotted fields paths of the numbers to read out of its reply
  • read a reply of alternating names and values (a value may be such a list again) as nested names, so BACKEND_INDEX.NUM_ID_HOLES is a path into it
  • run the command once per shard after the client tool finished, sum the numbers across shards like the INFO metrics, log them as Command metrics: {...} and export them with the other end of run metrics, in both run-remote and run-local
  • a malformed entry, a path that is missing or not a number, or a command that fails on a shard is logged and left out, so this never fails a run
  • document the key; the command must only read, which nothing checks
command_metrics:
  - name: vecsim_info
    command: ["_FT.DEBUG", "VECSIM_INFO", "idx", "vec"]
    fields:
      - BACKEND_INDEX.NUM_ID_HOLES
      - TIERED_HNSW_DISK_GC_MOVES

The metric names are <name>_<path with the dots as underscores>.

Tests: the helper (nested reply, sums across shards, missing or non-numeric paths, malformed entries, a non-list value, a failing shard, every shard failing) and the run-local wiring against a real redis-server. The run-remote wiring needs real infrastructure and is not covered.

🤖 Generated with Claude Code

… run (command_metrics)

Some counters are not in INFO, only in the reply of a command, for example the debug info of a
RediSearch vector index (FT.DEBUG VECSIM_INFO). The info_metrics of a benchmark cannot reach them.

- add an optional command_metrics list to the benchmark definition: a name, the command and the
  dotted paths of the numbers to read out of its reply
- read the reply of alternating names and values (a value may be such a list again) as nested names
- run the command once per shard after the client tool finished, sum the numbers across shards, log
  them as Command metrics and export them with the other end of run metrics, in run-remote and run-local
- a malformed entry, a path that is missing or not a number, or a command that fails on a shard is
  logged and left out, so this never fails a run
- document the key and cover the helper and the run-local wiring with tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

This looks good to me, and it follows the info_metrics shape from #574 closely. Here's what I checked:

  • The run-local call goes in right after collect_info_metrics and before post_commands, so the existing comment ("post_commands cannot mutate measured state") still holds for the new metrics.
  • In run-remote, command_metrics is assigned in the same block as info_metrics (around line 1180), and both are merged at the same export point. I found no new path where the variable is used before it's set.
  • It never fails a run: malformed entries, a missing path, a non-numeric value, a bool, and a shard that raises are all skipped and logged. This matches how metrics: skip None / non-numeric metric values instead of crashing #534 handles values it can't use. The tests cover each of these cases, plus summing across two shards.

A few small things, none of them blockers:

  1. RESP3 replies. _reply_to_tree handles bytes, lists and tuples, but returns a dict unchanged. If a connection ever speaks RESP3, FT.DEBUG can come back as a map with bytes keys. Every path lookup would then miss without any warning, and the metric would just be absent. flat_reply_to_dict in run/common.py already handles this case by decoding the keys. You can't import it here, because run.common already imports run.metrics and that would create a cycle, so writing a separate helper makes sense. Adding a dict branch that decodes keys and recurses into values would cover it.
  2. Name collisions. The output goes into overall_end_time_metrics with a plain update. A name + path such as memory + used_memory would silently overwrite the INFO metric with the same name. This is unlikely in practice, but logging a warning when a key already exists would be cheap.
  3. Lists that look like name/value pairs. Any even-length list whose even positions are strings becomes a dict, so a plain list of names like [b"a", b"b"] is read as {a: b}. Only numbers are read out in the end, so I don't think this causes a wrong value today. It's worth a line in the docstring so nobody is surprised later.

Coverage: the run-remote wiring isn't tested, which the description says up front. CONTRIBUTING.md asks for new behaviour to be tested, so Codecov will probably show those lines as uncovered. The change there is the same 15-line pattern as the info_metrics call next to it, so I don't think that should hold this up.

Thanks for the thorough description and the tests. Happy to see this land once someone who knows the run-remote export path has taken a look.

(I'd mark this as COMMENTED, leaning toward approval.)

@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.62%. Comparing base (626f30b) to head (42d298e).

Files with missing lines Patch % Lines
redisbench_admin/run/metrics.py 92.85% 4 Missing ⚠️
redisbench_admin/run_remote/run_remote.py 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #575      +/-   ##
==========================================
+ Coverage   50.33%   50.62%   +0.28%     
==========================================
  Files          74       74              
  Lines        9224     9288      +64     
==========================================
+ Hits         4643     4702      +59     
- Misses       4581     4586       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant