Skip to content

[ai-assisted=yes] Emit disk read/write operation counts - #343

Merged
jorbaum merged 1 commit into
cloudfoundry:mainfrom
ivan-asenchakarov-sys:tnz-disk-iops-metrics
Sep 16, 2026
Merged

jorbaum merged 1 commit into
cloudfoundry:mainfrom
ivan-asenchakarov-sys:tnz-disk-iops-metrics

Conversation

@ivan-asenchakarov-sys

@ivan-asenchakarov-sys ivan-asenchakarov-sys commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Adds system_disk_{system,ephemeral,persistent}_{read,write}_count gauges alongside the existing byte-throughput gauges.

There's currently no IOPS metric, only bytes/sec. gopsutil's disk.IOCountersStat already returns ReadCount/WriteCount per device; the collector just wasn't reading them.

Testing Done

  • Extended the existing collector and Prometheus-sender unit tests with the new fields/gauges.
  • Deployed to a real foundation across 3 different services, replacing the stock release via a runtime-config override. New counts landed live and increased over time:
Instance system_disk_persistent_read_count system_disk_persistent_write_count
VM 1 482 473
VM 2 363 179 → 348 → 456 → 511
VM 3 163 342 → 425 → 425

Also checked the system/ephemeral disk gauges on VM 1, all live and moving:

Metric Values
system_disk_system_read_count 3780 → 3893
system_disk_system_write_count 1513 → 1721 → 1938 → 2148 → 2240
system_disk_ephemeral_read_count 86
system_disk_ephemeral_write_count 3875 → 4120 → 4315 → 4520 → 4618

@linux-foundation-easycla

linux-foundation-easycla Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: ivan-asenchakarov-sys / name: ic005077 (2274b04)

@ivan-asenchakarov-sys
ivan-asenchakarov-sys force-pushed the tnz-disk-iops-metrics branch 2 times, most recently from 0a6397e to 39f807c Compare September 13, 2026 12:12
@ivan-asenchakarov-sys ivan-asenchakarov-sys changed the title Emit disk read/write operation counts [TNZ-145560] Emit disk read/write operation counts Sep 13, 2026
Comment thread src/pkg/egress/stats/prometheus_sender.go Outdated

@jorbaum jorbaum left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing. LGTM in general. Once you have signed the CLA we can go forward with this.

go test currently fails:

go test ./...
...
  [FAILED] Expected
      <int>: 144
  to equal
      <int>: 126
  In [It] at: /Users/myuser/Workspace/public/cloudfoundry/system-metrics-release/src/cmd/system-metrics-agent/app/system_metrics_agent_test.go:122 @ 09/15/26 08:07:49.52

AFAICS /tmp/expected-metrics in scripts/integration-tests.sh also needs your new metrics.

Please also disclose AI usage (if any).

@ivan-asenchakarov-sys
ivan-asenchakarov-sys force-pushed the tnz-disk-iops-metrics branch 2 times, most recently from fb16e92 to 2274b04 Compare September 15, 2026 12:46
@ivan-asenchakarov-sys ivan-asenchakarov-sys changed the title [TNZ-145560] Emit disk read/write operation counts [TNZ-145560][ai-assisted=yes] Emit disk read/write operation counts Sep 15, 2026
@ivan-asenchakarov-sys

Copy link
Copy Markdown
Contributor Author

Thanks @jorbaum , I have addressed your comment.
I have also added a ai-assisted=yes in the title

@ay901246 ay901246 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just need to sign the CLA; small nit is that you may want to remove the TNZ ticket title since it's not technically relevant to open source.

@ivan-asenchakarov-sys ivan-asenchakarov-sys changed the title [TNZ-145560][ai-assisted=yes] Emit disk read/write operation counts [ai-assisted=yes] Emit disk read/write operation counts Sep 15, 2026
@ivan-asenchakarov-sys

Copy link
Copy Markdown
Contributor Author

@ay901246 thanks, I have removed the ticket from the title

jorbaum
jorbaum previously approved these changes Sep 16, 2026

@jorbaum jorbaum left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM.

Ah. One last thing: The documentation at https://github.com/cloudfoundry/system-metrics-release/blob/main/docs/system-metrics-agent.md is missing the new metrics. it looks to me like you need to add them there as well.

DiskIOCountersWithContext's underlying gopsutil call already returns
ReadCount/WriteCount per disk device; the collector read every other
field off that struct but silently dropped these two. Wires them
through DiskStat and emits them as system_disk_{system,ephemeral,
persistent}_{read,write}_count gauges, alongside the existing byte
counters.

Ref: TNZ-145560
@ivan-asenchakarov-sys

Copy link
Copy Markdown
Contributor Author

@jorbaum I added to docs, thank you.

@jorbaum jorbaum left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jorbaum
jorbaum merged commit 54a2965 into cloudfoundry:main Sep 16, 2026
9 checks passed
@ivan-asenchakarov-sys

Copy link
Copy Markdown
Contributor Author

Hey @jorbaum what needs to be done now so a GH release is made with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants