Add commands for event/secret data ingestion by liyu-ma · Pull Request #237 · Azure/kperf

Liyu Ma (liyu-ma) · 2026-03-17T01:00:48Z

Summary

Extend runkperf data with Event, Secret, and Pod ingestion commands for API-server/etcd benchmarks, and consolidate common data-generation helpers.

Changes

Add event, secret, and pod commands with add, list, and delete (del) subcommands.
- Use --group-size to control concurrent creation and deletion requests.
- Page list requests in batches of 300 items when listing or discovering resources for deletion.
- Support client-side rate limiting through --qps and --burst for creation and deletion.
- Generate configurable payloads using --size: Event messages, Secret data, and Pod annotation padding.
- Identify generated resource sets using labels.
Share namespace preparation and the AppLabel constant across data commands, including existing ConfigMap and DaemonSet commands.
Allow --group-size to exceed --total for Event, Secret, and ConfigMap creation.
Consolidate random payload generation in the request package.
- Export RandomPayload and move RandBytes from the data helpers.
- Share one crypto/rand-backed implementation with unbiased alphanumeric sampling.
- Replace data.RandString callers with request.RandomPayload and use request.RandBytes for Secret data.
- Preserve RandomPayload's empty-string behavior for non-positive lengths and panic behavior on entropy-source errors; retain error returns for RandBytes.
Add Pod ingestion tests and expand random-generation tests for payload lengths, allowed characters, invalid inputs, and concurrent use.

Validation

go test ./request ./contrib/cmd/runkperf/commands/data/...
go test -race ./request -run 'Test(RandomPayload|RandBytes|RequestPatchBuilderRendersConcurrencySafe)$'
git diff --check

No live-cluster validation was performed.

Copilot

Pull request overview

This PR extends runkperf’s data command suite to support ingesting Kubernetes Events and Secrets, while refactoring shared Kubernetes client/namespace helpers into contrib/cmd/runkperf/commands/data/client.go for reuse across data subcommands.

Changes:

Add data event subcommands to create/list/delete benchmark Event objects.
Add data secret subcommands to create/list/delete benchmark Secret objects.
Refactor existing data commands to use shared client + namespace helpers, and centralize random data generation utilities.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
contrib/cmd/runkperf/commands/root.go	Registers the new `event` and `secret` data subcommands.
contrib/cmd/runkperf/commands/data/events/event.go	Implements event set creation, listing, and deletion (with rate limiting options).
contrib/cmd/runkperf/commands/data/secrets/secret.go	Implements secret set creation, listing, and deletion (with rate limiting options).
contrib/cmd/runkperf/commands/data/daemonsets/daemonset.go	Refactors to use centralized client + namespace helpers from `data`.
contrib/cmd/runkperf/commands/data/configmaps/configmap.go	Refactors to use centralized helpers and shared random string generator.
contrib/cmd/runkperf/commands/data/client.go	Adds `PrepareNamespace()` and shared random data helpers (`RandBytes`, `RandString`).
.gitignore	Ignores `.claude/` directory.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

This PR extends runkperf data with new subcommands for generating and managing Kubernetes Event and Secret objects, while refactoring shared client/namespace helpers into the data package to reduce duplication across data-ingestion commands.

Changes:

Add data event subcommands (add/list/delete) for generating labeled Event sets.
Add data secret subcommands (add/list/delete) for generating labeled Secret sets.
Refactor shared helpers into contrib/cmd/runkperf/commands/data/client.go and update existing configmap/daemonset commands to use them.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
contrib/cmd/runkperf/commands/root.go	Registers new `data event` and `data secret` subcommands.
contrib/cmd/runkperf/commands/data/secrets/secret.go	Implements secret set add/list/delete with batching + pagination.
contrib/cmd/runkperf/commands/data/events/event.go	Implements event set add/list/delete with batching + pagination.
contrib/cmd/runkperf/commands/data/daemonsets/daemonset.go	Switches to shared `data` client + namespace helper functions.
contrib/cmd/runkperf/commands/data/configmaps/configmap.go	Fixes label var name and switches to shared namespace/random helpers.
contrib/cmd/runkperf/commands/data/client.go	Adds `PrepareNamespace` and random data helpers used across subcommands.
.gitignore	Ignores `.claude/` directory.

Copilot

Pull request overview

Adds new runkperf data subcommands for generating, listing, and deleting Kubernetes Event and Secret objects, while refactoring shared client/helpers into the data client utilities to reduce duplication across existing data commands.

Changes:

Introduce data event and data secret command groups with add/list/del workflows, including paging and configurable concurrency (--group-size).
Refactor shared helpers into contrib/cmd/runkperf/commands/data/client.go (namespace preparation + random payload generators) and migrate existing commands to use them.
Register new subcommands and update ignore rules for local tooling.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
contrib/cmd/runkperf/commands/root.go	Registers the new `events` and `secrets` data subcommands under `runkperf data`.
contrib/cmd/runkperf/commands/data/events/event.go	Implements `data event add/list/del` with paging and parallelism controls.
contrib/cmd/runkperf/commands/data/secrets/secret.go	Implements `data secret add/list/del` with paging and parallelism controls.
contrib/cmd/runkperf/commands/data/daemonsets/daemonset.go	Switches to shared data client helpers (rate-limited client + PrepareNamespace).
contrib/cmd/runkperf/commands/data/configmaps/configmap.go	Switches to shared helpers (PrepareNamespace + RandString + shared AppLabel).
contrib/cmd/runkperf/commands/data/client.go	Adds `PrepareNamespace`, `RandString`/`RandBytes`, and `AppLabel` constant.
.gitignore	Ignores `.claude/` directory.

Fu Wei (fuweid) · 2026-10-21T17:00:39Z

+
+				padding := request.RandomPayload(size)
+
+				pod := &corev1.Pod{


the pod is still schedulable to me. probably we need to add random and non-exist scheduler name, or add taint

Added the fake scheduler as suggested.

Fu Wei (fuweid) · 2026-10-21T17:22:38Z

+	total     int
+}
+
+func createEvents(clientset *kubernetes.Clientset, namespace, eventSetName string, total, groupSize int, reason, eventType string, messageSize int, involvedObjectKind, involvedObjectName string) error {


could you please use option instead of using 10 argus in interface? thanks

Yes now wrap these args to createOption

Fu Wei (fuweid)

there is a lot of duplicated logic among event, secret and pod, like pagination and batched deletion. also, the client type is not consistent: pod uses kubernetes.Interface, while event and secret use *kubernetes.Clientset. could you please extract the common logic and keep the interface consistent? thanks

xinWeiWei24 · 2026-10-22T04:36:04Z

+							"eventSetName": eventSetName,
+						},
+					},
+					InvolvedObject: corev1.ObjectReference{


Would it make sense to group the InvolvedObject fields, EventType, and MessageSize into a struct and pass it in as an option? This could make the event configuration easier to extend and manage.

Sorry just saw your comment here. I added these together with other params to the createOption object. Does that look ok?

Liyu Ma (liyu-ma) · 2026-10-22T05:46:30Z

there is a lot of duplicated logic among event, secret and pod, like pagination and batched deletion. also, the client type is not consistent: pod uses kubernetes.Interface, while event and secret use *kubernetes.Clientset. could you please extract the common logic and keep the interface consistent? thanks

Fu Wei (@fuweid) thanks for the suggestion. I did below improvement accordingly:

create shared operations.go to abstract common func such as batch delete and paginated list.
Use kubernetes.Interface for the added 3 types (better for mock in UT)

I purposely did not touch existing configmap and daemonset types to keep the scope intact. Can do another PR to consolidate these two if you see it needed.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

… with secrets and configmap commands

…onfigmaps add commands The batch loop already caps work with `j < total`, so groupSize larger than total just results in a single batch — the validation adds no value.

Add `runkperf data pod` (alias `po`) with add/delete/list subcommands following the events/secrets pattern. Pods are bare unschedulable pause containers padded via a runkperf.io/padding annotation (default 10 KiB) to load the apiserver/etcd with pod objects. Includes unit tests using a fake clientset.

Reuse crypto/rand-backed helpers across benchmark data commands and expand payload tests. Revert the branch-specific Claude ignore entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Standardize Event, Secret, and Pod helpers on kubernetes.Interface, group Event creation arguments into options, and add regression tests for shared operations and resource adapters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use 64-bit atomic counters to satisfy gosec G115 checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use fake.NewClientset to satisfy staticcheck SA1019. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Liyu Ma (liyu-ma) force-pushed the maliyu/hyperscale-etcd branch from 0a9d91a to 834a06b Compare March 17, 2026 13:32

xinWeiWei24 requested a review from Copilot March 17, 2026 22:55

Copilot started reviewing on behalf of xinWeiWei24 March 17, 2026 22:57

View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Liyu Ma (liyu-ma) requested a review from Copilot March 18, 2026 02:35

Copilot started reviewing on behalf of Liyu Ma (liyu-ma) March 18, 2026 02:35

View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

Comment thread contrib/cmd/runkperf/commands/data/secrets/secret.go Outdated

Comment thread contrib/cmd/runkperf/commands/data/secrets/secret.go Outdated

Comment thread contrib/cmd/runkperf/commands/data/events/event.go Outdated

Liyu Ma (liyu-ma) force-pushed the maliyu/hyperscale-etcd branch from 60d6e23 to 62fdb8d Compare March 20, 2026 05:05

Liyu Ma (liyu-ma) requested a review from Copilot March 20, 2026 05:12

Copilot started reviewing on behalf of Liyu Ma (liyu-ma) March 20, 2026 05:13

View session

Copilot AI reviewed Mar 20, 2026

View reviewed changes

Comment thread contrib/cmd/runkperf/commands/data/secrets/secret.go Outdated

Comment thread contrib/cmd/runkperf/commands/data/client.go

Liyu Ma (liyu-ma) force-pushed the maliyu/hyperscale-etcd branch from a40aaa5 to 9ef6467 Compare October 21, 2026 04:32

Liyu Ma (liyu-ma) requested a review from Fu Wei (fuweid) October 21, 2026 07:00

Fu Wei (fuweid) reviewed Oct 21, 2026

View reviewed changes

xinWeiWei24 reviewed Oct 22, 2026

View reviewed changes

liyu-microsoft and others added 12 commits October 26, 2026 23:35

Add commands for event and secret data ingestion

9f4b40a

add git ignore

b09c440

Reuse PrepareNamespace func

5aa70f9

reuse RandBytes

bc0cc07

Apply rate limiter for deletion

fa063f9

fix linting

4f76a99

Potential fix for pull request finding

635ddd8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

1b16dec

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Address copilot comments

4342364

Consolidate AppLabel

dd953e0

update comments

1c266e5

Update contrib/cmd/runkperf/commands/data/secrets/secret.go

58665e6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

liyu-microsoft and others added 7 commits October 26, 2026 23:35

Rename --message-size to --size in events add command for consistency…

fc71c49

… with secrets and configmap commands

Remove unnecessary group-size > total check in events, secrets, and c…

8da6c98

…onfigmaps add commands The batch loop already caps work with `j < total`, so groupSize larger than total just results in a single batch — the validation adds no value.

Consolidate random payload generation in request package

a35a224

Reuse crypto/rand-backed helpers across benchmark data commands and expand payload tests. Revert the branch-specific Claude ignore entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep ingestion pods unscheduled with a dedicated scheduler name

7019f6b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix narrowing conversions in batch deletion tests

a8dd7bb

Use 64-bit atomic counters to satisfy gosec G115 checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Liyu Ma (liyu-ma) force-pushed the maliyu/hyperscale-etcd branch from 9442e6d to a8dd7bb Compare October 26, 2026 13:38

Replace deprecated fake client constructors in data tests

d122789

Use fake.NewClientset to satisfy staticcheck SA1019. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fu Wei (fuweid) approved these changes Oct 27, 2026

View reviewed changes

Fu Wei (fuweid) merged commit 9fca120 into main Oct 27, 2026
4 checks passed

Conversation

Liyu Ma (liyu-ma) commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Fu Wei (fuweid) Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Liyu Ma (liyu-ma) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Fu Wei (fuweid) Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Liyu Ma (liyu-ma) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Fu Wei (fuweid) left a comment

Choose a reason for hiding this comment

Uh oh!

xinWeiWei24 Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Liyu Ma (liyu-ma) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Liyu Ma (liyu-ma) commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Liyu Ma (liyu-ma) commented Mar 17, 2026 •

edited

Loading