Skip to content

storelimit: synchronize runtime limiters before scheduling - #11214

Draft
JmPotato wants to merge 2 commits into
tikv:masterfrom
JmPotato:storelimit-config-sync-master
Draft

storelimit: synchronize runtime limiters before scheduling#11214
JmPotato wants to merge 2 commits into
tikv:masterfrom
JmPotato:storelimit-config-sync-master

Conversation

@JmPotato

@JmPotato JmPotato commented Sep 9, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #11201

In standalone scheduling deployments, raising an exhausted store limit can leave the store filtered out: filters inspect the stale limiter, while refreshing its rate requires an operator that those filters prevent from being created.

What is changed and how does it work?

Synchronize v1 store limiters with the current configuration before
AddPeer and RemovePeer store selection checks.

Read rates from in-memory configuration snapshots, returning defaults
without inserting missing store entries. Atomically publish each rate
and bucket together to avoid an additional reader lock on the hot path.
Serialize rate changes and preserve the budget when the rate is unchanged.

Preserve v2 behavior, priority exemptions, and zero/Unlimited semantics.

The filter closes the stale-budget loop before operator creation. Existing operator admission and monolithic PD refresh paths continue to perform their own rate checks. The atomic bucket publication is retained because tested lock-based alternatives increased v1 filter time by approximately 1–12%.

Check List

Tests

  • Unit test: eight affected packages passed basic tests; storelimit, filter, and operator passed deadlock/race checks. Coverage includes concurrent refresh sharing one burst and missing-store reads preserving the configuration snapshot.
  • Integration test: real PD API updates and MCS configuration watch followed by scheduler selection, covering exhausted-low-rate recovery, decreases, and Unlimited for AddPeer and RemovePeer. Both deployment modes and the existing admission test passed three deadlock/race runs. The same regression fixture fails on unmodified master for MCS recovery.
  • Manual test: compare master 5abc379473 with this change using identical benchmark sources, Go 1.27.1, AMD Ryzen 9 9955HX, and CPU affinity 4–7. The harness is retained in commit ea62dc73fd: extract the benchmark-only diff for pkg/schedule/filter/store_limit_bench_test.go, pkg/schedule/operator/operator_controller_test.go, and pkg/schedule/schedulers/balance_benchmark_test.go, then apply it to both revisions in separate worktrees. Enable failpoints, build package test binaries, then disable failpoints. Run eight rotating samples with -test.run='^$' -test.benchmem -test.cpu=1,4; select BenchmarkStoreLimitFilter at 150ms, BenchmarkExceedStoreLimit at 300ms, and BenchmarkStoreLimitSchedule at 500ms per sample, and compare with benchstat.

Against master, operator budget checks improve by 2.23% in time geomean, with unchanged 704 B/op and 9 allocs/op. All eight v1 filter cases improve by 1.54–3.39%; all filter cases remain at 0 B/op and 0 allocs/op. Full scheduling shows no statistically significant time regression in four cases (time geomean +0.35%).

Scoped make check passed with golangci-lint v2.6.0 under Go 1.25.12 and without_dashboard; runtime tests and benchmarks used Go 1.27.1. One earlier run of the existing admission test failed when an already-finished operator was removed before its explicit removal assertion; the subsequent three-run check passed. Ten baseline runs did not reproduce it, so the baseline attribution remains unconfirmed.

Side effects

  • Possible performance regression: single-threaded v2 filter cases add approximately 1.1–1.4 ns (0.79–2.36%). One full-scheduling case increases B/op by 0.22%; allocation counts show no significant change. Rate changes allocate a new published state in addition to rebuilding the bucket; this slow path is outside the steady-state measurements. Keeping the admission implementation unchanged gives up some additional optimization in the earlier PR revision: operator time geomean is 0.90% higher than ea62dc7, while remaining 2.23% lower than master. These measurements do not establish zero overhead across all workloads.

Release note

Fix an issue where increasing an exhausted store limit might not restore scheduling in standalone scheduling service deployments.

Refresh v1 limiter rates from in-memory configuration before selecting
stores, so exhausted limits recover after configuration updates in the
standalone scheduling service. Reuse the synchronization in admission
and monolithic PD's eager refresh paths.

Publish rates and buckets atomically to keep reads inexpensive, and avoid
copying configuration or inserting missing entries during rate lookup.
Cover rate transitions and concurrent refresh with regression tests and
benchmarks.

Close tikv#11201

Signed-off-by: JmPotato <github@ipotato.me>
@ti-chi-bot ti-chi-bot Bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Sep 9, 2026
@ti-chi-bot

ti-chi-bot Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the dco. labels Sep 9, 2026
@ti-chi-bot

ti-chi-bot Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lhy1024, yudongusa for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 9, 2026
@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: JmPotato <github@ipotato.me>
@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storelimit: synchronize runtime limiters with configuration updates

1 participant