statistics: stop publishing tombstone store metrics in 7.5

[lhy1024]

What problem does this PR solve?

Issue Number: close #11206, ref #9855, ref #9942

#10060 moved tombstone state handling from Observe into observeStoreStatus without preserving the outer early return. Unlike v7.5.7, release-7.5 now republishes pd_scheduler_store_status for tombstone stores after bury-time cleanup. #10188 clears pd_cluster_status at final deletion, but leaves the republished scheduler metrics behind.

The v7.5.8 scale-in tests expose this in the subsequent region balance checks:

• https://tcms.pingcap.net/dashboard/executions/plan/8231253: removed store 201502 remains in the region-size query.
• https://tcms.pingcap.net/dashboard/executions/plan/8183340: tombstone count reaches zero after remove-tombstone, but scheduler metrics remain in the region-score query.

What is changed and how does it work?

Restore the tombstone early return after publishing cluster state metrics,
and skip hot-stat publication for removed stores.

Clear scheduler store metrics by store ID rather than a hand-maintained
list of types and a potentially outdated address. This also covers CPU,
disk, instant-rate and window metrics in the same metric family.

Use the complete statistics cleanup at final deletion in classic PD and
the scheduling watcher, preserving delete-before-cleanup ordering. Extend
the existing post-observation deletion check to clean scheduler metrics
recreated by stale snapshots as well as cluster status metrics.

This is a targeted release-7.5 fix. It does not backport the broader hotcache/filter/heartbeat cleanup from #11127.

Check List

Tests

• Unit test: tombstone observation preserves state metrics without republishing scheduler or storage metrics, including when rolling statistics still exist.
• Unit test: cleanup covers all scheduler types and old addresses; stale snapshots cannot leave metrics after final deletion.
• Passed: go test ./pkg/statistics ./pkg/mcs/scheduling/server/meta -count=1.
• Passed with failpoints enabled: go test ./pkg/statistics ./pkg/mcs/scheduling/server/meta ./server/cluster -run 'Test.*(Store|Tomb|Bury)' -count=1 -timeout=120s.
• Passed: git diff --check and gofmt.
• make check is blocked during tool installation: the branch's golang.org/x/tools/internal/tokeninternal fails to compile with the local Go toolchain (invalid array length -delta * delta).
• TCMS scale-in validation is pending and will be performed separately.

Release note

Fix stale scheduler store metrics after TiKV scale-in in v7.5.8 that could cause region balance monitoring to include tombstone stores.

Summary by CodeRabbit

• Bug Fixes
  • Corrected metrics cleanup when stores are deleted or tombstoned.
  • Removed stale store status, flow, storage-size, and address metrics after store removal.
  • Prevented deleted or tombstoned stores from continuing to report store-level metrics.
  • Preserved tombstone count metrics while clearing obsolete storage metrics.
  • Improved cleanup for stores removed during monitoring snapshots.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 46615875-0196-4c15-8843-8c0c79236619

📥 Commits

Reviewing files that changed from the base of the PR and between c8d0077 and 99afdb9.

📒 Files selected for processing (4)

• pkg/mcs/scheduling/server/meta/watcher.go
• pkg/statistics/metrics_test.go
• pkg/statistics/store_collection.go
• server/cluster/cluster.go

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Store statistics cleanup

Layer / File(s)	Summary
Skip metrics for removed stores pkg/statistics/store_collection.go	Observe and ObserveHotStat stop updating store-status and store-flow metrics for removed stores. ResetStoreStatistics removes all series matching the store ID, including series under an old address. Stale snapshot cleanup resets store and cluster statistics.
Reset metrics during store deletion pkg/mcs/scheduling/server/meta/watcher.go, server/cluster/cluster.go	Store deletion handlers call ResetStoreStatistics with the store address and formatted store ID.
Validate metric lifecycle pkg/statistics/metrics_test.go	Tests cover tombstone metrics, address changes, and cleanup after deletion during observation.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 99afd

This change removes stale scheduler metrics for tombstoned and deleted stores while retaining cluster-state metrics needed for monitoring. The covered cleanup paths and lifecycle tests indicate it is ready to merge.

Suggested reviewers: ti-chi-bot

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the statistics change and the affected release branch.
Description check	✅ Passed	The description covers the problem, issue references, implementation, tests, known validation limits, scope, and release note. It is complete enough for review.

• Fix all pre-merge checks with AI

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[lhy1024]

Before:

After:

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, coderabbitai[bot]

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bufferflies]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-09-10 02:57:01.84717151 +0000 UTC m=+1939257.018265620: ☑️ agreed by bufferflies.