Skip to content

tests: stabilize TSO consistency after leader change - #11220

Open
rleungx wants to merge 3 commits into
tikv:masterfrom
rleungx:fix-tso-consistency-leader-change-flake
Open

tests: stabilize TSO consistency after leader change#11220
rleungx wants to merge 3 commits into
tikv:masterfrom
rleungx:fix-tso-consistency-leader-change-flake

Conversation

@rleungx

@rleungx rleungx commented Sep 10, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

TestLegacyTSOConsistencySuite/TestRequestTSOConcurrently keeps a direct
gRPC client connected to the original PD leader. After resigning the leader,
the test can send requests to the old endpoint or start before the elected
leader's embedded TSO allocator is ready.

The two concurrent phases also reset their comparison state, so a timestamp
rollback across the leadership transition would not be detected.

Issue Number: close #11219

What is changed and how does it work?

Keep the existing member-election resign trigger. After the new election,
wait for the elected legacy PD leader's embedded TSO allocator to initialize,
and reconnect the direct gRPC client only if another PD becomes leader.

Carry the maximum timestamp from the first concurrent phase into the second,
so every TSO returned after the transition must exceed the pre-transition
high-water mark.

Check List

Tests

  • Legacy targeted test passed 10 consecutive runs
  • Legacy and microservice targeted tests passed 3 race-enabled runs

Release note

None.

Summary by CodeRabbit

  • Tests
    • Improved integration coverage for concurrent timestamp requests during leadership transitions.
    • Verified that timestamps requested in subsequent rounds continue beyond those issued in the initial round.
    • Added coverage for legacy-mode reconnection behavior when leadership changes, while preserving readiness checks.
    • Strengthened validation of leader transitions and timestamp ordering under concurrent request workloads.

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@ti-chi-bot ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has signed the dco. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 10, 2026
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: fba3f5e9-6f8c-4c26-960d-ff75a6c63b62

📥 Commits

Reviewing files that changed from the base of the PR and between 8280d50 and cc31d95.

📒 Files selected for processing (1)
  • tests/integrations/tso/consistency_test.go

Included review availability: Your plan provides up to 4 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The TSO consistency test tracks the maximum timestamp from the first concurrent round, synchronizes the leader transition, conditionally reconnects the legacy client, and passes the first maximum timestamp as the second round's lower bound.

Changes

TSO consistency test

Layer / File(s) Summary
Leader transition and timestamp continuity
tests/integrations/tso/consistency_test.go
requestTSOConcurrently accepts a lower bound, tracks the maximum timestamp atomically, and returns it. The test uses that value for the second round, records the old leader, resigns through GetMember().Resign(), waits for the new leader and allocator, and reconnects the legacy client only when the leader changes.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Suggested reviewers: jmpotato, lhy1024

Merge Risk: ⚪ Minimal · up to cc31d

No actionable merge-blocking risk remains in the reviewed change.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: stabilizing TSO consistency tests after a PD leader change.
Description check ✅ Passed The description covers the problem, linked issue, implementation, test coverage, and release note. The omitted checklist subsections are not required for the described test-only change.
Linked Issues check ✅ Passed The changes satisfy issue #11219 by waiting for the new leader's embedded TSO allocator, reconnecting the direct client when leadership changes, and starting requests after the transition. The timesta…
Out of Scope Changes check ✅ Passed The changes are limited to the flaky TSO consistency test and directly support leader-transition stability and timestamp consistency. No unrelated changes are identified.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot

ti-chi-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.70%. Comparing base (8c1783d) to head (cc31d95).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11220      +/-   ##
==========================================
+ Coverage   79.68%   79.70%   +0.02%     
==========================================
  Files         544      544              
  Lines       78827    78827              
==========================================
+ Hits        62813    62829      +16     
+ Misses      11649    11632      -17     
- Partials     4365     4366       +1     
Flag Coverage Δ
unittests 79.70% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@ti-chi-bot ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 10, 2026
@rleungx

rleungx commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/retest pull-unit-test-next-gen-2

@ti-chi-bot

ti-chi-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@rleungx: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-build
/test pull-build-next-gen
/test pull-check-deps
/test pull-error-log-review
/test pull-integration-realcluster-test
/test pull-unit-test-next-gen-1
/test pull-unit-test-next-gen-2
/test pull-unit-test-next-gen-3

The following commands are available to trigger optional jobs:

/test pull-integration-copr-test
/test pull-integration-realcluster-test-next-gen
/test pull-unit-test

Use /test all to run the following jobs that were automatically triggered:

pull-build
pull-build-next-gen
pull-check-deps
pull-error-log-review
pull-unit-test-next-gen-1
pull-unit-test-next-gen-2
pull-unit-test-next-gen-3
tikv/pd/pull_integration_realcluster_test
tikv/pd/pull_integration_realcluster_test_next_gen
Details

In response to this:

/retest pull-unit-test-next-gen-2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@rleungx

rleungx commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-2

@rleungx
rleungx requested review from JmPotato, bufferflies and lhy1024 and removed request for bufferflies September 10, 2026 07:11
@ti-chi-bot

ti-chi-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: coderabbitai[bot], lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 11, 2026
@ti-chi-bot

ti-chi-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-09-11 02:21:09.994086311 +0000 UTC m=+2023505.165180425: ☑️ agreed by lhy1024.

var wg sync.WaitGroup
var (
wg sync.WaitGroup
maxTS = lowerBound

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using atomic?

Resolve the consistency test conflict and use an atomic maximum TSO.

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx
rleungx requested a review from JmPotato September 11, 2026 03:06
@rleungx

rleungx commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

/retest

1 similar comment
@rleungx

rleungx commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. needs-1-more-lgtm Indicates a PR needs 1 more LGTM. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestLegacyTSOConsistencySuite/TestRequestTSOConcurrently is flaky after leader change

3 participants