Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test df1aaee |
1 of 12 tasks
Contributor
Author
|
/ok to test 5e70463 |
Sum the rows and weights of each cluster in a fixed order instead of with atomics, and use cub::DeviceReduce for the centroid shift and the weight sum. Identical fits now return bitwise-identical centroids, so the convergence check no longer stops at a different iteration from run to run.
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 10, 2026 20:13
5e70463 to
04e5635
Compare
Contributor
Author
|
/ok to test 04e5635 |
Use the environment-based cub::DeviceReduce with an explicit cuda::execution::determinism::run_to_run requirement for the cluster cost, the weight sum and the centroid shift. Temporary storage comes from the caller's workspace buffer through a small memory resource, so the reductions do not allocate on every call. Also let the per-cluster row sums take any data iterator and optional weights, and process the rows in passes of at most 2^22 to bound the workspace.
Contributor
Author
|
/ok to test a4418e0 |
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 10, 2026 21:26
04e5635 to
a4418e0
Compare
Contributor
Author
|
/ok to test 81c966e |
bdice
force-pushed
the
flaky-kmeans-batched
branch
2 times, most recently
from
October 10, 2026 23:49
81c966e to
92165b5
Compare
Contributor
Author
|
/ok to test 92165b5 |
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 11, 2026 00:36
92165b5 to
50a452d
Compare
Contributor
Author
|
/ok to test 50a452d |
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 11, 2026 01:19
50a452d to
b181c97
Compare
Contributor
Author
|
/ok to test b181c97 |
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 11, 2026 01:59
b181c97 to
dad0f83
Compare
Contributor
Author
|
/ok to test dad0f83 |
Sum the rows of each cluster exactly with fixed-point integer atomics instead of floating-point atomics: the result does not depend on the order of the atomics, and the M step still takes two kernels, because the per-column scales are computed once per EM loop. Also derive the random donor candidates from the receiver cluster instead of a shared atomic counter (with a stride coprime with the number of rows, so that every receiver still finds a donor), and rotate the donor offset with the balancing step instead of a process-wide static counter. Group the IVF-PQ training rows by cluster with a stable sort, so the per-cluster codebook training sets do not depend on scheduling.
bdice
force-pushed
the
flaky-kmeans-batched
branch
from
October 11, 2026 02:40
dad0f83 to
caf42ce
Compare
Contributor
Author
|
/ok to test caf42ce |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproduces the flaky
KmeansFitBatchedTestF.Result/Ncentroid mismatch (#2657) in CI on H100 with a reduced CI matrix. Part of #2783.