Skip to content

TL/UCP: add implementation for team-based segments - #1301

Open
wfaderhold21 wants to merge 2 commits into
openucx:masterfrom
wfaderhold21:topic/team_segments
Open

TL/UCP: add implementation for team-based segments#1301
wfaderhold21 wants to merge 2 commits into
openucx:masterfrom
wfaderhold21:topic/team_segments

Conversation

@wfaderhold21

Copy link
Copy Markdown
Collaborator

What

Implements a team-based memory map param support for TL/UCP.

Why ?

This adds an expensive path over team creation for TL/UCP if the user sets the memory map parameters during team creation. It enables segments that are restricted to a team rather than a context and would be useful for (1) OpenSHMEM Team's support and (2) using global-work buffer for features such as PR #1248 and PR #1149 to enable onesided scratch buffers.

@greptile-apps

greptile-apps Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds team-scoped memory segment registration for TL/UCP, enabling segments bound to a team rather than a context. A two-phase allgather protocol runs during team_create_test to distribute UCP rkeys, and resolve_p2p_by_va is extended to consult eager-unpacked team rkeys before the per-collective dst_memh path.

  • Core (tl_ucp_team_mem.c): size_exch, data_exch, and finalize drive a mem_map_phase state machine; destroy handles cleanup on both success and error paths.
  • Team creation (tl_ucp_team.c): state machine polled from team_create_test, avoiding direct task->progress() calls; team_destroy calls destroy unconditionally.
  • Test (test_alltoall.cc): new single_team_onesided test with per-team onesided buffers allocated in init_team and freed in ~UccTeam.

Confidence Score: 5/5

The change is safe to merge; core logic, error cleanup, and task lifecycle are all handled correctly.

The two-phase allgather protocol, rkey bookkeeping, and destroy-path cleanup are implemented carefully. The resolve_p2p_by_va extension correctly uses the pre-remapped team rank and the caller pre-initialized endpoint. No functional defects were found.

test/gtest/common/test_ucc.h and test_ucc.cc have the hardcoded array size and dangling-pointer lifetime note worth addressing before UCC_TEST_N_MEM_SEGMENTS is ever changed.

Important Files Changed

Filename Overview
src/components/tl/ucp/tl_ucp_team_mem.c New file: 3-phase team-scoped memory registration with thorough error cleanup; in-flight task cancellation documented as single-threaded only.
src/components/tl/ucp/tl_ucp_team_mem.h New header declaring the four team-mem lifecycle functions; includes tl_ucp.h, resolving the duplicate-declaration issue from prior review threads.
src/components/tl/ucp/tl_ucp.h Added 10 new fields to ucc_tl_ucp_team_t, three accessor macros, and moved allgather forward declaration here as canonical site.
src/components/tl/ucp/tl_ucp_sendrecv.h Step 2 added in resolve_p2p_by_va to search team segments; correctly uses pre-remap team_peer and caller-initialized ep.
src/components/tl/ucp/tl_ucp_team.c mem_map_phase state machine added to team_create_test; avoids direct task->progress() calls; destroy called unconditionally in team_destroy.
test/gtest/common/test_ucc.cc UccTeam extended with team-owned onesided buffers; seg_maps lifetime sufficient today but leaves a technically-dangling pointer in stored team params.
test/gtest/common/test_ucc.h Added is_team_onesided flag, per-proc onesided_buf array, and onesided_buf() accessor; proc::onesided_buf hardcodes 3 instead of UCC_TEST_N_MEM_SEGMENTS.
test/gtest/coll/test_alltoall.cc New single_team_onesided test exercises team-scoped segment path via alltoall ring algorithm.
src/components/tl/ucp/tl_ucp.c Removed redundant forward declarations of service_allgather and service_test; canonical declarations moved to tl_ucp.h.

Reviews (3): Last reviewed commit: "TEST: add team-based onesided tests" | Re-trigger Greptile

Comment thread src/components/tl/ucp/tl_ucp_team_mem.c Outdated
Comment on lines +289 to +295
ucs_status = ucp_ep_rkey_unpack(
ep, elem + 24, &UCC_TL_UCP_TEAM_RKEY(team, r, s));
if (UCS_OK != ucs_status) {
tl_error(UCC_TL_TEAM_LIB(team),
"ucp_ep_rkey_unpack failed for rank %u seg %lu: %s",
r, s, ucs_status_string(ucs_status));
return ucs_status_to_ucc_status(ucs_status);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Early return leaves rbuf and task un-cleaned

When ucp_ep_rkey_unpack fails for any rank, the function returns immediately without freeing team->mem_map_allgather_rbuf or finalizing team->mem_map_task. The cleanup relies entirely on ucc_tl_ucp_team_mem_map_destroy being called during teardown — this is safe today because team creation failure always leads to team_destroy, but the asymmetry makes the invariant fragile. Additionally, rkeys already unpacked for ranks 0..r-1 are left in team_rkeys[]; those are properly destroyed by the destroy function (NULL-checked loop), so no handle leak occurs — but it is worth making this contract explicit.

Comment on lines +250 to +256
ucc_status_t ucc_tl_ucp_service_test(ucc_coll_task_t *task)
{
if (task->status == UCC_INPROGRESS) {
task->progress(task);
}
return task->status;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 ucc_tl_ucp_service_test directly calls task->progress() for in-queue tasks

tl_ucp_team.c has an explicit warning (lines 270-273) that calling task->progress(task) directly on a task still in the context progress queue risks completing it while enqueued, corrupting the list on the next ucc_context_progress() call. This new helper does exactly that. While it is not called anywhere in this PR, having it available creates a footgun for future callers who use it with tasks posted by ucc_tl_ucp_service_allgather.

Comment thread src/components/tl/ucp/tl_ucp_team_mem.h Outdated
Comment on lines +13 to +16
ucc_status_t ucc_tl_ucp_service_allgather(ucc_base_team_t *team, void *sbuf,
void *rbuf, size_t msgsize,
ucc_subset_t subset,
ucc_coll_task_t **task_p);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Duplicate forward declaration of ucc_tl_ucp_service_allgather

ucc_tl_ucp_service_allgather is already forward-declared in tl_ucp.c (line 353). Declaring it again here creates two independent declaration sites that must stay in sync. Consider including the existing TL header that exposes this declaration, or removing the duplicate.

Comment thread src/components/tl/ucp/tl_ucp_team_mem.c Outdated
Comment on lines +198 to +200
if (n_segs == 0 || size == 0) {
return UCC_OK;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Dead defensive guard that conflicts with the caller's error check

The if (n_segs == 0 || size == 0) early return is unreachable: the call site in tl_ucp_team.c only enters the mem-map path when mem_params.n_segments > 0, and size == 0 cannot occur for a valid team. More importantly, the caller checks if (status != UCC_INPROGRESS) { goto err_preconnect; } — if this branch were ever taken it would goto err_preconnect with status == UCC_OK, silently marking team creation as complete without finishing the data exchange. Consider removing the guard, or if kept, handle UCC_OK explicitly in the caller.

@wfaderhold21
wfaderhold21 force-pushed the topic/team_segments branch from 1fc6bd3 to 41d9546 Compare May 7, 2026 19:43
Comment on lines +327 to +332
if (team->mem_map_task != NULL) {
if (team->mem_map_task->status == UCC_INPROGRESS) {
ucc_list_del(&team->mem_map_task->list_elem);
}
ucc_tl_ucp_coll_finalize(team->mem_map_task);
team->mem_map_task = NULL;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 ucc_list_del called without holding the progress queue lock

The MT-locked progress queue (ucc_pq_mt_locked_t) guards its doubly-linked list with a spinlock during enqueue and dequeue. This ucc_list_del bypasses that lock, so there is a window where the progress loop has already dequeued the task (temporarily removing it from the list to call progress()), but status is still UCC_INPROGRESS. If destroy fires in that window — e.g., because ucc_context_progress and ucc_team_destroy are called from different threads — the ucc_list_del operates on a list_elem that no longer belongs to any list, corrupting whichever list its stale prev/next pointers happen to point at.

For single-threaded callers this cannot happen (progress and destroy are serialised), but the code should either document the threading pre-condition or acquire the queue lock before manipulating list_elem.

@wfaderhold21
wfaderhold21 force-pushed the topic/team_segments branch from 1e3135c to 14c531d Compare May 14, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant