TL/UCP: add implementation for team-based segments - #1301
Conversation
|
| Filename | Overview |
|---|---|
| src/components/tl/ucp/tl_ucp_team_mem.c | New file: 3-phase team-scoped memory registration with thorough error cleanup; in-flight task cancellation documented as single-threaded only. |
| src/components/tl/ucp/tl_ucp_team_mem.h | New header declaring the four team-mem lifecycle functions; includes tl_ucp.h, resolving the duplicate-declaration issue from prior review threads. |
| src/components/tl/ucp/tl_ucp.h | Added 10 new fields to ucc_tl_ucp_team_t, three accessor macros, and moved allgather forward declaration here as canonical site. |
| src/components/tl/ucp/tl_ucp_sendrecv.h | Step 2 added in resolve_p2p_by_va to search team segments; correctly uses pre-remap team_peer and caller-initialized ep. |
| src/components/tl/ucp/tl_ucp_team.c | mem_map_phase state machine added to team_create_test; avoids direct task->progress() calls; destroy called unconditionally in team_destroy. |
| test/gtest/common/test_ucc.cc | UccTeam extended with team-owned onesided buffers; seg_maps lifetime sufficient today but leaves a technically-dangling pointer in stored team params. |
| test/gtest/common/test_ucc.h | Added is_team_onesided flag, per-proc onesided_buf array, and onesided_buf() accessor; proc::onesided_buf hardcodes 3 instead of UCC_TEST_N_MEM_SEGMENTS. |
| test/gtest/coll/test_alltoall.cc | New single_team_onesided test exercises team-scoped segment path via alltoall ring algorithm. |
| src/components/tl/ucp/tl_ucp.c | Removed redundant forward declarations of service_allgather and service_test; canonical declarations moved to tl_ucp.h. |
Reviews (3): Last reviewed commit: "TEST: add team-based onesided tests" | Re-trigger Greptile
| ucs_status = ucp_ep_rkey_unpack( | ||
| ep, elem + 24, &UCC_TL_UCP_TEAM_RKEY(team, r, s)); | ||
| if (UCS_OK != ucs_status) { | ||
| tl_error(UCC_TL_TEAM_LIB(team), | ||
| "ucp_ep_rkey_unpack failed for rank %u seg %lu: %s", | ||
| r, s, ucs_status_string(ucs_status)); | ||
| return ucs_status_to_ucc_status(ucs_status); |
There was a problem hiding this comment.
Early return leaves
rbuf and task un-cleaned
When ucp_ep_rkey_unpack fails for any rank, the function returns immediately without freeing team->mem_map_allgather_rbuf or finalizing team->mem_map_task. The cleanup relies entirely on ucc_tl_ucp_team_mem_map_destroy being called during teardown — this is safe today because team creation failure always leads to team_destroy, but the asymmetry makes the invariant fragile. Additionally, rkeys already unpacked for ranks 0..r-1 are left in team_rkeys[]; those are properly destroyed by the destroy function (NULL-checked loop), so no handle leak occurs — but it is worth making this contract explicit.
| ucc_status_t ucc_tl_ucp_service_test(ucc_coll_task_t *task) | ||
| { | ||
| if (task->status == UCC_INPROGRESS) { | ||
| task->progress(task); | ||
| } | ||
| return task->status; | ||
| } |
There was a problem hiding this comment.
ucc_tl_ucp_service_test directly calls task->progress() for in-queue tasks
tl_ucp_team.c has an explicit warning (lines 270-273) that calling task->progress(task) directly on a task still in the context progress queue risks completing it while enqueued, corrupting the list on the next ucc_context_progress() call. This new helper does exactly that. While it is not called anywhere in this PR, having it available creates a footgun for future callers who use it with tasks posted by ucc_tl_ucp_service_allgather.
| ucc_status_t ucc_tl_ucp_service_allgather(ucc_base_team_t *team, void *sbuf, | ||
| void *rbuf, size_t msgsize, | ||
| ucc_subset_t subset, | ||
| ucc_coll_task_t **task_p); |
There was a problem hiding this comment.
Duplicate forward declaration of
ucc_tl_ucp_service_allgather
ucc_tl_ucp_service_allgather is already forward-declared in tl_ucp.c (line 353). Declaring it again here creates two independent declaration sites that must stay in sync. Consider including the existing TL header that exposes this declaration, or removing the duplicate.
| if (n_segs == 0 || size == 0) { | ||
| return UCC_OK; | ||
| } |
There was a problem hiding this comment.
Dead defensive guard that conflicts with the caller's error check
The if (n_segs == 0 || size == 0) early return is unreachable: the call site in tl_ucp_team.c only enters the mem-map path when mem_params.n_segments > 0, and size == 0 cannot occur for a valid team. More importantly, the caller checks if (status != UCC_INPROGRESS) { goto err_preconnect; } — if this branch were ever taken it would goto err_preconnect with status == UCC_OK, silently marking team creation as complete without finishing the data exchange. Consider removing the guard, or if kept, handle UCC_OK explicitly in the caller.
1fc6bd3 to
41d9546
Compare
| if (team->mem_map_task != NULL) { | ||
| if (team->mem_map_task->status == UCC_INPROGRESS) { | ||
| ucc_list_del(&team->mem_map_task->list_elem); | ||
| } | ||
| ucc_tl_ucp_coll_finalize(team->mem_map_task); | ||
| team->mem_map_task = NULL; |
There was a problem hiding this comment.
ucc_list_del called without holding the progress queue lock
The MT-locked progress queue (ucc_pq_mt_locked_t) guards its doubly-linked list with a spinlock during enqueue and dequeue. This ucc_list_del bypasses that lock, so there is a window where the progress loop has already dequeued the task (temporarily removing it from the list to call progress()), but status is still UCC_INPROGRESS. If destroy fires in that window — e.g., because ucc_context_progress and ucc_team_destroy are called from different threads — the ucc_list_del operates on a list_elem that no longer belongs to any list, corrupting whichever list its stale prev/next pointers happen to point at.
For single-threaded callers this cannot happen (progress and destroy are serialised), but the code should either document the threading pre-condition or acquire the queue lock before manipulating list_elem.
1e3135c to
14c531d
Compare
14c531d to
c8b83ac
Compare
What
Implements a team-based memory map param support for TL/UCP.
Why ?
This adds an expensive path over team creation for TL/UCP if the user sets the memory map parameters during team creation. It enables segments that are restricted to a team rather than a context and would be useful for (1) OpenSHMEM Team's support and (2) using global-work buffer for features such as PR #1248 and PR #1149 to enable onesided scratch buffers.