Repository navigation
Conversation
2683ab3 to
a77c345
Compare
| Like data exchange semantics, the entry and completion | ||
| criteria of blocking and nonblocking alltoall is similar. | ||
|
|
||
| {\bf Entry criteria}: Before any \ac{PE} calls a \FUNC{shmem\_alltoall\_nb} routine, |
There was a problem hiding this comment.
This entry criteria is confusing - if we have the same entry criteria as blocking operations - then the users are supposed to make sure the src/dst buffers are available at the point of entry - meaning the users are expected to sync with themselves before launching the nb collective operation.
AFAIU - the src/dst needs to be available when everyone in the team reaches the state - which can be determined during runtime - without the need for explicit sync before launching the nb collective operation.
Please clarify the intended semantics.
There was a problem hiding this comment.
Naveen, my intent is to keep it consistent with the semantics of blocking collectives. I wanted to improve the description. I don’t see how the current wording is violating that. If the wording doesn’t reflect that, then we can fix it.
|
Feedback from Jim (May Spec Meeting): Should we consider shmem_local_complete ? |
| \item Collective Types: The nonblocking variants supported include barrier, alltoall, | ||
| and broadcast collectives. Other collective operations such as | ||
| reductions, collect, barrier, alltoalls, and sync will not have nonblocking variants. | ||
|
|
There was a problem hiding this comment.
Should sync_all be supported? My hunch is it seems possibly useful...
|
Feedback from Feb. 27, 2024 discussion:
|
| threads but on different request objects. It is the responsibility of the | ||
| \openshmem user to ensure that proper synchronization is used to prevent race | ||
| conditions or deadlock. Specifically, the \FUNC{shmem\_req\_wait} operation should | ||
| be called after the collective operation to ensure that the request object is |
There was a problem hiding this comment.
If the requests are always deallocated in wait or test call, and if request handles are not reused, it will not run into the scenario of "premature deallocation".
| \apidescription{ | ||
| A call to \FUNC{shmem\_req\_test} returns immediately. If the | ||
| operation identified by the request is completed, it returns | ||
| zero, and the request object is deallocated and cannot be reused. |
There was a problem hiding this comment.
Could request objects have value SHMEM_REQUEST_INVALID after they are deallocated? In which case this routine and shmem_req_wait would take an argument of type shmem_req_h *.
There was a problem hiding this comment.
@davidozog Just updated. Please let me know if it's missing something.
|
Notes from today's discussion -- 06-27-2024 -- NBI Collectives pSync Management.pptx |
ea99bf7 to
40891aa
Compare
|
Proof of concept implementation: https://github.com/wfaderhold21/ompi/tree/nb_scoll NOTE: oshmem does not have full teams implementation. so the team passed in is assumed to be SHMEM_TEAM_WORLD. |
| source = (int64_t **)shmem_malloc(nr_a2a * sizeof(int64_t *)); | ||
| dest = (int64_t **)shmem_malloc(nr_a2a * sizeof(int64_t *)); | ||
| requests = (shmem_req_h *)malloc(nr_a2a * sizeof(shmem_req_h)); | ||
| for (int i = 0; i < npes; i++) { |
There was a problem hiding this comment.
@wfaderhold21 This looks like a bug in the example that was not resolved. Should we incorporate the above change as a doc edit?
|
@naveen-rn please post the text document here so that I may respond to it accordingly. Thanks again for generating the use-cases and sharing your concerns. |
|
@wfaderhold21 Attached is the document discussing potential use cases for non-blocking collective operations. The objective of this attachment is not to highlight specific issues but to outline a preliminary program flow illustrating the usability of the proposed interface and to seek clarification on certain aspects of the proposal. The primary focus is on the cost implications of employing Two potential costs are associated with using sync:
The first cost is relative; when the payload size in the collective operation is large, the overhead of synchronization may be negligible. However, the second cost warrants further clarification, as it appears to significantly affect the look-ahead use case described in the document. Employing a sync-based model to address the look-ahead scenario could impede the insertion of non-blocking collective operations into the execution pipeline. In this context, a non-blocking collective synchronization mechanism or a form of fence to enforce operation ordering could be advantageous. Your insights would be valuable, given your implementation and experimentation with the proposed interface. |
|
@naveen-rn thank you for the feedback / document discussing the potential use cases for non-blocking collective operations and my apologies for responding so late. I'm curious regarding the motivation for the question regarding the usage of sync, however. With either nonblocking collectives or blocking collectives, you would need a sync as the specification is written today, correct? Additionally, you had mentioned in the specification meeting that MPI nonblocking collectives have global completion guarantees, but I'm unable to find any suggestion of this in the MPI specification. Can you please guide me in your understanding as I believe that may be helpful here. Additionally, given that a sync would be necessary in any situation currently available with OpenSHMEM, would the understanding of the overheads of sync add significant value here? |
|
ping @naveen-rn |
|
special ballot changes: 6e5a710..f253c6f |
|
Slides from today's WG meeting |
|
@manjugv @wfaderhold21 Please review and resolve all of the open comments on this PR. Let me know if any require a doc. edit change. |
|
Please add a changelog entry. |
No description provided.