Does the GinAlltoAllKernel have send/receive conflicts in in-place mode?

[shenhongqian]

Consider an in-place all-to-all operation where send and receive buffers are the same memory region, and there is no intermediate buffering. Is there a risk that a process receives data into a buffer location before it has finished sending the original data from that same location, thereby corrupting the outgoing message?

[shanedsnyder]

Is there a risk that a process receives data into a buffer location before it has finished sending the original data from that same location, thereby corrupting the outgoing message?

Yes. I believe this behavior is not necessarily unique to the GIN kernel, though -- there's not really a guarantee in any case that a rank's send buffer hasn't already been written into by remote ranks when doing in-place alltoall.

In fact, we disable data validation checking for in-place alltoall for this reason in all cases (GIN or not):

nccl-tests/src/alltoall.cu

Line 42 in da0b547

args->reportErrors = in_place ? 0 : 1;