[Cuda Ipc] Add barrier at the end of IpcHandleCache::exchangeHandles

[samnordmann]

A synchronization is needed at the IpcHandleCache::exchangeHandles to avoid the exporter exporting twice before the importer delete the key to the store.

Should fix a bug observed in the CI.

cc Team thread

[github-actions]

Review updated until commit 8ce05dd

Description

• Added barrier in IpcHandleCache::exchangeHandles to synchronize ranks

• Re-enabled P2PCommunicationTest.CudaComm test

• Re-enabled MultiDeviceTest.ShareIpcMemHandles test

---

Changes walkthrough 📝

	Relevant files
Bug fix	ipc_handle.cpp Add barrier for synchronization                                                    csrc/multidevice/ipc_handle.cpp Added a barrier at the end of IpcHandleCache::exchangeHandles to ensure synchronization +6/-0
Tests	test_multidevice_communications.cpp Re-enable CudaComm test                                                                    tests/cpp/test_multidevice_communications.cpp Re-enabled P2PCommunicationTest.CudaComm test +1/-1      test_multidevice_host_ir.cpp Re-enable ShareIpcMemHandles test                                                tests/cpp/test_multidevice_host_ir.cpp Re-enabled MultiDeviceTest.ShareIpcMemHandles test +1/-1

---

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

⚡ Recommended focus areas for review Barrier Efficiency The added barrier ensures synchronization but may introduce performance overhead. Consider optimizing the barrier to involve only necessary ranks. // a second barrier is needed here to ensure all ranks have received the // memhandles and the keys are deleted from the store before the next call to // exchangeHandles // TODO: precisely select what ranks need to wait on that barrier. communicator->barrier(); Test Re-enablement The CudaComm test was re-enabled. Ensure that the test is stable and passes consistently before merging. TEST_F(P2PCommunicationTest, CudaComm) { Test Re-enablement The ShareIpcMemHandles test was re-enabled. Ensure that the test is stable and passes consistently before merging. TEST_F(MultiDeviceTest, ShareIpcMemHandles) {

[wujingyue]

Can you also reenable the tests that I disabled?

[samnordmann]

!test

[samnordmann]

!test

[wujingyue]

I disabled two: #4283

(Sorry, I wrote my previous comment on my phone. I should have given you the code pointers)

[samnordmann]

!test