# Overlapping device to host copy with GPU collectives **URL:** \ **Category:** NVIDIA CUDA\ **Created:** [May 24, 2024, 7:28pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120 "2024-05-24T19:28:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![YangFei1990](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/yangfei1990/32/961_2.png) [@YangFei1990](https://dev-discuss.pytorch.org/u/YangFei1990)\ **Post date:** [May 24, 2024, 7:28pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/1 "2024-05-24T19:28:33Z")
Hello, I’m running tests on AWS P5 instances and I’m trying to asynchronously offload tensors from device to host while doing GPU collectives. My impression is that DtoH is using the PCIe and GPU CC is using RDMA through NVlink, so they should be irrelevant. However what I observed is that the CC ops can be highly impacted by the offloading, even running it synchronously. I create a simple script to run my tests with `torchrun -nproc_per_node 8` so it is a single node test with 8 GPUs. ```auto import torch import torch.distributed as dist NO_OFFLOAD = True SYNC_OFFLOAD = False dist.init_process_group("nccl") torch.cuda.set_device(dist.get_rank()) tensor = torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16) dist.all_reduce(tensor) # warm up nccl tensor_list = [torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16) for _ in range(8)] cpu_tensor = torch.empty(tensor.shape, dtype=tensor.dtype, device=torch.device("cpu"), pin_memory=True) d2h_stream = torch.cuda.Stream(device=dist.get_rank(), priority=-1) torch.cuda.synchronize() if not NO_OFFLOAD: with torch.cuda.stream(d2h_stream): with torch.no_grad(): for i in range(5): cpu_tensor.copy_(tensor, non_blocking=True) if SYNC_OFFLOAD: torch.cuda.synchronize() for i in range(32): dist.all_gather(tensor_list, tensor) ``` Firstly with `NO_OFFLOAD = True` I see below profile ![no_offload](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/068f290704f4383272cfb2a27a9af54be0555ff6.png) where each allgather takes around 2.7ms. Now I turn on offload ![offload](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/84bea153d766c8f9f721d0ffb6379f89018c21ef.png) the first several allgathers becomes very slow, up to 13ms. Even the offloading is finished, it still take ~4.4ms for the rest of allgathers. Next I enable sync offload so that allgathers will happen after the offloading. ![sync offload](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/5/5af3e8dec965ad9be509daf33d0a040c3bf17a8c.png) However the first allgather becomes extremely slow, ~110ms. After that the rest allgathers become 2.7ms again. Could someone help me understand this behavior? Help is greatly appreciated! I’m on torch 2.2.0 with `NCCL version 2.19.4+cuda12.1`. ---
**Author:** ![kwen2501](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/kwen2501/32/1989_2.png) [@kwen2501](https://dev-discuss.pytorch.org/u/kwen2501)\ **Post date:** [May 30, 2024, 5:39pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/2 "2024-05-30T17:39:32Z")
Hi, for the 3rd profile (SYNC\_OFFLOAD), could the long first all\_gather be due to straggler effect? Can you mind adding a barrier here and see if the profile changes? Thanks! ```auto if SYNC_OFFLOAD: torch.cuda.synchronize() dist.barrier() ``` ---
**Author:** ![YangFei1990](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/yangfei1990/32/961_2.png) [@YangFei1990](https://dev-discuss.pytorch.org/u/YangFei1990)\ **Post date:** [May 31, 2024, 5:49pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/3 "2024-05-31T17:49:18Z")
Thank @kwen2501! Actually I found that weird behavior for the 3rd profile (SYNC\_OFFLOAD) is because different process have different offloading time. After applying the barrier that behavior is gone. However I still observe slowness in NCCL when overlapping with DtoH copies. ---
**Author:** ![kwen2501](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/kwen2501/32/1989_2.png) [@kwen2501](https://dev-discuss.pytorch.org/u/kwen2501)\ **Post date:** [June 3, 2024, 4:32pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/4 "2024-06-03T16:32:08Z")
Great, thanks! That matches our expectation! Now back to Fig. 1 vs Fig. 2, I wonder why Fig. 2 has one more line of all-gather kernels? I am referring to the kernels in the dotted box. Thanks! ![84bea153d766c8f9f721d0ffb6379f89018c21ef](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/03ccb5f7d6c9979dac40a7a5bae86b5a148b1b7b.png) ---
**Author:** ![YangFei1990](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/yangfei1990/32/961_2.png) [@YangFei1990](https://dev-discuss.pytorch.org/u/YangFei1990)\ **Post date:** [June 3, 2024, 5:54pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/5 "2024-06-03T17:54:27Z")
@kwen2501 what do you mean more line? The total number of the all-gathers are the same (32). If you are referring to the number of rows I think it is just about how I expand the Nsys App for the stream details. ---
**Author:** ![kwen2501](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/kwen2501/32/1989_2.png) [@kwen2501](https://dev-discuss.pytorch.org/u/kwen2501)\ **Post date:** [June 4, 2024, 4:30pm UTC](https://dev-discuss.pytorch.org/t/overlapping-device-to-host-copy-with-gpu-collectives/2120/6 "2024-06-04T16:30:48Z")
Thanks! I reported this to NCCL team. If you’d like, you can also open an issue on NCCL’s GitHub: [GitHub - NVIDIA/nccl: Optimized primitives for collective multi-GPU communication](https://github.com/NVIDIA/nccl), for easier tracking.