# How to capture NCCL communication ops in FakeTensorMode?

**URL:** <https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410>\
**Category:** compiler\
**Created:** [August 1, 2023, 2:06pm UTC](https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410 "2023-08-01T14:06:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![weberxie](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/weberxie/32/1267_2.png) [@weberxie](https://dev-discuss.pytorch.org/u/weberxie)\
**Post date:** [August 1, 2023, 2:06pm UTC](https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410/1 "2023-08-01T14:06:37Z")

</div>

Hi team, I’m tracing 2D parallel graph with FakeTensor and FakeTensorMode, but it throws the below error

```auto
Exception has occurred: UnsupportedOperatorException
c10d.allreduce_.default
  File "/usr/local/conda/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py", line 1404, in dispatch
    r = func(*args, **kwargs)
  File "/usr/local/conda/lib/python3.9/site-packages/torch/_ops.py", line 437, in __call__
    return self._op(*args, **kwargs or {})
NotImplementedError: Could not run 'c10d::allreduce_' with arguments from the 'Meta' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'c10d::allreduce_' is only available for these backends: [CPU, CUDA, PrivateUse1, SparseCPU, SparseCUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMeta, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

```

So, are there any solutions that can escape this error? Thanks!

---

<div class="post-metadata">

**Author:** ![weberxie](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/weberxie/32/1267_2.png) [@weberxie](https://dev-discuss.pytorch.org/u/weberxie)\
**Post date:** [August 1, 2023, 2:08pm UTC](https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410/2 "2023-08-01T14:08:58Z")

</div>

and the stack is:

```auto
During handling of the above exception, another exception occurred:

  File "/usr/local/conda/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py", line 1408, in dispatch
    raise UnsupportedOperatorException(func)

```

The PyTorch version is 2.1.0.

---

<div class="post-metadata">

**Author:** ![kumpera](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/kumpera/32/645_2.png) [@kumpera](https://dev-discuss.pytorch.org/u/kumpera)\
**Post date:** [August 2, 2023, 5:35pm UTC](https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410/3 "2023-08-02T17:35:43Z")

</div>

You can provide a meta device implementation of that op yourself from python, that will get you pass this step.

Be aware that c10d ops cannot be used with dynamo as they use dynamically created torchbinding objects that will be baked as constants on your trace.

---

<div class="post-metadata">

**Author:** ![Richie](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@Richie](https://dev-discuss.pytorch.org/u/Richie)\
**Post date:** [August 3, 2023, 12:07pm UTC](https://dev-discuss.pytorch.org/t/how-to-capture-nccl-communication-ops-in-faketensormode/1410/4 "2023-08-03T12:07:15Z")

</div>

Hi , Here is the Meta version of allreduce\_ that I implemented, but there is a problem with the return value Work. Is there any way to solve it?

 ![截屏2023-08-03 下午8.05.36](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/7/7cd9035daeca4d58efce99daf43bb47eac0d2f92.png)  
 ![截屏2023-08-03 下午8.05.25](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/0384e7fde2c2c682cede5b474401d6eb0b9d4b78.png)