# DTensor random RNG state support for non-CUDA backends

**URL:** <https://dev-discuss.pytorch.org/t/dtensor-random-rng-state-support-for-non-cuda-backends/2536>\
**Category:** hardware-backends\
**Created:** [October 16, 2024, 3:36pm UTC](https://dev-discuss.pytorch.org/t/dtensor-random-rng-state-support-for-non-cuda-backends/2536 "2024-10-16T15:36:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sujoy\_Saraswati](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/sujoy_saraswati/32/783_2.png) [@Sujoy\_Saraswati](https://dev-discuss.pytorch.org/u/Sujoy_Saraswati)\
**Post date:** [October 16, 2024, 3:36pm UTC](https://dev-discuss.pytorch.org/t/dtensor-random-rng-state-support-for-non-cuda-backends/2536/1 "2024-10-16T15:36:41Z")

</div>

DTensor random ops mesh support for a backend is checked with is\_rng\_supported\_mesh, which checks the presence of hasattr(device\_handle, “set\_rng\_state”).

For CPU RNG state, this seems to return False. However, If a backend uses CPU RNG state and has a set\_rng\_state() implemented, it sets is\_rng\_supported\_mesh to be True and DTensor random mechanism tries to use OffsetBasedRNGTracker. The seed and offset based APIs assume the RNG state to be a CUDA like offset based. CPU RNG state doesn’t work and fails as shown below -

```auto
[rank0]: File ".../.local/lib/python3.10/site-packages/torch/distributed/_tensor/random.py", line 176, in _distribute_region
[rank0]: old_offset = self.get_offset("parallel-rng")
[rank0]: File ".../.local/lib/python3.10/site-packages/torch/distributed/_tensor/random.py", line 195, in get_offset
[rank0]: return int(offset_tensor.item())
[rank0]: RuntimeError: a Tensor with 631 elements cannot be converted to Scalar

```

Is there a plan to enhance the DTensor to use CPU like RNG state, or is it possible to mark is\_rng\_supported\_mesh to be false for backends that use CPU RNG state and not the offset based RNG state?

---

<div class="post-metadata">

**Author:** ![albanD](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/alband/32/9_2.png) [@albanD](https://dev-discuss.pytorch.org/u/albanD)\
**Post date:** [October 17, 2024, 1:55pm UTC](https://dev-discuss.pytorch.org/t/dtensor-random-rng-state-support-for-non-cuda-backends/2536/2 "2024-10-17T13:55:43Z")

</div>

Hey!

cc @wanchaol in case he’s seeing this.  
This kind of request might be best as an issue on github though as this is where we track features and improvements usually!

---

<div class="post-metadata">

**Author:** ![Sujoy\_Saraswati](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/sujoy_saraswati/32/783_2.png) [@Sujoy\_Saraswati](https://dev-discuss.pytorch.org/u/Sujoy_Saraswati)\
**Post date:** [October 18, 2024, 2:11pm UTC](https://dev-discuss.pytorch.org/t/dtensor-random-rng-state-support-for-non-cuda-backends/2536/3 "2024-10-18T14:11:15Z")

</div>

> [@Sujoy\_Saraswati](#):
>
> ```auto
> [rank0]: File ".../.local/lib/python3.10/site-packages/torch/distributed/_tensor/random.py", line 176, in _distribute_region
> [rank0]: old_offset = self.get_offset("parallel-rng")
> [rank0]: File ".../.local/lib/python3.10/site-packages/torch/distributed/_tensor/random.py", line 195, in get_offset
> [rank0]: return int(offset_tensor.item())
> [rank0]: RuntimeError: a Tensor with 631 elements cannot be converted to Scalar
> 
> ```

Thanks @albanD . I have submitted [DTensor RNG state for non CUDA backends · Issue #138329 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/138329) for this.