# distributed **URL:** https://dev-discuss.pytorch.org/c/distributed/6.md [Latest](https://dev-discuss.pytorch.org/latest.md) · [Categories](https://dev-discuss.pytorch.org/categories.md) --- ## [About the distributed category](https://dev-discuss.pytorch.org/t/about-the-distributed-category/15)
**Author:** [@smth](https://dev-discuss.pytorch.org/u/smth)\ **Replies:** 0
Topics about torch.distributed --- ## [Design question: supporting one-sided distributed GEMMs in DTensor](https://dev-discuss.pytorch.org/t/design-question-supporting-one-sided-distributed-gemms-in-dtensor/3336)
**Author:** [@BenBrock](https://dev-discuss.pytorch.org/u/BenBrock)\ **Replies:** 1\ **Last updated:** [April 20, 2026, 5:15pm UTC](https://dev-discuss.pytorch.org/t/design-question-supporting-one-sided-distributed-gemms-in-dtensor/3336 "2026-04-20T17:15:09Z")
tl;dr I am interested in adding support for one-sided distributed GEMMs to DTensor. I’m seeking comment about the right way to integrate these into DTensor. There are essentially two issues I see: 1) DTensor would like… --- ## [Fully\_shard with 2D mesh (4,1) still runs all-gather / reduce-scatter on the shard dimension](https://dev-discuss.pytorch.org/t/fully-shard-with-2d-mesh-4-1-still-runs-all-gather-reduce-scatter-on-the-shard-dimension/3310)
**Author:** [@HarryZJ](https://dev-discuss.pytorch.org/u/HarryZJ)\ **Replies:** 0\ **Last updated:** [February 11, 2026, 4:49am UTC](https://dev-discuss.pytorch.org/t/fully-shard-with-2d-mesh-4-1-still-runs-all-gather-reduce-scatter-on-the-shard-dimension/3310 "2026-02-11T04:49:53Z")
Description When passing a 2D mesh (4,1) to fully\_shard, the shard dimension has world size 1, which is semantically equivalent to pure replication . However, the current implementation still takes the HSDP path and perf… --- ## [Distributed Debug Server](https://dev-discuss.pytorch.org/t/distributed-debug-server/3274)
**Author:** [@d4l3k](https://dev-discuss.pytorch.org/u/d4l3k)\ **Replies:** 3\ **Last updated:** [December 2, 2025, 8:53pm UTC](https://dev-discuss.pytorch.org/t/distributed-debug-server/3274 "2025-12-02T20:53:33Z")
Hi folks, We’ve been working on creating an interactive “Debug Server” that consists of an interactive frontend along with HTTP handlers that run on every rank. The key idea is to create a place to store generally usefu… --- ## [PyTorch SymmetricMemory: Harnessing NVLink Programmability with Ease](https://dev-discuss.pytorch.org/t/pytorch-symmetricmemory-harnessing-nvlink-programmability-with-ease/2798)
**Author:** [@yifuwang](https://dev-discuss.pytorch.org/u/yifuwang)\ **Replies:** 6\ **Last updated:** [October 27, 2025, 6:21pm UTC](https://dev-discuss.pytorch.org/t/pytorch-symmetricmemory-harnessing-nvlink-programmability-with-ease/2798 "2025-10-27T18:21:43Z")
PyTorch SymmetricMemory: Harnessing NVLink Programmability with Ease with Horace He, Luca Wehrstedt TL;DR We introduced SymmetricMemory in PyTorch to enable users to harness NVLink programmability with ease. SymmetricMe… --- ## [Correctly using wait() in process group communications like all\_gather](https://dev-discuss.pytorch.org/t/correctly-using-wait-in-process-group-communications-like-all-gather/3252)
**Author:** [@garv901](https://dev-discuss.pytorch.org/u/garv901)\ **Replies:** 3\ **Last updated:** [October 27, 2025, 7:04am UTC](https://dev-discuss.pytorch.org/t/correctly-using-wait-in-process-group-communications-like-all-gather/3252 "2025-10-27T07:04:09Z")
torch::Tensor ParallelOps::GatherFromTensorModelParallelRegion( const torch::Tensor& input /\*\[in\]\*/, const c10::intrusive\_ptr\& process\_group /\*\[in\]\*/) { std::int64\_t world\_size = process\_gro… --- ## [RFC: PyTorch DistributedTensor](https://dev-discuss.pytorch.org/t/rfc-pytorch-distributedtensor/740)
**Author:** [@wanchaol](https://dev-discuss.pytorch.org/u/wanchaol)\ **Replies:** 6\ **Last updated:** [October 1, 2025, 2:22pm UTC](https://dev-discuss.pytorch.org/t/rfc-pytorch-distributedtensor/740 "2025-10-01T14:22:34Z")
RFC: PyTorch DistributedTensor We propose distributed tensor primitives to allow easier distributed computation authoring in SPMD(Single Program Multiple Devices) paradigm. The primitives are simple but powerful when use… --- ## [DTensor - Status, Design and Looking Forward](https://dev-discuss.pytorch.org/t/dtensor-status-design-and-looking-forward/2749)
**Author:** [@wanchaol](https://dev-discuss.pytorch.org/u/wanchaol)\ **Replies:** 3\ **Last updated:** [July 14, 2025, 6:21pm UTC](https://dev-discuss.pytorch.org/t/dtensor-status-design-and-looking-forward/2749 "2025-07-14T18:21:00Z")
This post outlines the current status, design principles, and future work items for PyTorch DTensor. Overall Design Objective and Principles DTensor is the PyTorch native tensor sharding primitive. For the motivations a… --- ## [FSDPv2 communication overlap with compute will slow down compute a lot](https://dev-discuss.pytorch.org/t/fsdpv2-communication-overlap-with-compute-will-slow-down-compute-a-lot/3098)
**Author:** [@AlexW](https://dev-discuss.pytorch.org/u/AlexW)\ **Replies:** 0\ **Last updated:** [July 2, 2025, 7:24am UTC](https://dev-discuss.pytorch.org/t/fsdpv2-communication-overlap-with-compute-will-slow-down-compute-a-lot/3098 "2025-07-02T07:24:55Z")
I’m using FSDPv2 to training a LLM with 8 H200 GPU with NVLinks. When I look at the profile, I found when there is communication-computer overlap. For example gemm overlap with allgather in fsdp, gemm’s mfu will decreas… --- ## [New Contributor Interested in torch.distributed.pipelining](https://dev-discuss.pytorch.org/t/new-contributor-interested-in-torch-distributed-pipelining/3052)
**Author:** [@Juliandlb](https://dev-discuss.pytorch.org/u/Juliandlb)\ **Replies:** 0\ **Last updated:** [June 7, 2025, 1:06pm UTC](https://dev-discuss.pytorch.org/t/new-contributor-interested-in-torch-distributed-pipelining/3052 "2025-06-07T13:06:13Z")
Hi everyone, I’m Julian, an ML engineer with experience in distributed systems, and I’m currently setting up my local development environment to contribute to PyTorch. I’m particularly interested in the torch.distribut… --- ## [FSDP & CUDACachingAllocator: an outsider newb perspective](https://dev-discuss.pytorch.org/t/fsdp-cudacachingallocator-an-outsider-newb-perspective/1486)
**Author:** [@janeyx99](https://dev-discuss.pytorch.org/u/janeyx99)\ **Replies:** 10\ **Last updated:** [December 13, 2024, 4:14pm UTC](https://dev-discuss.pytorch.org/t/fsdp-cudacachingallocator-an-outsider-newb-perspective/1486 "2024-12-13T16:14:04Z")
Hello there. The main motivator of this discussion is: Questionable profile results for FSDP which led to Ke W. + Alban D. + Andrew G. discussing solutions which led to my benefiting from Alban’s takeaways which hopefu… --- ## [Rethinking PyTorch Fully Sharded Data Parallel (FSDP) from First Principles](https://dev-discuss.pytorch.org/t/rethinking-pytorch-fully-sharded-data-parallel-fsdp-from-first-principles/1019)
**Author:** [@awgu](https://dev-discuss.pytorch.org/u/awgu)\ **Replies:** 19\ **Last updated:** [September 17, 2024, 1:25am UTC](https://dev-discuss.pytorch.org/t/rethinking-pytorch-fully-sharded-data-parallel-fsdp-from-first-principles/1019 "2024-09-17T01:25:27Z")
Given some interest, I am sharing a note (first written internally) on the PyTorch Fully Sharded Data Parallel (FSDP) design. This covers much but not all of it (e.g. it excludes autograd and CUDA caching allocator inter… --- ## [Location to add new rendezvous handlers](https://dev-discuss.pytorch.org/t/location-to-add-new-rendezvous-handlers/2365)
**Author:** [@sathyanarays](https://dev-discuss.pytorch.org/u/sathyanarays)\ **Replies:** 1\ **Last updated:** [September 11, 2024, 4:59pm UTC](https://dev-discuss.pytorch.org/t/location-to-add-new-rendezvous-handlers/2365 "2024-09-11T16:59:22Z")
Hi team, We have proposed this issue to support out-of-tree rendezvous backend to the elastic pytorch. We have raised a PR and it is under review. Once this PR is merged, we are planning to contribute additional handle… --- ## [Memcpy based P2P communication for pipeline parallelism instead NCCL](https://dev-discuss.pytorch.org/t/memcpy-based-p2p-communication-for-pipeline-parallelism-instead-nccl/2184)
**Author:** [@MerHS](https://dev-discuss.pytorch.org/u/MerHS)\ **Replies:** 9\ **Last updated:** [September 4, 2024, 11:05am UTC](https://dev-discuss.pytorch.org/t/memcpy-based-p2p-communication-for-pipeline-parallelism-instead-nccl/2184 "2024-09-04T11:05:40Z")
TL;DR: Memcpy-based communication (e.g. tensor.to and tensor.copy) is way better than NCCL P2P APIs for pipeline parallelism, but how can we enable it for multi-node, multi-process training with torchrun? Context I’ve b… --- ## [Enabling Float8 All-Gather in FSDP2](https://dev-discuss.pytorch.org/t/enabling-float8-all-gather-in-fsdp2/2359)
**Author:** [@weifengpy](https://dev-discuss.pytorch.org/u/weifengpy)\ **Replies:** 6\ **Last updated:** [August 26, 2024, 6:31pm UTC](https://dev-discuss.pytorch.org/t/enabling-float8-all-gather-in-fsdp2/2359 "2024-08-26T18:31:23Z")
with Andrew Gu, Wanchao Liang, Driss Guessous, Vasiliy Kuznetsov, Brian Hirsh TL;DR We focus on float8 because it speeds up large GEMMs on H100s and saves network bandwidth with reduced message size. We enabled float8 … --- ## [\[RFC\]\[c10d\] a new Pytorch API (split\_group) to create a process group through ncclCommSplit](https://dev-discuss.pytorch.org/t/rfc-c10d-a-new-pytorch-api-split-group-to-create-a-process-group-through-ncclcommsplit/2233)
**Author:** [@shuqiangzhang](https://dev-discuss.pytorch.org/u/shuqiangzhang)\ **Replies:** 0\ **Last updated:** [July 10, 2024, 4:53pm UTC](https://dev-discuss.pytorch.org/t/rfc-c10d-a-new-pytorch-api-split-group-to-create-a-process-group-through-ncclcommsplit/2233 "2024-07-10T16:53:48Z")
This is a repost of the RFC in Github: \[RFC\]\[c10d\] a new Pytorch API (split\_group) to create a process group through ncclCommSplit · Issue #130407 · pytorch/pytorch · GitHub Motivation In current Pytorch/c10d, the new… --- ## [Relationship between TorchSnapshot and PyTorch's distributed checkpointing](https://dev-discuss.pytorch.org/t/relationship-between-torchsnapshot-and-pytorchs-distributed-checkpointing/751)
**Author:** [@kumpera](https://dev-discuss.pytorch.org/u/kumpera)\ **Replies:** 0\ **Last updated:** [August 31, 2022, 9:39pm UTC](https://dev-discuss.pytorch.org/t/relationship-between-torchsnapshot-and-pytorchs-distributed-checkpointing/751 "2022-08-31T21:39:35Z")
A common source of confusion when users encounter the two efforts is to understand and decide which one to use and how they relate to each other. Introduction Recent efforts in extremely large models faced the issue tha…