# Which papers are relevant to understand how the core of PyTorch and the GPU kernels and torch.compile are put together? **URL:** \ **Category:** Uncategorized\ **Created:** [May 10, 2025, 12:01pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983 "2025-05-10T12:01:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Pawl](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Pawl](https://dev-discuss.pytorch.org/u/Pawl)\ **Post date:** [May 10, 2025, 12:01pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/1 "2025-05-10T12:01:21Z")
Hi, I’m a newbie who is embarking on this project of studying PyTorch holistically. At this point I am mostly interested in the core of PyTorch, the CUDA kernels and the torch.compile project. I am already somewhat familiar with the CUDA kernels as my job involves some hardware aware optimizations. As a first step I want to take a step back and read some relevant papers to understand how PyTorch is put together, before doing a deep dive in the code. These are the papers I have found on the topic: - PyTorch: An Imperative Style, High-Performance Deep Learning Library - PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation - Automatic differentiation in PyTorch For completeness sake I wanted to ask whether there are other papers or documentation that would be of relevance to read. In particular I cannot find any papers that detail the system design decisions of aten-CUDA. And are there any papers on torch.inductor and torch.dynamo and everything related to torch.compile? EDIT I have since read the first two papers. I see that the second paper covers torch.inductor and torch.dynamo. ---
**Author:** ![mooglevich](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mooglevich](https://dev-discuss.pytorch.org/u/mooglevich)\ **Post date:** [May 12, 2025, 7:45pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/2 "2025-05-12T19:45:57Z")
I’m not sure if this is exactly what you want, but this is a well-known blog (by one of the PyTorch devs) that’s quite good [PyTorch internals : ezyang’s blog](https://blog.ezyang.com/2019/05/pytorch-internals/). He also hosts a great podcast where he explains design decisions and history on various PyTorch libraries etc. [https://pytorch-dev-podcast.simplecast.com/episodes](https://pytorch-dev-podcast.simplecast.com/episodes) It may not be specifically torch.compile or how PyTorch goes down to CUDA kernels, but maybe it’s helpful. I haven’t read the papers you’ve listed, but as a PyTorch beginner myself, it’s most welcome to see you share those. I will read them myself now 🙂 Update: Oh, and here’s another amazing resource with some PyTorch folks involved [https://www.youtube.com/channel/UCJgIbYl6C5no72a0NUAPcTA](https://www.youtube.com/channel/UCJgIbYl6C5no72a0NUAPcTA). Again, it may not 100% be what you’re specifically asking for here, but it’s a great resource for learning. ---
**Author:** ![Pawl](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Pawl](https://dev-discuss.pytorch.org/u/Pawl)\ **Post date:** [May 13, 2025, 7:46am UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/3 "2025-05-13T07:46:53Z")
Thanks, I found the blog quite helpful. I also found the Wiki on the project page on github contains a lot of information. It is unfortunately all a bit all over the place. I was hoping there would be a more organized documentation of PyTorch internals, but I suspect the way the project evolved that was never possible. ---
**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\ **Post date:** [May 13, 2025, 5:48pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/4 "2025-05-13T17:48:16Z")
There is also [https://docs.pytorch.org/assets/pytorch2-2.pdf](https://docs.pytorch.org/assets/pytorch2-2.pdf) ---
**Author:** ![Pawl](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Pawl](https://dev-discuss.pytorch.org/u/Pawl)\ **Post date:** [May 13, 2025, 8:36pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/5 "2025-05-13T20:36:09Z")
Perfect, thanks, I had have already started studying it. ---
**Author:** ![mozuysal](https://avatars.discourse-cdn.com/v4/letter/m/6f9a4e/32.png) [@mozuysal](https://dev-discuss.pytorch.org/u/mozuysal)\ **Post date:** [May 16, 2025, 7:24am UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/6 "2025-05-16T07:24:07Z")
I came across these that might be relevant: - PyTorch2 Presentation slides @ASPLOS’24 on compiler components: [https://github.com/pytorch/workshops/blob/master/ASPLOS\_2024/README.md](https://github.com/pytorch/workshops/blob/master/ASPLOS_2024/README.md) - PyTorch OnBoarding Docs: [https://github.com/pytorch/pytorch/wiki/Core-Frontend-Onboarding/9989d142ea694d47a3fbeaa728a448c74706e51f](https://github.com/pytorch/pytorch/wiki/Core-Frontend-Onboarding/9989d142ea694d47a3fbeaa728a448c74706e51f) The onboarding docs also have some labs, but the cuda ones seems to be internal-only. Hope this helps. ---
**Author:** ![Pawl](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Pawl](https://dev-discuss.pytorch.org/u/Pawl)\ **Post date:** [May 16, 2025, 7:06pm UTC](https://dev-discuss.pytorch.org/t/which-papers-are-relevant-to-understand-how-the-core-of-pytorch-and-the-gpu-kernels-and-torch-compile-are-put-together/2983/7 "2025-05-16T19:06:05Z")
Thanks I didn’t have those yet. Yeah, I think I can get quite far just reading the source and then I’ll just ask on the forum when I need advice.