# How to find the c++/cuda implementation of specific operators in pytorch source code **URL:** \ **Category:** Uncategorized\ **Created:** [October 9, 2023, 9:02am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551 "2023-10-09T09:02:32Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![shuokay](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/shuokay/32/594_2.png) [@shuokay](https://dev-discuss.pytorch.org/u/shuokay)\ **Post date:** [October 9, 2023, 9:02am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/1 "2023-10-09T09:02:32Z")
Recently, I’ve been working on resolving some numerical precision alignment issues. Take the division operator as an example; the computation yields different results on CPU and CUDA or when expressed using different syntax, as seen in the attached screenshot. I’m endeavoring to uncover the underlying reasons through various methods, and the first thing that comes to mind is to review the C++ source code or CUDA source code. However, I’ve encountered challenges in understanding the intricacies of PyTorch’s C++ code and locating the corresponding source code. Is there anyone who can help me understand how to learn PyTorch’s C++ source code, particularly how to find the implementation details of C++ operators? ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/6b537bea2c64a613c487c1d95b592489fa2835e1.png) ---
**Author:** ![shuokay](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/shuokay/32/594_2.png) [@shuokay](https://dev-discuss.pytorch.org/u/shuokay)\ **Post date:** [October 9, 2023, 9:10am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/2 "2023-10-09T09:10:22Z")
In most cases, I can find the code like these: ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/f94c7e4d2bbc2154a2c9f54884de4a697e813f88.png) but I failed to find the source code of `op.call` function. ---
**Author:** ![bdhirsh](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/bdhirsh/32/575_2.png) [@bdhirsh](https://dev-discuss.pytorch.org/u/bdhirsh)\ **Post date:** [October 17, 2023, 12:57am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/3 "2023-10-17T00:57:53Z")
@shuokay the main reason finding the kernel you’re looking for from there is a pain is because `op.call(self, other)` uses the pytorch dispatcher, and dynamally dispatches to the right kernel (more on the dispatcher here: [Let’s talk about the PyTorch dispatcher : ezyang’s blog](http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/)) The source of truth (well, 99% of the time) for every ATen operator and the names of its CPU/CUDA kernels in the codebase is in native\_functions.yaml: [https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native\_functions.yaml](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml) For `div.Tensor`: I happen to know that’s it’s implemented as a structured kernel (more on those here [Codegen and Structured Kernels · pytorch/pytorch Wiki · GitHub](https://github.com/pytorch/pytorch/wiki/Codegen-and-Structured-Kernels)). I can tell because of the `structured_delegate` key in native\_functions.yaml here: [https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native\_functions.yaml#L2058C3-L2058C22](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml#L2058C3-L2058C22) As a structured kernel, the op `aten::div.Tensor` has a “meta” function that performs shape error checks and computes the output shape defined here: [https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/BinaryOps.cpp#L174](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/BinaryOps.cpp#L174) And it has an implementation that uses TensorIterator, defined here: [https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/BinaryOps.cpp#L448](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/BinaryOps.cpp#L448) (from there you should be able to grep for each function name to find it). ---
**Author:** ![shuokay](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/shuokay/32/594_2.png) [@shuokay](https://dev-discuss.pytorch.org/u/shuokay)\ **Post date:** [October 31, 2023, 9:15am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/4 "2023-10-31T09:15:33Z")
@bdhirsh Thank you very much for your reply, it was really helpful. Based on your explanation, I have “debugged” the PyTorch code step by step, and I think I have a deeper understanding of the implementation and dispatch of PyTorch operators now. ---
**Author:** ![hungpham3112](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/hungpham3112/32/1700_2.png) [@hungpham3112](https://dev-discuss.pytorch.org/u/hungpham3112)\ **Post date:** [January 5, 2024, 9:56am UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/5 "2024-01-05T09:56:37Z")
What about matmul function? What is correct way to find matmul for CPU and matmul CUDA version? ---
**Author:** ![Lezcano](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/lezcano/32/300_2.png) [@Lezcano](https://dev-discuss.pytorch.org/u/Lezcano)\ **Post date:** [January 5, 2024, 2:30pm UTC](https://dev-discuss.pytorch.org/t/how-to-find-the-c-cuda-implementation-of-specific-operators-in-pytorch-source-code/1551/6 "2024-01-05T14:30:46Z")
native\_functions.yaml + grepping for the name there should take you to the function. As Brian mentioned, this works for any function implemented in C++/CUDA