How to get a fast debug build

1

Following Allow to specify specific files for debug info by albanD · Pull Request #111748 · pytorch/pytorch · GitHub being merged, there is a new compilation flag that can be used to specify debug informations ONLY for a given subset of the source files via:

# Your usual build
BUILD_CONFIG python setup.py develop

# Enable debug mode for two files
BUILD_CONFIG python setup.py clean
USE_CUSTOM_DEBINFO=torch/csrc/Module.cpp;aten/src/ATen/native/cpu/SortingKernel.cpp BUILD_CONFIG python setup.py develop

If you use ccache (as you should), this will only recompile these two files and relink the binary. You will now have full debug info for these files!

This has two main benefits: going from your usual build to enabling debug infos only recompiles the files you care about and the final binary only has bebug info for these two files. This means it won’t be gigabytes big and will not slow gdb to a crawl!

Side note, @malfet has another trick to do this if you don’t like the clean:

touch path/to/file.cpp; ninja -n >build.sh; sed '-O3/-g' build.sh; sh -f build.sh
3 Likes
2

Here is the correct command (replace sed with gsed on MacOS):

ninja -j1 -v -n torch_cpu |sed -e 's/-O[23]/-g/g' -e 's#\[[0-9]\+\/[0-9]\+\] \+##' |sh
3

The python script, BUILD_WITHDEBINFO aren’t working well for me.

Let’s say I want to get debugging info for aten/src/ATen/native/cuda/group_norm_kernel.cu

I’ve tried:

./tools/build_with_debinfo.py aten/src/ATen/native/cuda/group_norm_kernel.cu
USE_CUSTOM_DEBINFO=aten/src/ATen/native/cuda/group_norm_kernel.cu; python setup.py develop`

And neither will let simply break into cuda-gdb with break group_normal_kernel.cu:570 for example. I can’t just build the entire thing with debugging symbols either because then cuda-gdb becomes miserably slow.

DEBUG=1 DEBUG_CUDA=1 python setup.py develop
I can do something like that and it’ll seemingly work. What am i doing wrong? Also when do we want to use install vs develop? If you don’t use install then it wouldn’t work no? I generally just use cuda-gdb into a python script that ends up calling the code.

4

Hey!

Ho that is weird but I have never tried to use this on cuda files to be honest.
If you’re using regular gdb, can it find the debug info for the c++ code in there?

5

Wdym? Like use info functions ? Curious to know why you mention specifically gdb and not cuda-gdb though.
Even with a full debug build it’s kind of painful stepping through the kernel code atm. Working on my 3rd PR so I’m still very new to the internals, I use the debugger to better visualize the flow.

Tangential - is there a slack or discord that is available? Sometimes I have questions that could be easily answered in a few minutes but take a few hours to figure out on my own since I’m new to pytorch internals so it would be helpful to have an avenue for that. Are the forums the only place?