# Debugging back-propogation results

**URL:** <https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396>\
**Category:** hardware-backends\
**Created:** [November 11, 2021, 8:32pm UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396 "2021-11-11T20:32:36Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![artyom-beilis](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/artyom-beilis/32/207_2.png) [@artyom-beilis](https://dev-discuss.pytorch.org/u/artyom-beilis)\
**Post date:** [November 11, 2021, 8:32pm UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396/1 "2021-11-11T20:32:36Z")

</div>

I’m working on OpenCL backend for pytorch. I currently validate standard torchvision models, in forward and back-propogation. All nets but `efficientnet_bX` I tested work. `efficientnet_bX` give wrong results in backward computations.

Now when I had similar issues with forward propogation I copied then net and saved “checkpoints” in different places (after suspicious operations) till I found one that generated results that aren’t similar to `cpu`.

I can look on gradients generated in parameters of layers like conv of linear but I can’t do it for layers that don’t have parameters.

How can I do this for backpropogation, since it is all automatic? In caffe for example with its static graph I could run `net.backward(from,to)` and look into intermediate layers.

Is there anything that can help me extracting each and every value for CPU and my device?

* * *

Below example of runtest output where maximal difference and maximal output difference is comouted

```auto
Testing mnist_mlp
Accessing device #1:GeForce RTX 2060 SUPER on NVIDIA CUDA
    Ok od=0.00000 md=0.00000
Testing mnist_cnn
    Ok od=0.00000 md=0.00000
Testing mnist_bn
    Ok od=0.00000 md=0.00000
Testing alexnet
    Ok od=0.00002 md=0.00002
Testing resnet18
    Ok od=0.00001 md=0.00001
Testing vgg16
    Ok od=0.00002 md=0.00022
squeezenet1_0 is blacklisted
Testing densenet161
    Ok od=0.00001 md=0.00274
Testing inception_v3
    Ok od=0.00002 md=0.00002
googlenet is blacklisted
Testing shufflenet_v2_x1_0
    Ok od=0.00002 md=0.00002
Testing mobilenet_v2
    Ok od=0.00002 md=0.00051
Testing mobilenet_v3_large
    Ok od=0.00003 md=0.00003
Testing mobilenet_v3_small
    Ok od=0.00003 md=0.00003
Testing resnext50_32x4d
    Ok od=0.00002 md=0.00711
Testing wide_resnet50_2
    Ok od=0.00001 md=0.00130
Testing mnasnet1_0
    Ok od=0.00004 md=0.00031
Testing efficientnet_b0
    FAIL od=0.00001 md=1.65422
Testing efficientnet_b4
    FAIL od=0.00001 md=0.27367
Testing regnet_y_400mf
    Ok od=0.00001 md=0.00259

```

---

<div class="post-metadata">

**Author:** ![artyom-beilis](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/artyom-beilis/32/207_2.png) [@artyom-beilis](https://dev-discuss.pytorch.org/u/artyom-beilis)\
**Post date:** [November 13, 2021, 10:04pm UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396/2 "2021-11-13T22:04:07Z")

</div>

Ok I found the bug… mostly because I discovered that on Intel GPU platform I had no issues so it allowed me to narrow down the case. However the question is still very valid.

Currently pytorch opencl/dlprimitives backend passes forward/backward computations tests against reference for following networks

```auto
alexnet
resnet18
vgg16
densenet161
inception_v3 (fwd only - backprop fails on cuda/cpu)
shufflenet_v2_x1_0
mobilenet_v2
mobilenet_v3_large
mobilenet_v3_small (fwd only - same failure on bwd on cuda) 
resnext50_32x4d
wide_resnet50_2
mnasnet1_0
efficientnet_b0
efficientnet_b4
regnet_y_400mf

```

---

<div class="post-metadata">

**Author:** ![tom](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/tom/32/26_2.png) [@tom](https://dev-discuss.pytorch.org/u/tom)\
**Post date:** [November 16, 2021, 9:29am UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396/3 "2021-11-16T09:29:33Z")

</div>

My very manual methods probably are

- use retain grad,
- use backward hooks,
- check individual modules,
- check individual functions with `torch.autograd.gradcheck`.

I do almost all my testing in Python, even when writing backend code. And of course PyTorch does come with a fair amount of tests.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![artyom-beilis](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/artyom-beilis/32/207_2.png) [@artyom-beilis](https://dev-discuss.pytorch.org/u/artyom-beilis)\
**Post date:** [November 16, 2021, 5:46pm UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396/4 "2021-11-16T17:46:37Z")

</div>

> [@tom](#):
>
> use backward hooks,

That is exactly what I was looking for. I’ve read about them and seems that they fit my needs.

Thanks

---

<div class="post-metadata">

**Author:** ![albanD](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/alband/32/9_2.png) [@albanD](https://dev-discuss.pytorch.org/u/albanD)\
**Post date:** [November 18, 2021, 8:37am UTC](https://dev-discuss.pytorch.org/t/debugging-back-propogation-results/396/5 "2021-11-18T08:37:30Z")

</div>

> [@artyom-beilis](#):
>
> In caffe for example with its static graph I could run `net.backward(from,to)` and look into intermediate layers.

You can do the same with `autograd.grad(from, to, retain_graph=True)`.