Recent autograd hooks improvements

Highlighting some of the improvements we’ve made to how backward hooks work in autograd. Here are a few of them:

  • The order in which backward hooks are fired is better defined. Tensor hooks always fire before grad_fn pre-hooks (this hook was recently added).
  • Bug fix: Tensor hooks always fire when gradients are computed for that tensor, even of that tensor’s grad_fn will not be fired, e.g. if that tensor is passed as input= to .grad.
  • Bug fix: registering a hook to Tensor, modifying that Tensor in-place, and then registering a new hook no longer produces silently incorrect results
  • Deprecation: Capture hooks (not really a public API anyway) are deprecated, we will continue to support its single and only use in dist engine for now (though it would be a good idea to migrate to tensor hooks if possible).
  • Bug fix: accumulate grad’s grad_fn post hooks can now observe changes made by hooks registered to leaf nodes

For more details about recent changes and how hooks work/are implemented in general, see:


Thank you for the update! Also, could you move the design doc to the public gdrive?