Highlighting some of the improvements we’ve made to how backward hooks work in autograd. Here are a few of them:
- The order in which backward hooks are fired is better defined. Tensor hooks always fire before grad_fn pre-hooks (this hook was recently added).
- Bug fix: Tensor hooks always fire when gradients are computed for that tensor, even of that tensor’s grad_fn will not be fired, e.g. if that tensor is passed as input= to .grad.
- Bug fix: registering a hook to Tensor, modifying that Tensor in-place, and then registering a new hook no longer produces silently incorrect results
- Deprecation: Capture hooks (not really a public API anyway) are deprecated, we will continue to support its single and only use in dist engine for now (though it would be a good idea to migrate to tensor hooks if possible).
- Bug fix: accumulate grad’s grad_fn post hooks can now observe changes made by hooks registered to leaf nodes
For more details about recent changes and how hooks work/are implemented in general, see: