Hey PyTorch developers!
This note is an update on OpInfos, their goals, and planned future work. If you have feedback on OpInfos or you’d like to request a new OpInfo feature then please respond to this post!
OpInfos Today
OpInfos are Python objects that describe operators and how to test them. See the recently updated developer wiki article, Running and Writing Tests in PyTorch 1.9, for links with detailed descriptions of what they define and how they’re used to generate tests.
There are currently 305 OpInfos, mostly covering operators in the torch, torch.fft, torch.linalg and torch.special namespaces, and these OpInfos are used to automatically validate operators work as expected with a variety of “operator consuming” systems, including autograd, torchscript, fx, and nnc. OpInfos have also replaced several historic test generators including “method_tests", and are now the definitive way to generate tests that run on a large set of operators.
OpInfo Goals
OpInfos are supposed to make it easy to comprehensively test operators.
Part of comprehensive operator testing is ensuring that each operator works as expected with PyTorch’s many “operator consuming” systems, like autograd, torchscript, fx, nnc, etc. Having to manually write tests for each of these would be incredibly time consuming, so instead developers can write an OpInfo and automatically generate them. Using OpInfos as the interface between operator developers and operator consumers greatly simplifies testing.
While OpInfos have been tremendously successful at validating operators work as expected across PyTorch’s different systems, developers still typically have to write their own tests to check that an operator actually produces the right result. That is, it’s easy to automatically check that torch.foo performs the same operation when running in eager mode or when traced with the jit, but harder to automatically check that torch.foo actually foo s. That requires defining a reference implementation and input generator, and today only unary elementwise operations and some spectral operations do so.
Future OpInfo Work
In H2 there will be three major OpInfo changes:
-
More OpInfo coverage, still focused on the torch, torch.fft, torch.linalg and torch.special namespaces
-
Automatic validation of binary elementwise and reduction operators
-
ModuleInfos, developed by @jbschlosser, for automated Module testing
If you’d like to suggest more OpInfo functionality or see OpInfo coverage beyond the above namespaces please post below! It’d be great to hear your feedback.