FYI: Many Dynamo tests were erroneously passing

compiler
1

FYI: Many Dynamo tests were erroneously passing

with: @voz

PyTorch CI runs regular PyTorch unittests with Dynamo enabled (i.e. PYTORCH_TEST_WITH_DYNAMO=1). Previously:

  • these ran with suppress_errors=True, which ignored errors
  • Many tests didn’t actually run with Dynamo enabled. Whenever the Dynamo cache filled up, tests would fall back to eager-mode PyTorch.

This caused a large amount of test flakiness and regressions: a test may fail when individually run with PYTORCH_TEST_WITH_DYNAMO=1, but pass when run with other tests due to the Dynamo cache filling up.

We fixed this by (1) turning suppress_errors off and (2) resetting the Dynamo state (including caches) after each test. This exposed ~8000 test failures that we added “unittest.skip and unittest.expectedFailure”-like decorators for to keep CI green.

We generate decorators for the tests listed in dynamo_test_failures.py. To debug a test, please remove it from this file and then run the test. If you’re making large changes to dynamo_test_failures.py, this script might help.

Next steps

  • If you own a feature, please take a look through dynamo_test_failures.py to see if your tests were impacted.
  • The overall pass rate for these tests is 69%.
  • It’s fairly difficult to manage 10000 unittest.skip and unittest.expectedFailure, especially when a single PR can fix ~100 issues. A better test disabling system for PyTorch CI could really help here.

Thanks to @voz and @bdhirsh for some very speedy code reviews.

3 Likes