# State of symbolic shapes branch

**URL:** <https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777>\
**Category:** compiler\
**Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")\
**Posts on this page:** 1\
**Showing post:** 20

<div class="post-metadata">

**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\
**Post date:** [December 2, 2022, 5:59am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/20 "2022-12-02T05:59:36Z")

</div>

# State of symbolic shapes branch: Dec 1 edition (even of PyTorch Conference)

The symbolic-shapes branch (PyTorch: [Symbolic shapes by ezyang · Pull Request #84246 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/84246)) is a long running branch containing a large number of features and bugfixes related to dynamic shapes support in PyTorch. Previous update: [State of symbolic shapes branch - #18 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/18)

Commit ID at time of writing: a05b7b1c73247ff562a82aac0edca79bbaebc2bd

## Executive summary

It is the eve of the PyTorch Conference and we have been busy getting things ready for some big announcements. 😉 Before and after Thanksgiving, many folks involved with dynamic shapes were deputized to help fix some major release blockers in the general compiler workstream; Brian and Jane landed all of the pieces needed to properly update batch norm running stats, and Alban and Edward found and fixed some more fairly major AOTAutograd bugs. On the dynamic shapes front, Voz has been steadily working on getting all of the Dynamo changes passing CI on master; half of the preparatory changes have been landed so far, and the branch has been resync’ed after those merges. There is some regression in the aot\_eager pass rate as we remove hacks and redo fixes properly.

- [Lazily guarding for duck sizing and views - Google Docs](https://docs.google.com/document/d/1VbiXkIhKy744Q7Lx2e8UUBvP_5RH_WU0FEc3VadVX4U/edit#heading=h.bk4qqlofhsq) is our plan (Voz + Edward, with some assistance from Horace) for dealing with the major outstanding soundness bug in dynamic shapes, which is that we are not guarding on duck sizing. Edward’s previous attempt at [Total revamp of how ShapeEnv symbol allocation works by ezyang · Pull Request #89695 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/89695) is hopelessly broken on master, so we are redoing it more incrementally with the new game plan. A bit of this has involved quite a lot of Dynamo refactoring.
- Batch norm running stats are now properly updated after [first draft of input mutation handling for aot autograd](https://github.com/pytorch/pytorch/pull/88817) and [fixes for inductor \<\> batch norm](https://github.com/pytorch/pytorch/pull/89603). Mark Saroufim has confirmed this resolves training instability when using inductor (turns out, batch norms running stats matter, whodathought.) Alban posted a really nice fix [Fix CopySlices logic to ensure wrapped node runs properly.](https://github.com/pytorch/pytorch/pull/89812) for a bug that took Voz and Ed two days to track down. But still there are more bugs: [AOTAutograd and internal grad\_fn structure - Google Docs](https://docs.google.com/document/d/1kBQUNIt1tRyKSiXV-db3C4dTvLYF2fC53dxCXqk08GQ/edit#)
- **Model training status on symbolic-shapes.** See also [Symbolic shapes work items tracker - Google Sheets](https://docs.google.com/spreadsheets/d/1-ghURoD6tzTushbCb22tFd5B5uNg7HSxVG4dQBwoaFA/edit?pli=1)
  - aot\_eager: 136 out of 166 (+8 WoW) [logs](https://gist.github.com/fa65798977b0d018ec5637e601ce2b47) [csv](https://gist.github.com/e03fce6e07ecdc184893e15755f7c63e)
  - inductor: TODO

- **OpInfo tests on symbolic shapes.**
  - `pytest test/test_proxy_tensor.py -k test_make_fx_symbolic_exhaustive` - TODO
  - `pytest test/functorch/test_aotdispatch.py -k test_aot_autograd_symbolic_exhaustive` - TODO

Previous branch diff: 68 files changed, 2612 insertions(+), 554 deletions(-)  
Current branch diff: 68 files changed, 1440 insertions(+), 290 deletions(-)

## What’s new on the branch these two weeks?

Metas/decompositions

- [Don’t decompose copy (sic)](https://github.com/pytorch/pytorch/pull/84246/commits/2a36e2b73cc2565fabecd3d739f074e06ab9884c) ezyang

Infrastructure

- [Make aten.copy preserve strides (hf\_Longformer)](https://github.com/pytorch/pytorch/pull/84246/commits/937e1e7b226841d0e75fc81d6089b81c46ad7001) ezyang
- [Retrace backwards in new shape environment](https://github.com/pytorch/pytorch/pull/84246/commits/17e9937041f69e3fb0933df566f65752ff8510c4) ezyang
- [Handle some edge cases regarding constant nodes and None SymInt tangents](https://github.com/pytorch/pytorch/pull/84246/commits/1a5ff545370639f17d40f5f6157eefd9fc74d93b) ezyang
- [Suppress guards on as\_strided call only.](https://github.com/pytorch/pytorch/pull/84246/commits/7fd7aeb8896ce17b78979fcc86ef5b5527ff7780) ezyang

Debug interpreter

- [Make DebugInterpreter work with varying dynamic shapes](https://github.com/pytorch/pytorch/pull/84246/commits/1e4e774e5d024a3f53da53142ee43222512604f4) ezyang  
[Properly handle out of order bindings](https://github.com/pytorch/pytorch/pull/84246/commits/94a91a8758da67702df79472e279bcb9b3051db9) ezyang
- [Minor QOL improvement for deferred equality tests](https://github.com/pytorch/pytorch/pull/84246/commits/f19ab61cd16db48f6eeb9aca0c25dd4525cbd0f0) ezyang

Dynamo

- [Add guard\_source for RandomValueSource](https://github.com/pytorch/pytorch/pull/84246/commits/54c29d7f15911f7bd62456353611b8eab4ca42dd) ezyang
- [Don’t use explain() for --explain; instead read it off the counters](https://github.com/pytorch/pytorch/pull/84246/commits/75638700460ed356c115c74409e0b49808299737) ezyang
- [Remove fake\_tensors\_available](https://github.com/pytorch/pytorch/pull/84246/commits/2ccb16874ef7c3047006931848c98ac72d91dd5c) ezyang
- [Cond capture with fake tensors actually works; don’t raise in this case](https://github.com/pytorch/pytorch/pull/84246/commits/9a35ff8a45a3ce6708fbeb708d259c1c09ecea1f) ezyang
- [Support unspecialized integers with dynamic shapes](https://github.com/pytorch/pytorch/pull/84246/commits/88cb25eb8651f0a71fd316af543c507fecbc898e) ezyang
- [Easy: These tests work with fake\_tensor\_propagation on](https://github.com/pytorch/pytorch/pull/84246/commits/6fefc424e239d84b4c3aeb1d52782297b636f1e0) ezyang
- [Force test\_rng\_state to run with fake tensor prop](https://github.com/pytorch/pytorch/pull/84246/commits/7387c229b3950e5168513ea26504747034510fde) ezyang (nb: was reverted)
- [Run optimizer tests with fake tensors](https://github.com/pytorch/pytorch/pull/84246/commits/ef05f1d14ee8d0caf369ca1a139758d2ed8010a7) ezyang
- [Reenable fake\_tensor\_propagation on test\_cudnn\_rnn](https://github.com/pytorch/pytorch/pull/84246/commits/4109c7b526b2e524889c9fafde8457d94760f78b) ezyang
- [Graph break on torch.tensor failure, allowing maml to run with fake t…](https://github.com/pytorch/pytorch/pull/84246/commits/0b76c3a34a0158172ab1c6610e53ae735fadea4a) ezyang
- [Remove fake\_tensor\_propagation](https://github.com/pytorch/pytorch/pull/84246/commits/e0c187959a828cdc38945fe6f75c6ca7f5d97848) ezyang
- [Delay verify correctness wrapping to call site.](https://github.com/pytorch/pytorch/pull/84246/commits/d51a5f294d1960299a193be682c088cac66e0d4d) ezyang
- [Don’t support kwargs at runtime in aot\_module\_simplified](https://github.com/pytorch/pytorch/pull/84246/commits/fd2bb3df4e23462e4e2b0f03072dd71c3119f31c) ezyang
- [Simplify aot\_module\_simplified by removing top\_args/top\_kwargs](https://github.com/pytorch/pytorch/pull/84246/commits/10c7366c3b88f6a91682e1aa5726261a0483c2b8) ezyang
- [Change aot\_module\_simplified to take take arguments directly](https://github.com/pytorch/pytorch/pull/84246/commits/73221ef23e7179edf85b2efebb4c1b5ff93dbad1) ezyang
- [Make aot\_module\_simplified accept fake tensors](https://github.com/pytorch/pytorch/pull/84246/commits/b6a710edee5a792f8ef69eeb4883129ae97d7dc5) ezyang
- [Use isinstance test rather than exact type test for wrap to fake](https://github.com/pytorch/pytorch/pull/84246/commits/feeac4f5bbf0c803165db4357915e89948847d0b) ezyang
- [Disable cache to restore accuracy](https://github.com/pytorch/pytorch/pull/84246/commits/7940f154ed88d4011967ef8a80cdeb2ce01442f5) ezyang

Inductor

- [Sufficient to get inductor working on BERT\_pytorch again](https://github.com/pytorch/pytorch/pull/84246/commits/062e3713d7e351dd3effcb317bcefd1bbf7dd7da) ezyang
- [[UPDATED PROTOTYPE] Use dynamo fake tensor mode in aot\_autograd, move…](https://github.com/pytorch/pytorch/pull/84246/commits/19b514f532473560de5651f883ad082c73f498b7) voz/ezyang
- [Restore the base fix, which fixes most of the missing symbol errors](https://github.com/pytorch/pytorch/pull/84246/commits/e02dd8600354098497fea552751af907ef0a5618) ezyang
- [Restore enable\_python\_dispatcher on has\_mutation analysis](https://github.com/pytorch/pytorch/pull/84246/commits/1931c23b17b07393e54fd850511a5e97b75e1f85) ezyang
- [Restore RANDOM\_VALUE fix](https://github.com/pytorch/pytorch/pull/84246/commits/2bf057facd3e744ce361ef4951c0c0bc0228bff1) ezyang
- [Restore TENSOR\_MATCH fix](https://github.com/pytorch/pytorch/pull/84246/commits/3766348ddb8d541f0b39f51292b831bba842efc5) ezyang
- [Restore stack tracking for sympy symbols](https://github.com/pytorch/pytorch/pull/84246/commits/2292c12f275a404cf2823b054de0ab0a101f74f9) ezyang

QOL

- [Make log\_extract.py able to deal with NotImplementedError](https://github.com/pytorch/pytorch/pull/84246/commits/27360fbed6a35ad7c8600ef1de0d3f636a64162e) ezyang
- [print graph breaks by default](https://github.com/pytorch/pytorch/pull/84246/commits/dfd20e52c77d8fcaf91735b325c29e077e135001) ezyang
- [Dasboard runner cmd](https://github.com/pytorch/pytorch/pull/84246/commits/a05b7b1c73247ff562a82aac0edca79bbaebc2bd) anijain

## Merge to master retrospective

- [Reland “Add single process version of dynamo distributed hf\_Bert tests (#89721)”](https://github.com/pytorch/pytorch/pull/89756) - this got bounced because not enough tests ran on PR. We added more files to automatically trigger inductor tests.
- [Refactor how AOTAutograd backends are defined](https://github.com/pytorch/pytorch/pull/89736) - this is just an example of a few cases where folks ran inductor CI, got accuracy failure on a model, and then spent a bunch of time trying to debug what had happened; when in fact, the failure was a preexisting master failure. It is not easy to identify these because ciflow/inductor does not run on every master commit.
- [Change aot\_module\_simplified to take take arguments directly](https://github.com/pytorch/pytorch/pull/89669) - this broke a timm model, and lead us on a pretty big chase that eventually revealed that example inputs being passed to backends did not have correct requires grad because they were being cloned. This was fixed by refactoring the AOTAutograd-Dynamo integration to not clone example inputs.
- [Remove fake\_tensor\_propagation](https://github.com/pytorch/pytorch/pull/89646) - this nearly got bounced because it broke some internal users who didn’t have fake tensor support for some operations. Averted because (1) their tests weren’t in CI and (2) it turned out to be pretty easy to add meta tensor support.
- [Don’t unsafely clone autograd meta](https://github.com/pytorch/pytorch/pull/89720) - this couldn’t be landed because it broke an inductor model, causing it to raise an error where previously it passed. This lead to a very long debugging session by Alban until we finally nailed the problem.

## What’s made it to master this week?

ezyang

- [Add manual meta implementations to quantize\_per\_tensor.tensor and co](https://github.com/pytorch/pytorch/pull/89958)
- [Guarantee symbol allocation for all sizes/strides/storage offset](https://github.com/pytorch/pytorch/pull/89879)
- [Add definitely\_not\_01 set to ShapeEnv.](https://github.com/pytorch/pytorch/pull/89871)
- [Reland “Add single process version of dynamo distributed hf\_Bert tests (#89721)”](https://github.com/pytorch/pytorch/pull/89756)
- [Refactor how AOTAutograd backends are defined](https://github.com/pytorch/pytorch/pull/89736)
- [Don’t unsafely clone autograd meta](https://github.com/pytorch/pytorch/pull/89720)
- [Implement guard\_source on RandomValueSource](https://github.com/pytorch/pytorch/pull/89711)
- [Beef up AOTAutograd logging with aot\_id and input descriptions](https://github.com/pytorch/pytorch/pull/89710)
- [Use isinstance test rather than exact type test for wrap to fake](https://github.com/pytorch/pytorch/pull/89671)
- [Make aot\_module\_simplified accept fake tensors](https://github.com/pytorch/pytorch/pull/89670)
- [Change aot\_module\_simplified to take take arguments directly](https://github.com/pytorch/pytorch/pull/89669)
- [Simplify aot\_module\_simplified by removing top\_args/top\_kwargs](https://github.com/pytorch/pytorch/pull/89666)
- [Don’t support kwargs at runtime in aot\_module\_simplified](https://github.com/pytorch/pytorch/pull/89664)
- [Delay verify correctness wrapping to call site.](https://github.com/pytorch/pytorch/pull/89662)
- [Ablate \_torchdynamo\_orig\_callable in wrap\_compiler\_fn](https://github.com/pytorch/pytorch/pull/89658)
- [Don’t suppress exceptions from backends](https://github.com/pytorch/pytorch/pull/89656)
- [Remove fake\_tensor\_propagation](https://github.com/pytorch/pytorch/pull/89646)
- [Support unspecialized integers with dynamic shapes](https://github.com/pytorch/pytorch/pull/89639)
- [Remove fake\_tensors\_available](https://github.com/pytorch/pytorch/pull/89637)
- [Access named parameters/buffers/etc via getattr rather than index](https://github.com/pytorch/pytorch/pull/89625)
- [Suppress guards on as\_strided call only.](https://github.com/pytorch/pytorch/pull/89569)
- [Don’t use explain() for --explain; instead read it off the counters](https://github.com/pytorch/pytorch/pull/89518)
- [Add crossref debug mode for functionalization, catches stride errors](https://github.com/pytorch/pytorch/pull/89498)
- [Make aten.copy preserve strides (hf\_Longformer)](https://github.com/pytorch/pytorch/pull/89464)
- [Don’t decompose copy (sic)](https://github.com/pytorch/pytorch/pull/89461)
- [Bind DispatchKey.Functionalonalize in pybind11](https://github.com/pytorch/pytorch/pull/89452)
- [Suppress guards when creating fake tensors](https://github.com/pytorch/pytorch/pull/89349)
- [When dealing with dupe arguments, prefer leafifying if possible](https://github.com/pytorch/pytorch/pull/89896)
- [Add debug asserts to AOTAutograd for input consistency with compilation](https://github.com/pytorch/pytorch/pull/89702)
- [Factor input deduplication into a separate function](https://github.com/pytorch/pytorch/pull/89701)

bdhirsh

- [don’t run input mutation analysis in dynamo](https://github.com/pytorch/pytorch/pull/89760)
- [fixes for inductor \<\> batch norm](https://github.com/pytorch/pytorch/pull/89603)
- [first draft of input mutation handling for aot autograd](https://github.com/pytorch/pytorch/pull/88817)

anjali411

- [Meta impl for linalg\_cholesky and linalg\_cholesky\_ex](https://github.com/pytorch/pytorch/pull/89430)

nkaretnikov

- [[primTorch] Unify checks for `embedding`](https://github.com/pytorch/pytorch/pull/89329)
- [[primTorch] Add decomp for `embedding_renorm_`](https://github.com/pytorch/pytorch/pull/89328)
- [Symintify `embedding`](https://github.com/pytorch/pytorch/pull/89327)
- [Symintify `select`](https://github.com/pytorch/pytorch/pull/89326)

voz

- [Add simple assert to detect fake tensors on modules](https://github.com/pytorch/pytorch/pull/89723)
- [Fix try/except flow where DataDependentOutputException is getting wrapped in a RuntimeError](https://github.com/pytorch/pytorch/pull/89314)

albanD

- [Fix CopySlices logic to ensure wrapped node runs properly.](https://github.com/pytorch/pytorch/pull/89812) and [beef up inplace/view note on copy slices](https://github.com/pytorch/pytorch/pull/89856)

## What’s coming next?

- Land fake tensor propagation from Dynamo to AOTAutograd (voz)
- ShapeEnv revamp to get guards for duck sizing (ezyang)
- GuardEnv for non-shape related extra guards produced by AOTAutograd (voz)
- Address CI comments for AOTAutograd input mutation, factoring it to be more modular (bdhirsh)
- Proper inductor integration (Chillee didn’t end up working on it, unallocated; mildly blocked on ShapeEnv revamp)

Our north star:

- All benchmark models are passing aot\_eager and inductor training on branch
- Fallback implementation for custom operators without symbolic shape propagation, inferred by running fallback on real operators
- All OpInfo tests passing
- Dynamic shapes on by default for developers / users

---

_[View the full topic](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777)._