# State of symbolic shapes branch **URL:** \ **Category:** compiler\ **Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")\ **Posts on this page:** 1\ **Showing post:** 8
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [October 22, 2022, 10:20pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/8 "2022-10-22T22:20:25Z")
# State of symbolic shapes branch: Oct 22 edition The symbolic-shapes branch (PyTorch: [Symbolic shapes by ezyang · Pull Request #84246 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/84246)) is a long running branch containing a large number of features and bugfixes related to dynamic shapes support in PyTorch. Previous update: [State of symbolic shapes branch - #7 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/7#state-of-symbolic-shapes-branch-oct-16-edition-1) Commit ID at time of writing: 5f11aa560bdb406e6826e355edf19bda6174d63f ## Executive summary We focused on model enablement this week. We’re starting to hit the last mile on training, which means our pace is slowing as we spend time fixing more complicated bugs, though the pace on the branch is still faster than the rate we are merging code to master. - This major bug deserves a bullet on its own: we identified that torchdynamo was over-suppressing errors (even when dynamic shapes was off), artificially inflating the PASS rate of the dynamic shapes dashboard. This was fixed in [[dynamo] Unify raise\_on\_\* config to suppress\_errors and raise by default by ezyang · Pull Request #87440 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/87440) . This update post will report training status with errors suppressed, but in subsequent update posts we will stop suppressing errors for a more accurate depiction of our status. (On the plus side; error suppression would not have affected inductor speedup numbers, so even if we were skipping blocks to compile, we are still getting good speedups.) - **Model training status on symbolic-shapes.** (run by @ezyang); see also [Symbolic shapes work items tracker - Google Sheets](https://docs.google.com/spreadsheets/d/1-ghURoD6tzTushbCb22tFd5B5uNg7HSxVG4dQBwoaFA/edit?pli=1#gid=401920381) for hand audited results (not completely up to date atm, but actively used for task assignment). This run was done prior to fixing error suppression, so they suppress errors. - torchbench: 44 out of 55 (+8 WoW) - huggingface: 34 out of 44 (new) - timm: 38 out of 62 (new) - **Model inference status on master** (run by @ezyang); these runs are all WITHOUT suppressing errors. Pass rate is artificially depressed as PyTorch the run was done with was built without numpy support. - torchbench inductor: 27 out of 46 (+18 WoW) - torchbench aot\_eager: 35 out of 46 (+20 WoW) - torchbench aot\_eager, without dynamic shapes (baseline): 39 out of 46 (new) - **OpInfo tests on symbolic-shapes** (@ezyang) - 339 passed (+34 WoW), 381 failed (+114 WoW), 496 skipped (+461 WoW) (note: fail/skip increase stems from increased inplace testing from @albanD) - ~~208 passed (-1 WoW), 275 failed (+4 WoW), 125 skipped (-2 WoW).~~ (Logs at [gist:4eb8f7e6a861fb35096fad1cdf3c2b3b · GitHub](https://gist.github.com/ezyang/4eb8f7e6a861fb35096fad1cdf3c2b3b) ; at least a partial fix should be [Fix meta for meta\_fill\_](https://github.com/pytorch/pytorch/pull/84246/commits/aa865e2e32de06f4905319de698e061c8512bad9) and [Use functionalize in symbolic exhaustive tests, not sound otherwise](https://github.com/pytorch/pytorch/pull/84246/commits/d0b2f80479b55540630fec5bc71988f9d4a5e16c)). After fix: 219 passed (+10 WoW), 264 failed (-7 WoW), 125 skipped (-2 WoW) Previous branch diff: 30 files changed, 1209 insertions(+), 225 deletions(-) Current branch diff: 70 files changed, 1681 insertions(+), 430 deletions(-) ## Notable bug fixes - [[HACK] Don’t clone, instead don’t clobber proxies](https://github.com/pytorch/pytorch/pull/84246/commits/277ec2e4ac29168e7a18dcb2df33355f1af21e4d). This is a multi-week saga. The beginning of the story is that we noticed partitioning was failing, because occasionally size computation in the forward pass depended on size expressions on backward inputs. @wconstab identified that this was because we were overwriting proxies when a SymIntNode was reused on multiple tensors. To fix this, Will introduced a clone when we set SymInts into tensors ([Clone symint on set\_sizes\_and\_strides by wconstab · Pull Request #85878 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/85878)), so that when we assign proxies after a tensor returned by an operator, we would always have fresh SymIntNodes, preventing overwriting. However, after this fix landed, we noticed that occasionally SymIntNodes would show up that didn’t have any proxies at all! Brian fixed one particular instance of this ([functionalization: skip meta reference compute for aot autograd](https://github.com/pytorch/pytorch/pull/87108)), but there were others. We still don’t know how this situation arises (the minifier didn’t work), but @ezyang is testing an alternate fix on the branch, where instead of cloning SymInts, we simply avoid overwriting proxies if one already exists. This is a hack because it means we can’t faithfully report user programs (e.g., if you write `r = torch.add(x, y); s = r.size(0)`, this might end up reporting as `s = x.size(0)`), but with the few days of testing it seems to have fixed the problem and not caused any other regressions on benchmark models. - [Fixed FakeTensor not calling CompositeImplicitAutograd decomps sometimes](https://github.com/pytorch/pytorch/pull/87252). @anjali411 noticed that one of her models was stuck in an infinite loop involving `_to_copy` decomps. She and @ngimel unsuccessfully tried to debug it. @Chillee eventually pinned down the root cause by tracing through each decomposition the loop went through, and breaking the loop with a small change to fake tensor. This spawned a discussion about our use of decomposition tables being too complicated, which @SherlockNoMad has been working to refactor. - [fix minifier](https://github.com/pytorch/pytorch/pull/84246/commits/55dfd189542df88860abe99897e42b97794d10e9) and [minor improvements to minifier](https://github.com/pytorch/pytorch/pull/84246/commits/d5d8240d66a651d471b0893c6fa601a5fe41597b). In many situations, the minifier would fail to minify errors involving dynamic shapes. These fixes make more programs minify successfully; many bugs fixed this week were fixed with help from the updated minifier. - [Added some hacks to unblock hf\_reformer](https://github.com/pytorch/pytorch/pull/84246/commits/09d5312d93daba210a0d1b81972cc94d059de090). This bug manifested as another infinite loop in Sympy. @Chillee put in some hacks to fix it; I don’t really understand how it worked. - [Support symbolic as\_strided\_, and make unsqueeze\_ correct](https://github.com/pytorch/pytorch/pull/84246/commits/ddd172a00266bae86438c1b153fe2b3d1a1c604e). The bug here is pretty simple: `unsqueeze_` didn’t actually modify the tensor inplace. It was very difficult to diagnose without the minifier; after minification, it became clear that there was some unsqueeze\_ shenanigans. @albanD helped by remembering that unsqueeze\_ was implemented incorrectly on the branch. Going forward, I request people avoid committing _known wrong_ logic to the branch. - [Correct dtype calc when arange gets SymInt as argument](https://github.com/pytorch/pytorch/pull/84246/commits/07525926383bdbc404265e2d8df3468a0b30dee6). This was tricky to diagnose because arange passes fine (with an incorrect return dtype), there is a graph break, and then the program finally fails in another subgraph. The bug itself was also subtle, and we added a lint rule to catch future occurrences of it ([Audit for error prone isinstance int/float and add lint](https://github.com/pytorch/pytorch/pull/87345)) - [properly return NotImplemented on binOp(symInt, tensor) calls](https://github.com/pytorch/pytorch/pull/84246/commits/b312aabc2910db7a98794b94f017b96e729d7cd4). This bug was tricky to diagnose because understanding why it’s gone wrong requires knowing how reversible magic methods in Python work. First, Python attempts ` __mul__ `, and if it raises a TypeError or returns NotImplemented, it will silently swallow the error and try ` __rmul__ `. Previously this was accidentally working, but changes @bdhirsh made this no longer work. - [Add support for torch.tensor(scalar\_sym{int/float})](https://github.com/pytorch/pytorch/pull/84246/commits/bfbef1fdde75650b92a98f0b3a5197e5fd7611d0). The short term fix involves just guarding on the contents of the SymInt, but @anjali411 and @ezyang discussed a more permanent solution, which involves introducing a new operator `int_tensor(SymInt[] data)` which can be used to directly propagate small amounts of tensor data through without guarding. - [Bugfixes for metas](https://github.com/pytorch/pytorch/pull/84246/commits/531eaf4fe4be9fb800e299acfe2cf8e0cc3f8318). A lot of meta operations are incorrectly implemented because they use `.to(memory_format=...)`, which doesn’t do what you think it does. An old issue [`x.to(memory_format=torch.contiguous_format)` does not always return a contiguous tensor · Issue #62027 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/62027) has been revived and we are discussing what to do about this at the API level. - [Improve argument printing](https://github.com/pytorch/pytorch/pull/84246/commits/5f11aa560bdb406e6826e355edf19bda6174d63f). This is a dumb cosmetic problem that @ezyang eventually got fed up with and fixed. Sometimes, we would report an error like “Argument types did not match: expected tuple of ints but got tuple.” This is because argument parser did not consistently report what the inner type of a tuple/list was that caused the problem. This is now fixed. - [Convert torch.Size() argument to sym size in test\_proxy\_tensor](https://github.com/pytorch/pytorch/pull/87304). Some bugs in factory functions were not caught because while we symintify input tensors, we don’t symintify inputs which could be tensors. @albanD worked to beef up our testing here, taking advantage of the fact that OpInfos explicitly denote Size arguments with the Size tuple subclass. However, we also need to still increase testing for int arguments that aren’t Size. - [Add inplace function testing to test\_proxy\_tensor](https://github.com/pytorch/pytorch/pull/87324). We didn’t have any testing for inplace operators at all; in fact, OpInfo inplace testing is very poorly exercised. @albanD added more coverage here. - [[discussion] fix for aot autograd outputs that dont require grad](https://github.com/pytorch/pytorch/pull/86838). This is a kind of embarrassing bug in AOTAutograd where we just didn’t think hard enough about what to do in various edge cases involving `requires_grad`. @soumith suggested that we should do a more careful audit of AOTAutograd for other edge cases along these lines. - [symintify nll loss fns (#86915) by anjali411 · Pull Request #87095 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/87095) uncovered a bug in our default argument handling (specifically, we didn’t handle it at all for SymIntList). It was difficult to diagnose because we had uninitialized memory that would unpredictably fail asserts after using the default. @anjali411 has a follow up to make it explicitly error for any unrecognized argument types so that this doesn’t occur in the future. ## Merge to master retrospective We had very few reverts this week (hooray!) - [Add meta support for \_adaptive\_avg\_pool2d\_backward](https://github.com/pytorch/pytorch/pull/86359) was reverted because of a land race. We resolved to let the merge captain take care of PRs like this. ## What’s new on the branch this week? - [Make \_embedding\_bag\_backward\_symint call sym\_numel](https://github.com/pytorch/pytorch/pull/84246/commits/5626e3c97026646af3b009cf50faaaa43bbc2696) ezyang - [Symintify embedding\_sparse\_backward](https://github.com/pytorch/pytorch/pull/84246/commits/4f4a145111570f4af108d139afc964b968adef2e) ezyang - [Add support for symbolic shapes to sparse tensor](https://github.com/pytorch/pytorch/pull/84246/commits/0f4c378051aee6b1b67af5eca513fd10f44d7517) ezyang - [Symintify set\_indices\_and\_values\_unsafe](https://github.com/pytorch/pytorch/pull/84246/commits/bd457f53a90a3672e17988ff20eb9c20d10da7d5) ezyang - [floor() support for sym floats, yolov3 passes](https://github.com/pytorch/pytorch/pull/84246/commits/33a395da236a096caf87510c0c3058529d4f9e67) bdhirsh - [Preserve backtrace for KeyError](https://github.com/pytorch/pytorch/pull/84246/commits/7337578c990c2a6a048ee7517262b10acc5f5d06) ezyang - [Use str to render SymInt directly, rather than groveling in expr](https://github.com/pytorch/pytorch/pull/84246/commits/e760f15593ad43779e8a174567b4a0a6e7842c33) ezyang - [aten.upsample\_nearest2d\_backward.vec and aten.\_thnn\_fused\_lstm\_cell.d…](https://github.com/pytorch/pytorch/pull/84246/commits/aa3cf8f30fc8c2168cfe6788a8be82deae87dab9) bdhirsh - [[HACK] Don’t clone, instead don’t clobber proxies](https://github.com/pytorch/pytorch/pull/84246/commits/277ec2e4ac29168e7a18dcb2df33355f1af21e4d) - [Make Storage.nbytes support SymInt in Python](https://github.com/pytorch/pytorch/pull/84246/commits/d31566d96d2eb16b2f5e5312866360623a221d62) ezyang - [Minor improvement for storage size when its symbolic](https://github.com/pytorch/pytorch/pull/84246/commits/e9d2a768d1eb9d33d23d4d76488f27f8cd746f3a) ezyang - [Add meta for Scalar inplace overloads](https://github.com/pytorch/pytorch/pull/84246/commits/262f9291c907607bc7967292c5c004dd29b7721b) ezyang - [Fix bug when making storage\_offset symbolic](https://github.com/pytorch/pytorch/pull/84246/commits/87091025bfee36887eafa96303f515ebe27d4226) ezyang - [Improve gather dtype error message](https://github.com/pytorch/pytorch/pull/84246/commits/d5527f64667f711c66f9e80f841d164dfec10dc5) ezyang - [fix minifier](https://github.com/pytorch/pytorch/pull/84246/commits/55dfd189542df88860abe99897e42b97794d10e9) Chillee - [Also add inplace Tensor metas; needed for moco](https://github.com/pytorch/pytorch/pull/84246/commits/4f7ab7bd3d583d1bb9979edace848571509a8ee4) ezyang - [Add unary op neg, fix bugs in truediv and remove custom Ceil in favor…](https://github.com/pytorch/pytorch/pull/84246/commits/a960f4c37a807d8f264b8a88eeb7b220549c0ffc) anjali411 - [Symintify group\_norm](https://github.com/pytorch/pytorch/pull/84246/commits/5a646f79e9b67770040f8c2979948261c40a2995) ezyang - [symintify input metadata mismatch check](https://github.com/pytorch/pytorch/pull/84246/commits/ea4d3d7b753d3744aa09b4df2644b365679bee24) ezyang - [minor improvements to minifier](https://github.com/pytorch/pytorch/pull/84246/commits/d5d8240d66a651d471b0893c6fa601a5fe41597b) Chille - [Added some hacks to unblock hf\_reformer](https://github.com/pytorch/pytorch/pull/84246/commits/09d5312d93daba210a0d1b81972cc94d059de090) Chillee - [Support symbolic as\_strided\_, and make unsqueeze\_ correct](https://github.com/pytorch/pytorch/pull/84246/commits/ddd172a00266bae86438c1b153fe2b3d1a1c604e) + [Add regression test for unsqueeze\_](https://github.com/pytorch/pytorch/pull/84246/commits/d23016a2a6ae9c50f4e977411390cd2218a2a428) ezyang - [Remove hacky and wrong arange](https://github.com/pytorch/pytorch/pull/84246/commits/f8e8ea405f87b74d7cffc3ba8d9397b8a9fb1ab3) ezyang - [Correct dtype calc when arange gets SymInt as argument](https://github.com/pytorch/pytorch/pull/84246/commits/07525926383bdbc404265e2d8df3468a0b30dee6) + [Regression test for arange bug](https://github.com/pytorch/pytorch/pull/84246/commits/fbd64ebabe1cb789ba00aa50716d0be129f2c459) ezyang - [Audit for error prone isinstance int/float and add lint](https://github.com/pytorch/pytorch/pull/84246/commits/6d75d9edf90034b20a4dc9d6663a4dcdd74ab396) ezyang - [argmax and argmin meta registration](https://github.com/pytorch/pytorch/pull/84246/commits/4460be679fae9a4ac11d303bb1b3a48a80346556) ezyang - [symintify grid\_sampler + others](https://github.com/pytorch/pytorch/pull/84246/commits/ab5b9d347bc515b720c2f8bce75b01af7f7092e4) bdhirsh - [sym\_float()/sym\_int() instead of int/float() in decomps](https://github.com/pytorch/pytorch/pull/84246/commits/c2a41b67c9005bd508f733e9d2651307ed65cd2b) + [fix sym\_float/sym\_int usage](https://github.com/pytorch/pytorch/pull/84246/commits/147aa817faa151e86cfeb9d363ee729c4996bafa) bdhirsh - [sort symbolic meta](https://github.com/pytorch/pytorch/pull/84246/commits/493fc04023d1334809e476c145fe1fbe037183d8) bdhirsh - [support mixed type sym arithmetic, add some missing ops, fix floor()](https://github.com/pytorch/pytorch/pull/84246/commits/6995dc7cbdd3cb20ca748b280adf04dddb13fd50) bdhirsh - [gridsample + upsample\_bilinear2d backward meta](https://github.com/pytorch/pytorch/pull/84246/commits/c22785d3d5810aac80680ffccd515edabfb5e90a) bdhirsh - [properly return NotImplemented on binOp(symInt, tensor) calls](https://github.com/pytorch/pytorch/pull/84246/commits/b312aabc2910db7a98794b94f017b96e729d7cd4) bdhirsh - [scatter\_ and scatter metas](https://github.com/pytorch/pytorch/pull/84246/commits/e59f51bccf58f17dc42bc7b460020f665a8af0ae) bdhirsh - [symintify tensor.new()](https://github.com/pytorch/pytorch/pull/84246/commits/f7a008b35cbbf683d45cd8b776419f6ae26b6ed7) bdhirsh - [Add support for torch.tensor(scalar\_sym{int/float})](https://github.com/pytorch/pytorch/pull/84246/commits/bfbef1fdde75650b92a98f0b3a5197e5fd7611d0) + [fix for torch.tensor](https://github.com/pytorch/pytorch/pull/84246/commits/4bc36a1db907ff1132287aa0db2bc2208bfa7119) anjali411 - [Bugfixes for metas](https://github.com/pytorch/pytorch/pull/84246/commits/531eaf4fe4be9fb800e299acfe2cf8e0cc3f8318) ezyang - [Fix \_thnn\_fused\_lstm\_cell\_meta bug and beef up error checking](https://github.com/pytorch/pytorch/pull/84246/commits/0edd219fe639e2e7b78ac7b1cc3ddfde7cd3245d) ezyang - [Improve argument printing](https://github.com/pytorch/pytorch/pull/84246/commits/5f11aa560bdb406e6826e355edf19bda6174d63f) ezyang ## What’s made it to master this week? - albanD (merge captain) - [Add inplace function testing to test\_proxy\_tensor](https://github.com/pytorch/pytorch/pull/87324) - [Convert torch.Size() argument to sym size in test\_proxy\_tensor](https://github.com/pytorch/pytorch/pull/87304) - [Add meta inplace testing](https://github.com/pytorch/pytorch/pull/87291) - [Add randint OpInfo](https://github.com/pytorch/pytorch/pull/87231) (ezyang) - [conv backward impl](https://github.com/pytorch/pytorch/pull/87047) (Krovatkin) - [Symintify pad ops](https://github.com/pytorch/pytorch/pull/87046) (bdhirsh) - [autograd clone\_obey\_contract() symint support](https://github.com/pytorch/pytorch/pull/87044) (bdhirsh) - [meta funcs for avg\_pool2d and avg\_pool2d\_backward](https://github.com/pytorch/pytorch/pull/87043) (bdhirsh) - [Very limited pow support](https://github.com/pytorch/pytorch/pull/87042) (ezyang) - [Reland 3 of Merge more symbolic meta kernels and symint changes from branch](https://github.com/pytorch/pytorch/pull/86795) - ezyang - [[dynamo] Unify raise\_on\_\* config to suppress\_errors and raise by default](https://github.com/pytorch/pytorch/pull/87440) - [Audit for error prone isinstance int/float and add lint](https://github.com/pytorch/pytorch/pull/87345) - bdhirsh - [[discussion] fix for aot autograd outputs that dont require grad](https://github.com/pytorch/pytorch/pull/86838) - [functionalization: skip meta reference compute for aot autograd](https://github.com/pytorch/pytorch/pull/87108) - [ban .sizes() and .strides() calls in derivatives.yaml](https://github.com/pytorch/pytorch/pull/86611) - anjali411 - [symintify nll loss fns (#86915)](https://github.com/pytorch/pytorch/pull/87095) - [add sym\_int](https://github.com/pytorch/pytorch/pull/86916) - [Add meta support for \_adaptive\_avg\_pool2d\_backward (#86359)](https://github.com/pytorch/pytorch/pull/87074) - Chillee - [Ensure that symbolic variables incorporate fresh constraints before they’re used](https://github.com/pytorch/pytorch/pull/87254) - [Fixed FakeTensor not calling CompositeImplicitAutograd decomps sometimes](https://github.com/pytorch/pytorch/pull/87252) - [Unified symbolic shape variables between Inductor and AOTDispatcher](https://github.com/pytorch/pytorch/pull/87161) ## What’s coming next? - E2E training on master with inductor. - Plumb fake tensors up to torchdynamo - Plumb ShapeEnv guards up to torchdynamo guards (voz, this is in progress, check [Sym Shapes, Control Flow in Dynamo - RFC + Plan of Record - Google Docs](https://docs.google.com/document/d/1QJ-M4zfMkD-fjHIqW089RptjLl9EgozZGCceUbvmgfY/edit#heading=h.o9xfjd8yjybf) for design details) - Resolve strategy for sharing ShapeEnv between forward and backwards (@ezyang’s take: don’t return SymInts from forward pass) - All benchmark models are passing aot\_eager training on branch; tracked at [Symbolic shapes work items tracker - Google Sheets](https://docs.google.com/spreadsheets/d/1-ghURoD6tzTushbCb22tFd5B5uNg7HSxVG4dQBwoaFA/edit?pli=1#gid=401920381) - Fallback implementation for custom operators without symbolic shape propagation, inferred by running fallback on real operators - All OpInfo tests passing We also had some discussions about “unbacked” symbolic integers (e.g., as produced by item()). @Lezcano has a proposal for how to organize decompositions [Proposal for a property-based tag system for prims, refs, and decompositions - Google Docs](https://docs.google.com/document/d/1I15FEMfyeFahanFruDsQxfEhBtFbV_NPODlm79_ZC7M/edit?usp=sharing) --- _[View the full topic](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777)._