# State of symbolic shapes branch **URL:** \ **Category:** compiler\ **Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")\ **Posts on this page:** 1\ **Showing post:** 48
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [March 26, 2023, 5:39pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/48 "2023-03-26T17:39:03Z")
# State of symbolic shapes: Mar 26 edition Previous update: [State of symbolic shapes branch - #46 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/46#state-of-symbolic-shapes-mar-18-edition-1) ## Executive summary This was a sleepy week. There’s a lot of activity going on in various dynamic shapes bugs on the bug tracker, though less fixes than I would have liked. - **Some bug fixes:** - [DCE inference graphs too](https://github.com/pytorch/pytorch/pull/97275) - [Make split reduction warning only emit once](https://github.com/pytorch/pytorch/pull/97112) - [Don’t run fallback if symbolic sizes in fake tensor](https://github.com/pytorch/pytorch/pull/97148) - [[inductor] hoist symbolic padding expressions](https://github.com/pytorch/pytorch/pull/97099) - Horace has a dynamic shapes minifier fix PR up at: [Fix minifier for symbolic shapes by Chillee · Pull Request #97428 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/97428) patch it in if you need to minify stuff - Brian has posted initial draft of AOTDispatch: [https://github.com/pytorch/pytorch/pull/97540](https://github.com/pytorch/pytorch/pull/97540) - **State of real world model enablement.** - [MNIST with dynamic batches](https://github.com/pytorch/pytorch/issues/96469) triggering lots of workstreams, check out [Notes from training mnist dynamic=True - Google Docs](https://docs.google.com/document/d/15dBE3-fJRNJ7FrhL-tprBelrFH4O4r4sR-wds06sjRo/edit#) - [HuggingFace T5 accuracy failure](https://github.com/pytorch/pytorch/issues/96971) has a repro, ready to investigate! - [wav2vec2 inference](https://github.com/pytorch/pytorch/issues/91719) is working again - [torch\_geometric performance](https://github.com/pytorch/pytorch/issues/94640) - a lot of subtasks: - puririshi98’s GCN NeighborSampling minimal test is only slightly faster with static compile (and only after fixing graph break); dynamic shapes is slower even with graph break fix, it could be split reductions, but even with static there is not much payoff. - rusty1s’s SAGEConv minimal test is not faster E2E even with static compile. pytorch-geometric has benchmarks at [Compiled Graph Neural Networks — pytorch\_geometric documentation](https://pytorch-geometric.readthedocs.io/en/latest/tutorial/compile.html#benchmark) showing E2E speedup, I don’t know how to reproduce it yet. - In any case, they are affected by overspecialization in backwards, maybe. [torch.compile (torch dynamo specifically) failing for simple GNNs trained with Neighbor Sampling (dynamic batches) · Issue #94640 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/94640#issuecomment-1479101869) - [vision\_maskrcnn](https://github.com/pytorch/pytorch/issues/93598) (also [`torch.compile` + `torch.no_grad` not working for Mask R-CNN · Issue #97340 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/97340)) - although it’s failing accuracy in the benchmark script, it can be run locally after [SymIntify roi\_align by ezyang · Pull Request #7448 · pytorch/vision · GitHub](https://github.com/pytorch/vision/pull/7448) and some massaging with static by default. Repro script at [maskrcnn.py · GitHub](https://gist.github.com/ezyang/2024bfdb6a2161f65ad7820264057fe7) - No update: LLAMA, InstructPix2Pix, detectron2, OpenNMT, fused boolean mask The numbers: - **Model status on master.** See also [Symbolic shapes work items tracker - Google Sheets](https://docs.google.com/spreadsheets/d/1-ghURoD6tzTushbCb22tFd5B5uNg7HSxVG4dQBwoaFA/edit?pli=1#gid=188393643) - aot\_eager inference: -1 (unchanged). Still vision\_maskrcnn sympy error from `reshape(torch.empty(s1, (s0 + 1)//2, 2), (s1, s0))`. - aot\_eager training: -2 (unchanged). Still botnet26t\_256 and eca\_botnext26ts\_256. - inductor inference: -7 (-1 WoW). The regression is tf\_efficientnet\_b0 from [https://github.com/pytorch/pytorch/pull/97099](https://github.com/pytorch/pytorch/pull/97099); this is a case of temporarily breaking things to move globally in a better direction - inductor training: -1 (+1 WoW). pytorch\_unet was fixed, leaving tf\_efficientnet\_b0 (NameError: name ‘s1’ is not defined) - **Graph breaks on master.** Sorry, CI is still not updated to do dynamic, maybe next week. We think the graph breaks are not super high priority at the moment, unless they are also affecting performance. - **Tracing cost of enabling dynamic shapes (aot\_eager).** `benchmarks/dynamo/run_delta.sh --backend aot_eager --devices cuda --cold-start-latency --ci`. Mean: 12s (unchanged), Max: 153s (+2 WoW, probably noise) ## What’s coming next? - Horace on PTO next week - Voz: finish refactor at [https://github.com/pytorch/pytorch/pull/97164](https://github.com/pytorch/pytorch/pull/97164), then shape\_env + 2 tier guard cache - Avik: Work on dynamic\_dim(x, 0) \<= 2 constraints API after Voz finishes refactor - Brian: More AOTDispatch - Edward: Not sure, flag me if you’re blocked --- _[View the full topic](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777)._