# State of symbolic shapes branch **URL:** \ **Category:** compiler\ **Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")\ **Posts on this page:** 1\ **Showing post:** 61
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 5, 2023, 4:27am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/61 "2023-07-05T04:27:16Z")
# State of symbolic shapes: Jul 4 edition Previous update: [State of symbolic shapes branch - #58 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/58#state-of-symbolic-shapes-jun-10-edition-1) ## Executive summary This is a little more than two week’s worth of updates, covering PSC week, Edward on vacation and July 4th holiday. - **Dynamic shapes by default is landed.** To be clear, this is “automatically enable dynamic shapes if recompiling due to size changes.” Most models running PT2 should not see any difference, as they are static already. If your model has dynamism, expect dramatically lower compilation times at the cost of some E2E performance. There may be performance regressions, please file bugs if you encounter any. You can use TORCH\_LOGS=dynamic to diagnose if dynamic shapes is doing something. Check also the [Meta only post](https://fb.workplace.com/groups/257735836456307/?multi_permalinks=528970869332801&hoisted_section_header_type=recently_seen) - **Internal telemetry for dynamic shapes.** [Add signpost\_event to dynamic\_shapes](https://github.com/pytorch/pytorch/pull/103882) adds a hook which we use internally to record all uses of dynamic shapes. You can check if dynamic shapes was actually used when `free_symbols` is non-zero. - **Notable bug fixes.** - [Allow Unequality in top level IR too](https://github.com/pytorch/pytorch/pull/103746) and [Support printing inequality in ExprPrinter](https://github.com/pytorch/pytorch/pull/104104) - fixes HuggingFace StableDiffusion - **Notable new issues.** - [[torch.compile] Guards failures due to storage offsets in new nightly ](https://github.com/pytorch/pytorch/issues/104563) - I believe this was fixed by reverting [https://github.com/pytorch/pytorch/pull/104204](https://github.com/pytorch/pytorch/pull/104204) **CI skips.** -3, -1, -1, -2 (no change). **Training dashboard (as of 7ae100628e).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2005%20Jun%202023%2004%3A18%3A01%20GMT&stopTime=Wed%2C%2005%20Jul%202023%2004%3A18%3A01%20GMT&granularity=day&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32&rBranch=main&rCommit=258d398eecd4c215c238e3318ac7d0d14251cf4f) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 89%, 57/64 → 91%, 58/64 | 98%, 45/46 | 100%, 60/60 | 88%, 7/8 → 100%, 8/8 | | Speedup | 1.11x → 1.08x | 1.59x → 1.58x | 1.19x → 1.21x | 1.30x | | Comptime | 67s → 78s | 99s → 152s | 110s → 134s | 31s → 78s | | Memory | 0.94x → 0.80x | 1.00x → 1.01x | 1.01x → 1.00x | 1.59x → 0.76x | - vision\_maskrcnn is responsible for the pass rate increase, but it’s fake: accuracy runs pass, but performance runs are still failing. Tracking issue: [vision\_maskrcnn: AssertionError: expected size 368==368, stride 156==28 at dim=0 · Issue #104653 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/104653) - Major HF compilation time regression is due to [Re-enable low memory dropout by eellison · Pull Request #103330 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/103330) which is being reverted - Memory compression change is due to [Add num\_elements\_per\_warp as an triton\_config by ipiszy · Pull Request #103702 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/103702) ; we are discussing how to deal with it, eellison’s position is that the change is “not real” (because it’s just due to an extra 250MB used for Triton autotuning) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/dff3c8c5d25cbfc4dfffd4b56213757469777a41.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/8d61f749d7d657840461a98ca686ba3255b04f2b.jpeg) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/b/b87ca0979c8d4eca953fff5b24d04054ca59db47.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a7ce7b8ddf03ba8ee1d80538aa11035bb1e53b0b.png) **Inference dashboard (as of 7b3242d5f7).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2005%20Jun%202023%2004%3A18%3A01%20GMT&stopTime=Wed%2C%2005%20Jul%202023%2004%3A18%3A01%20GMT&granularity=day&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32&rBranch=main&rCommit=7b3242d5f737d4a63870f8d6a008a9ae43a2fac9) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 88%, 63/72 → 86%, 63/73 | 100%, 46/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.52x → 1.53x | 1.64x | 1.72x → 1.73x | 1.92x → 1.96x | | Comptime | 24s → 28s | 38s → 45s | 30s → 34s | 45s → 53s | | Memory | 0.82x → 0.67x | 1.15x → 1.11x | 1.06x → 0.84x | 1.11x → 0.86x | - New model added: DALLE2\_pytorch - Compile time regression on Jun 29 is not entirely clear; maybe it is [fix specialization when you pass an unspec int into slicing on a Python list. by cdzhan · Pull Request #104142 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/104142) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e61b3428d54c6d4716bef57e5443489d07c82348.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e3444b44b27546ea06a12ccadda884be5d1fb2e9.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/29f90e9da218656de9079f4cdc58d04f33012160.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/4/4bd999ad23df7d5face8a193e94a26380a39dab7.png) --- _[View the full topic](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777)._