# State of symbolic shapes branch

**URL:** https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777
**Category:** compiler
**Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")
**Posts on this page:** 1
**Showing post:** 56

<div class="post-metadata">

### Author: ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)
#### Post date: [May 14, 2023, 4:58pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/56 "2023-05-14T16:58:14Z")

</div>

# State of symbolic shapes: May 14 edition

Previous update: [State of symbolic shapes branch - #54 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/54#state-of-symbolic-shapes-may-6-edition-1)

## Executive summary

- **New benchmarks.** Edward’s focus on this week was adding new benchmarks. We now have autoregressive generation benchmarks (hf\_T5\_generate, cm3leon\_generate, llama is now enabled in our benchmark suite, and coming soon are some new GNN benchmarks as well as working vision\_maskrcnn (having solved its eager mode determinism problems.) This finally makes good on our promise last month to add a dynamic-shapes oriented set of benchmarks for more fine-grained tracking. Once these benchmarks stabilize we will report for these models specifically in our updates. What’s still missing? We’d also like to get detectron2 working, and it would be good to have llama\_generate and gpt2\_generate (broken due to HF bug.) There’s also not much representation from generative image models at the moment. We also want to be tracking [HuggingFace’s compile experiments](https://github.com/younesbelkada/hf-torch-compile-benchmark) closely. The new benchmarks are also quite broken with torch.compile, more bug fixing required!
- **Translation validation.** Yukio has posted an initial [translation validator](https://github.com/pytorch/pytorch/pull/101146) for our symbolic shape reasoning. How exciting!
- **Hint sets.** One hypothesis we have about the perf regression from [State of symbolic shapes branch - #54 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/54#state-of-symbolic-shapes-may-6-edition-1) is that we are nixing optional optimizations that, actually in practice, would be OK to apply. (A simple example is 32-bit indexing, though Natalia tells us that this one is unlikely to matter.) If this is true, we could potentially allow for more optimizations by allowing for hint sets: rather than having only one hint when inductor compiles a graph, we maintain a hint for EVERY concrete instantiation of the size vars we have seen so far. Optional optimizations that generalize to all instantiations continue to be allowed. No one is working on this yet, and we don’t know how much it will buy.
- **Towards more determinism.** Check out this post [A pure Python implementation of roi\_align that looks just like its CUDA kernel](https://dev-discuss.pytorch.org/t/a-pure-python-implementation-of-roi-align-that-looks-just-like-its-cuda-kernel/1266) which discusses how we got vision\_maskrcnn passing the eager bitwise determinism check.
- **Notable bug fixes.**
  - [Turn on anomaly detection for AOTAutograd backward tracing](https://github.com/pytorch/pytorch/pull/101047) - We used this to diagnose a production bug regarding an operator with a data-dependent backwards implementation.
  - [Rewrite frame state to use a struct for shapes, splitting scalar and size, prep for stride](https://github.com/pytorch/pytorch/pull/101250) - This showed up in prod!
  - [Apply static policy correctly to unspec](https://github.com/pytorch/pytorch/pull/98983) - long overdue to land

**CI skips.** -2, -1, 0, -2 (-2, -1, 0, 0 WoW). The new failures are from new models (cm3leon\_generate, hf\_T5\_generate and llama) which we’ve enabled in torchbench.

**The dashboard (as of e406125).** This week on HUD.

| Metric | Torchbench | Huggingface | TIMM models |
| --- | --- | --- | --- |
| Passrate | 85%, 50/59 → 90%, 53/59 | 98%, 44/45 | 100%, 61/61 |
| Speedup | 1.09x | 1.41x → 1.39x 🔻 | 1.08x → 1.15x |
| Comptime | 84s → 90s 🔺 | 110s → 118s 🔺 | 133s → 134s 🔺 |
| Memory | 0.86x → 0.87x | 1.06x → 1.00x 🔻 | 1.01x → 0.97x 🔻 |

Some of this is just movement over all models in torchbench. However, in dynamic shapes specifically:

- torchbench dynamic only pass rate improvement on the 10th is hf\_BigBird, is likely due to Voz’s [Apply static policy correctly to unspec by voznesenskym · Pull Request #98983 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/98983) (this may also be responsible for the memory compression improvement)
- The bump on TIMM performance is likely due to the update to the TIMM commit [update timm commit by ngimel · Pull Request #100931 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/100931) though there is no obvious reason why performance improved again on the 13th.

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/aa30aa74a21b6d171e1d346285f0c1ea065f60b0.png)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/1a8b21f4863ca5e6953f162f7231a9e29c80d299.png)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/d114ac651ae087dfff168a7db4bed4103c519aea.png)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/c910c83564118276c486e77fbe77f455ba60ffef.png)

## What’s coming next?

- Voz: dynamic by default
- Edward: Fix bugs until I die, probably fix the minifier again

---

_[View the full topic](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777)._