# State of symbolic shapes branch **URL:** \ **Category:** compiler\ **Created:** [September 19, 2022, 5:32pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777 "2022-09-19T17:32:53Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 5, 2023, 4:27am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/61 "2023-07-05T04:27:16Z")
# State of symbolic shapes: Jul 4 edition Previous update: [State of symbolic shapes branch - #58 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/58#state-of-symbolic-shapes-jun-10-edition-1) ## Executive summary This is a little more than two week’s worth of updates, covering PSC week, Edward on vacation and July 4th holiday. - **Dynamic shapes by default is landed.** To be clear, this is “automatically enable dynamic shapes if recompiling due to size changes.” Most models running PT2 should not see any difference, as they are static already. If your model has dynamism, expect dramatically lower compilation times at the cost of some E2E performance. There may be performance regressions, please file bugs if you encounter any. You can use TORCH\_LOGS=dynamic to diagnose if dynamic shapes is doing something. Check also the [Meta only post](https://fb.workplace.com/groups/257735836456307/?multi_permalinks=528970869332801&hoisted_section_header_type=recently_seen) - **Internal telemetry for dynamic shapes.** [Add signpost\_event to dynamic\_shapes](https://github.com/pytorch/pytorch/pull/103882) adds a hook which we use internally to record all uses of dynamic shapes. You can check if dynamic shapes was actually used when `free_symbols` is non-zero. - **Notable bug fixes.** - [Allow Unequality in top level IR too](https://github.com/pytorch/pytorch/pull/103746) and [Support printing inequality in ExprPrinter](https://github.com/pytorch/pytorch/pull/104104) - fixes HuggingFace StableDiffusion - **Notable new issues.** - [[torch.compile] Guards failures due to storage offsets in new nightly ](https://github.com/pytorch/pytorch/issues/104563) - I believe this was fixed by reverting [https://github.com/pytorch/pytorch/pull/104204](https://github.com/pytorch/pytorch/pull/104204) **CI skips.** -3, -1, -1, -2 (no change). **Training dashboard (as of 7ae100628e).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2005%20Jun%202023%2004%3A18%3A01%20GMT&stopTime=Wed%2C%2005%20Jul%202023%2004%3A18%3A01%20GMT&granularity=day&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32&rBranch=main&rCommit=258d398eecd4c215c238e3318ac7d0d14251cf4f) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 89%, 57/64 → 91%, 58/64 | 98%, 45/46 | 100%, 60/60 | 88%, 7/8 → 100%, 8/8 | | Speedup | 1.11x → 1.08x | 1.59x → 1.58x | 1.19x → 1.21x | 1.30x | | Comptime | 67s → 78s | 99s → 152s | 110s → 134s | 31s → 78s | | Memory | 0.94x → 0.80x | 1.00x → 1.01x | 1.01x → 1.00x | 1.59x → 0.76x | - vision\_maskrcnn is responsible for the pass rate increase, but it’s fake: accuracy runs pass, but performance runs are still failing. Tracking issue: [vision\_maskrcnn: AssertionError: expected size 368==368, stride 156==28 at dim=0 · Issue #104653 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/104653) - Major HF compilation time regression is due to [Re-enable low memory dropout by eellison · Pull Request #103330 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/103330) which is being reverted - Memory compression change is due to [Add num\_elements\_per\_warp as an triton\_config by ipiszy · Pull Request #103702 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/103702) ; we are discussing how to deal with it, eellison’s position is that the change is “not real” (because it’s just due to an extra 250MB used for Triton autotuning) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/dff3c8c5d25cbfc4dfffd4b56213757469777a41.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/8d61f749d7d657840461a98ca686ba3255b04f2b.jpeg) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/b/b87ca0979c8d4eca953fff5b24d04054ca59db47.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a7ce7b8ddf03ba8ee1d80538aa11035bb1e53b0b.png) **Inference dashboard (as of 7b3242d5f7).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2005%20Jun%202023%2004%3A18%3A01%20GMT&stopTime=Wed%2C%2005%20Jul%202023%2004%3A18%3A01%20GMT&granularity=day&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32&rBranch=main&rCommit=7b3242d5f737d4a63870f8d6a008a9ae43a2fac9) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 88%, 63/72 → 86%, 63/73 | 100%, 46/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.52x → 1.53x | 1.64x | 1.72x → 1.73x | 1.92x → 1.96x | | Comptime | 24s → 28s | 38s → 45s | 30s → 34s | 45s → 53s | | Memory | 0.82x → 0.67x | 1.15x → 1.11x | 1.06x → 0.84x | 1.11x → 0.86x | - New model added: DALLE2\_pytorch - Compile time regression on Jun 29 is not entirely clear; maybe it is [fix specialization when you pass an unspec int into slicing on a Python list. by cdzhan · Pull Request #104142 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/104142) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e61b3428d54c6d4716bef57e5443489d07c82348.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e3444b44b27546ea06a12ccadda884be5d1fb2e9.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/29f90e9da218656de9079f4cdc58d04f33012160.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/4/4bd999ad23df7d5face8a193e94a26380a39dab7.png) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 10, 2023, 1:47am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/62 "2023-07-10T01:47:31Z")
# State of symbolic shapes: Jul 9 edition Previous update: [State of symbolic shapes branch - #60 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/60#state-of-symbolic-shapes-jul-4-edition-1) ## Executive summary - **Roadmap review for H2 2023.** We had roadmap review for PyTorch teams last week. Dynamic shapes presence on the roadmaps looks like this: (1) we have a bunch of internal enablement plans which require dynamic shapes to be well supported, make sure we are on point here ([Meta only](https://docs.google.com/document/d/1os-sp_jAKyQFTIdUwAfsgweRDEYx_bOXV7R-JZJqRVI/edit#heading=h.5px12wz913gg)), (2) we’re really interested in getting good inference performance on LLMs comparable to SOTA, e.g., llama (there’s some kv-cache / cuda graphs pieces here), (3) there’s still jagged/nested tensor work to do. On a more atomic level, the infra investments that dynamic shapes need to make are probably (a) two level guards for backwards shape guards, (b) improved accuracy/compile time debugging tools, (c) more aggressive symbolic reasoning enabled by translation validation, (d) obvious inductor compilation perf improvements, e.g., from split reductions, (e) [Unbacked integers for eager mode](https://docs.google.com/document/d/1Dge173HVbXnTysnvp8716mi_0BlEpJZbmhXbI_-WdqI/edit). I’d also like to finally get vision\_maskrcnn and detectron2 working on PT2, but LLMs take priority over this. - **Which operators specialize their inputs?** In the old days, dynamic shapes enablement would typically fail because of missing meta functions. These days, things usually don’t fail, but you may end up having specialized and recompiling anyway. @anijain2305 has been working on sweeping operators to find out which arguments get specialized, to help folks have a better understanding of what will be dynamic versus not. - **Translation validation landed!** [Re-land: Turn translation validation on for tests and accuracy runs by default.](https://github.com/pytorch/pytorch/pull/104467) was reverted last week, but has successfully relanded for real. This paved the way for simplification improvements including [Value range refinement using uni-variate expressions.](https://github.com/pytorch/pytorch/pull/97963), which are important because they reduce the number of guards we emit in the end. - **Notable bug fixes.** - We landed a few fixes to help fix issues in [https://github.com/fxmarty/accelerated-pytorch-transformers-generation/:](https://github.com/fxmarty/accelerated-pytorch-transformers-generation/:) [Generalize sympy.Rel test to sympy.logic.boolalg.Boolean](https://github.com/pytorch/pytorch/pull/104833), [Allow for torch.sym\_int to return int while tracing](https://github.com/pytorch/pytorch/pull/104837); there’s a few more coming too - **Notable new bugs.** None of these are user bugs; they were all filed by the team - [[compile][dynamic] dsplit is seeing a list of mixed ints and symints](https://github.com/pytorch/pytorch/issues/104814) - [[compile][dynamic] amin dim kwarg is getting symintified](https://github.com/pytorch/pytorch/issues/104812) - [Regression in Maml due to dynamic shapes](https://github.com/pytorch/pytorch/issues/104805) - [Regression in Dalle2 due to dynamic shapes](https://github.com/pytorch/pytorch/issues/104797) **CI skips.** -3, -1, -1, -2 (no change). **Training dashboard (as of dd6c38cb59).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sun%2C%2002%20Jul%202023%2019%3A26%3A45%20GMT&stopTime=Sun%2C%2009%20Jul%202023%2019%3A26%3A45%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=5da4745c24859d501c9555dbe67a20dfcd199942&rBranch=main&rCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 91%, 58/64 → 89%, 57/64 | 98%, 45/46 | 100%, 60/60 → 97%, 58/60 | 100%, 8/8 → 88%, 7/8 | | Speedup | 1.08x → 1.11x | 1.58x → 1.60x | 1.21x → 1.20x | 1.30x | | Comptime | 78s → 97s | 152s → 124s | 134s → 178s | 78s → 40s | | Memory | 0.80x | 1.01x → 0.97x | 1.00x | 0.76x → 0.73x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/9/996eaba3d3f93bebd20ebd7c78c3415c2e717f0b.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e2e97dc8ca84323e599a8cae8608365597c90919.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/b/b73c881f265fbe5abd9e988c1bda837114fa71de.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/eb84bfb45cb41872fc0fa4df65d6210a428885ea.png) - vision\_maskrcnn went back to failing, seems flaky. 🤷 - eca\_botnext26ts\_256 and mobilevit\_s timed out due to translation validation being enabled. #104654 fixed it (to be visible in next perf run.) Compilation time increase also appears to be due to TV. **Inference dashboard (as of dd6c38cb59)** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2026%20Jun%202023%2001%3A09%3A56%20GMT&stopTime=Mon%2C%2010%20Jul%202023%2001%3A09%3A56%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=dd6c38cb596199d70bba76688c99ed919d77002d&rBranch=main&rCommit=7bc181d374a54db4b1556d263be965f6a848eecb) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 86%, 63/73 | 100%, 46/46 → 98%, 45/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.52x | 1.65x → 1.64x | 1.73x | 1.92x → 1.96x | | Comptime | 28s | 44s | 34s | 53s | | Memory | 0.67x | 1.11x | 0.84x | 0.86x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/d085431a0702119fa4ab497c05cefaca5c2e2bc0.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/caddcab953036708e60a5e70cf4d98867d8456f4.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/f5a2350768b6357e6b065bbec3006221fd9fcd8e.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/fea1e14b3ba9528bb4f66d13e46ebcc58a6137a3.png) - GPT2ForSequenceClassification is having some trouble across the board on all configurations; it’s currently failing accuracy. ## What’s next? - Edward: Keep helping HF on their llama optimization; two level guards for backwards ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 15, 2023, 8:25pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/63 "2023-07-15T20:25:25Z")
# State of symbolic shapes: Jul 15, 2023 edition Previous update: [State of symbolic shapes branch - #61 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/61#state-of-symbolic-shapes-jul-9-edition-1) ## Executive summary - **Dynamic shapes now support mode=“reduce-overhead” (CUDA graphs).** Conventional wisdom was that dynamic shapes are incompatible with CUDA graphs, because any given CUDA graph recording can only work for a single static shape, and CUDA graphs requirement of hard coded memory addresses means that each CUDA graph takes up quite a lot of CUDA memory. However, this conventional wisdom is wrong: (1) multiple CUDA graphs can share the same memory pool, as long as you don’t have any live tensors from one pool to the next (this is precisely what CUDA graph trees by @eellison implements), and (2) recording a CUDA graph is much, much cheaper than running the entire PT2 compilation stack, so it is profitable to compile a dynamic program once and then CUDA graph it multiple times. [https://github.com/pytorch/pytorch/pull/105064](https://github.com/pytorch/pytorch/pull/105064) realizes these gains and switches our dynamic shapes benchmark configuration to use CUDA graphs, resulting in hefty performance gains with only a modest increase in compile time. Importantly, these benchmarks cover our \_generate inference benchmarks, which actually make use of multiple sizes as sequence length varies. There’s more to be done here: our memory usage for this use case can be suboptimal, because the caching allocator doesn’t know that it’s OK to waste space for small allocations by fitting them inside larger allocations for a larger dynamic size. We also observed that folks using this CUDA graphs trick tend not to generate CUDA graphs for every size, but instead prefer to linearly sample sizes and pad; we should make it easier to do this (perhaps with a padding tensor subclass.) One cool result is a 6x performance improvement on [cm3leon](https://ai.meta.com/blog/generative-ai-text-images-cm3leon/), a newly announced multi-modal model from Meta AI. - **New API: torch.\_dynamo.maybe\_mark\_dynamic.** [Add torch.\_dynamo.maybe\_mark\_dynamic](https://github.com/pytorch/pytorch/pull/105145) lets you suggest that we should try compiling a tensor dynamically, but doesn’t raise an error if it gets specialized (unlike `mark_dynamic`). - **Infer valid input sizes from programs.** Horace has wanted this for some time, and with Yukio’s recent Z3 translation validation work landed, it turned out to be pretty easy to write a PoC to exhaustively search the space of valid inputs, using guards to turn us away from portions of the space we’ve seen before. Check it out at [dinfer.py · GitHub](https://gist.github.com/ezyang/192c1b5eb57ff95a46d3b50aa46e3193). If anyone is interested in productionizing this, it would be a neat little project to (1) put this code in PyTorch and put a nicer API on it (note that as written, you have to specify the input dimensions and dtypes of input tensors, so you’ll need to figure out a good way of specifying or inferring this info), (2) improve the solving code to minimize the generated sizes for an equivalence class, and (3) use it for something cool; e.g., you could use it to automatically generate sample inputs for OpInfo tests. Tag me (@ezyang) as reviewer if you send a PR! - **Enabling automatic\_dynamic\_shapes in fbcode, for real this time.** It turns out that I failed to actually turn things on in fbcode last time, so actually do it for real this time: [Switch automatic\_dynamic\_shapes to True by default in fbcode](https://github.com/pytorch/pytorch/pull/104883). This got reverted once for breaking an internal model unit test ([Incorrect ValueRanges analysis · Issue #105097 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/105097), fixed by [Perform value range analysis with rationals when possible by lezcano · Pull Request #105137 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/105137), thanks @Lezcano for the speedy fix.) At time of writing, the PR has not actually hit fbcode yet. - **lit\_llama is finally landed in torchbench.** [Add lit-llama benchmarks (logits, autoregressive generation, lora fine tuning) by ezyang · Pull Request #1730 · pytorch/benchmark · GitHub](https://github.com/pytorch/benchmark/pull/1730) At time of writing this model is in canary models because the weight download is a little flaky. This is the only 7B model in our benchmark suite and there’s a bit of pain associated with this; for example, we can’t run accuracy tests on this model, because accuracy tests are implemented by holding two copies of the model in memory, which we can’t do at 7B parameters. - **Notable bug fixes.** - [Transmute refined SymInt into int](https://github.com/pytorch/pytorch/pull/104828) makes it more likely you’ll get an int rather than a SymInt if the SymInt got specialized into a constant. This sometimes caused some bugs with downstream components that can handle SymInt but choke on int. - [Fix AttributeError(“‘constexpr’ object has no attribute ‘type’”)](https://github.com/pytorch/pytorch/pull/104831); another fix for HF llama - Coming soon: [Immediately compile backwards graph in AOTAutograd if dynamic shapes](https://github.com/pytorch/pytorch/pull/104971) will fix “guard ignored, could cause correctness problems” warning. It’s waiting on review right now. - **Notable new bugs.** - [Inductor backend for CPU inference extremely slow](https://github.com/pytorch/pytorch/issues/105075) - this bug actually seems to be fixed on main thanks to dynamic shapes. Moral of the story: if you want dynamic shapes, use a nightly! We have soooo many improvements. - [StableDiffusion with dynamic=True still recompiles](https://github.com/pytorch/pytorch/issues/104913) **CI skips.** -3, -1, -1, -2 (no change). \*\*Training dashboard (as of 7b4d080496). [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2001%20Jul%202023%2019%3A35%3A39%20GMT&stopTime=Sat%2C%2015%20Jul%202023%2019%3A35%3A39%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=7b4d080496a971563998c3a1bcc500e76e8baffe&rBranch=main&rCommit=5da4745c24859d501c9555dbe67a20dfcd199942) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 89%, 57/64 → 92%, 59/64 | 98%, 45/46 → 96%, 44/46 | 97%, 58/60 → 98%, 59/60 | 88%, 7/8 → 100%, 8/8 | | Speedup | 1.11x → 1.52x | 1.60x → 1.66x | 1.20x → 1.27x | 1.30x → 1.93x | | Comptime | 97s → 86s | 124s → 120s | 178s → 142s | 40s → 42s | | Memory | 0.80x | 0.97x | 1.00x → 1.01x | 0.73x → 0.69x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/cfaf13142989a3d781612d3e86758ea884271bee.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/5/5fe28a73e3df51a7aafa28a1319c19528f152075.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/872ae87757ebaa2d1bd46a84756593a4ba2222af.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a64689ddb330083fe2a1ee66ff801fd03a37e1dd.png) - Now passing: hf\_Longformer (this used to fail with `ValueError: Cannot view a tensor with shape torch.Size([4, 12, 1024, 513]) and strides (6303744, 513, 6156, 1) as a tensor with shape (48, 4, 256, 513)`, this is thanks to Brian Hirsh finally landing his AOTAutograd longformer fix), vision\_maskrcnn (flaky), eca\_botnext26ts\_256 and mobilevit\_s (used to timeout; maybe the speedup from CUDA graphs was enough to get it under the timeout again) - Now failing: DebertaV2ForQuestionAnswering (failing accuracy due to cudagraphs, failing on inductor\_with\_cudagraphs too), cait\_m36\_384 (OOMing on accuracy due to increased CUDA graph memory usage) - Speedups: The majority of our speedups are due to the enablement of CUDA graphs for dynamic shapes. Some notable models and their speedups: BERT\_pytorch (1.7698 → 3.3071), hf\_GPT2 (1.7728 → 2.0056), basic\_gnn\_gin (1.3151 → 2.4841). The improvements on HF and TIMM models are much more modest since these are not super overhead bound models. Note that these numbers are still behind inductor\_with\_cudagraphs, because we are still losing some optimizations from running the PT2 compiler stack without static shapes. - Slowdowns: dlrm (infra failure due to cudagraphs, failing on inductor\_with\_cudagraphs too), hf\_T5 (2.0252 → 1.8939, oddly enough–could this be due to memory pressure? But even more weirdly, hf\_T5\_large imporved perf) - Comptime/Memory: By in large compilation time did not increase, but for our training setup this is expected as we only actually run at one batch size, so you are simply measuring the cost of a single CUDA graph recording. As expected, memory compression ratio gets worse, due to standing allocation from CUDA graphs. **Inference dashboard (as of 7b4d080496)**. [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2008%20Jul%202023%2020%3A02%3A15%20GMT&stopTime=Sat%2C%2015%20Jul%202023%2020%3A02%3A15%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=7b4d080496a971563998c3a1bcc500e76e8baffe&rBranch=main&rCommit=dd6c38cb596199d70bba76688c99ed919d77002d) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 86%, 63/73 → 88%, 64/73 | 98%, 45/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.52x → 1.50x | 1.64x → 1.76x | 1.73x → 1.62x | 1.96x → 2.94x | | Comptime | 28s → 36s | 44s → 46s | 34s | 53s → 72s | | Memory | 0.67x → 0.68x | 1.11x | 0.84x → 0.85x | 0.86x → 0.87x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/4/476b4dc65d2051162fd108f4c0d49e5ed87f52c9.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/1396585a6f181eedf8225032d41f40017eee5264.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/5/542daa27ec41eecce080be7f7483805e2b52b13f.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a9cdd2fa2ab26735b0654d02fb747dd4a8005ef6.png) - Now passing: hf\_Longformer (see training above) - Speedups: torchbench numbers are actually a huge mixed bag. Here are some of the wins: BERT\_pytorch (2.2317 → 2.4529), basic\_gnn\_edgecnn (1.7809 → 1.8732). Note that for some reason many of the GNN variants are failing performance on inference (but not accuracy), cm3leon\_generate (1.3037 → 5.7822, WOW! This is consistent with some perf analysis Armen and I did months ago, where I concluded that cm3leon was hella overhead bound), hf\_T5\_generate (2.2619 → 8.2081), hf\_T5\_large (3.1690 → 5.1747) - Slowdowns: A lot more models did worse with CUDA graphs enabled, including LearningToPaint (1.9209 → 1.6812), resnet18 (1.7779 → 1.4028), shufflenet\_v2\_x1\_0 (1.9882 → 1.6010), squeezenet1\_1 (1.8625 → 1.0040), yolov3 (2.0997 → 1.8843). It’s not entirely clear what’s going on here, but we will note that there was sizable dip in CUDA graphs performance without dynamic shapes too this week on torchbench. There is an across the board performance regression on TIMM models (and a slight regression on HuggingFace too.) - Comptime/Memory: Comptime generally got worse across the board, but not too much worse. Particularly notable are the generate models: hf\_T5\_generate (881 → 1032), cm3leon\_generate (131 → 203). CUDA graphs is not free, but given that we’re running at much more than two sequence lengths, you can see the bulk of the compile cost is the PT2 stack. For the most part, memory usage stayed fairly stable, interestingly enough. ## What’s next? - I think I want to investigate the memory planning situation with CUDA graphs a bit more; I also think it’s a good time to teach Inductor how to deal with data-dependent ops (without having to graph break on them.) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 22, 2023, 5:43pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/64 "2023-07-22T17:43:07Z")
# State of symbolic shapes: Jul 22, 2023 edition Previous update: [State of symbolic shapes branch - #62 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/62#state-of-symbolic-shapes-jul-15-2023-edition-1) ## Executive summary - **llama\_v2 is out.** @msaroufim has a PR adding it to torchbench suite: [llama v2 7b by msaroufim · Pull Request #1775 · pytorch/benchmark · GitHub](https://github.com/pytorch/benchmark/pull/1775) - **Whole model compilation for sparse architecture in recommendation models.** @anijain2305 has been looking at improving the ability to slap torch.compile on an arbitrary function and have it just work. One of the more challenging situations is when we try to compile the sparse architecture of recommendation models; e.g., code that interacts with [torchrec.sparse/([torchrec/torchrec/sparse at main · meta-pytorch/torchrec · GitHub](https://github.com/pytorch/torchrec/tree/main/torchrec/sparse)). In one example, a KeyedJaggedTensor is being compiled, but it contains a list of 500 integers, each of which varies over time and participates in many guards. This is a worst case scenario for dynamic shapes compile time. However, we are also running into lots of graph breaks, which are resulting in us trying to compile smaller fragments than we should. There will be a mix of fixing graph breaks (some of them are due to data dependent output size operators like nonzero–time to fix this!) and otherwise figuring out what else needs to be done. - **Notable bug fixes.** - [Immediately compile backwards graph in AOTAutograd if dynamic shapes](https://github.com/pytorch/pytorch/pull/104971) landed. It appears to not be a complete fix [Make guard after freeze a hard error](https://github.com/pytorch/pytorch/pull/105734) edge case involving export and fake cloning - **Notable new bugs.** - [Tweak dynamic=False behavior](https://github.com/pytorch/pytorch/pull/105715). Once this PR is in, dynamic=False will disable dynamic shapes, which seems like intuitive behavior. - [Error using torch.compile with HF transformers and model `mosaicml/mpt-7b`](https://github.com/pytorch/pytorch/issues/105686). This is einops rearrange messing up on SymInt caching again. - [[inductor] unexpected dynamic shape error encountered in TritonTemplate](https://github.com/pytorch/pytorch/issues/105634) - Triton templates broken with dynamic shapes, fix in the works at [https://github.com/pytorch/pytorch/pull/105295](https://github.com/pytorch/pytorch/pull/105295) - [Dynamic int not being propagated when used on ` __setitem__ `](https://github.com/pytorch/pytorch/issues/105533). We haven’t started looking into this yet. **CI skips.** -3, -1, -1, -2 (no change). **Training dashboard (as of 0ad93a3d56).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2008%20Jul%202023%2016%3A13%3A56%20GMT&stopTime=Sat%2C%2022%20Jul%202023%2016%3A13%3A56%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=0ad93a3d5684c2026bda8ff4ab7c72c6596a225b&rBranch=main&rCommit=7b4d080496a971563998c3a1bcc500e76e8baffe) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 92%, 59/64 | 96%, 44/46 | 98%, 59/60 | 100%, 8/8 | | Speedup | 1.52x → 1.54x | 1.66x → 1.69x | 1.27x → 1.28x | 1.93x → 1.97x | | Comptime | 86s → 81s | 120s → 107s | 142s | 42s → 38s | | Memory | 0.80x → 0.79x | 0.97x → 0.96x | 1.01x | 0.69x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/7/748dbc56bcd30898b415be68ccdd2235539f47d5.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e27e48334bcf5690c5d23585475bd750ec0a809f.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/d5dcfe3e1f52347a2fca7029bc348d4ee98eb660.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/4/48d06ca162df203346fab4ad9596db7d976d374a.png) Not really much to say; the slight improvements appear to be within noise. **Inference dashboard (as of 0ad93a3d56).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2008%20Jul%202023%2017%3A22%3A57%20GMT&stopTime=Sat%2C%2022%20Jul%202023%2017%3A22%3A57%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=0ad93a3d5684c2026bda8ff4ab7c72c6596a225b&rBranch=main&rCommit=7b4d080496a971563998c3a1bcc500e76e8baffe) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 88%, 65/74 | 98%, 45/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.50x → 1.55x | 1.76x → 1.78x | 1.62x → 1.79x | 2.94x → 3.03x | | Comptime | 36s → 35s | 46s → 44s | 34s → 36s | 72s | | Memory | 0.68x | 1.11x | 0.85x → 0.84x | 0.87x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/1e1518d9bdd0ccb506f71c3e521f8b6112a9bbb4.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/6599492751d5b3ec61dba03ab9212f778b6cfd95.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/4/49eadfd5f93eb0517badf85ccfebc110c88faf6b.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/808674d839db8921e9b0fa004b8f1c40019e79b2.png) - Across the board timm improvement is due to reverting regressing PR from last week [https://github.com/pytorch/pytorch/pull/105102](https://github.com/pytorch/pytorch/pull/105102) (bucketization related). This also explains the slight boost in torchbench numbers. - inductor\_with\_cudagraph\_freezing is a one off benchmark run for [Add Freezing Option to Benchmarking by eellison · Pull Request #105616 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/105616) ## What’s next - CUDA graphs memory planning is lower priority for now (@eellison may take a look, but higher priority is actually being able to turn on CUDA graphs in prod situations; a big problem here is when we fail to compile the entire extent of the model, causing CUDA graphs to increase overall memory usage.) It looks like we definitely need data-dependent op support in inductor though, based on sparse arch investigation. ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [July 29, 2023, 9:38pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/65 "2023-07-29T21:38:26Z")
# State of symbolic shapes: Jul 29, 2023 edition Previous update: [State of symbolic shapes branch - #63 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/63#state-of-symbolic-shapes-jul-22-2023-edition-1) ## Executive summary - **Data dependent shape support in Inductor.** I got an end to end PoC of a pointwise and then reduction with hacks working in Inductor: [gist:1293a41299604c44310341b7540eabcb · GitHub](https://gist.github.com/ezyang/1293a41299604c44310341b7540eabcb) The main gaps: (1) optional optimizations failing to retrieve hints (Triton size hints (pick 8192 to prevent the block size from shrinking), multiple of 16 hints (pick something not multiple of 16), 32-bit indexing), (3) buffer reuse (key’ing on the str rep is fine, use sympy\_str), (4) updating wrapper codegen to create bindings to i0 variables. In general, it seems it’s pretty useful to have accurate maximum size information, for which ValueRanges is an incomplete fix because we don’t support symbols (s0) in bounds. Another trick we plan to implement is a special irrefutable guard, where if we guard on an unbacked symint, we instead just assume it is True and add a runtime assertion. One question is whether or not we _always_ can get dynamic shapes working no matter what. It seems that in Inductor, we usually can just turn off optimizations to avoid guards. So it seems we just need to get host-side torch.cond working to handle everything else. Some fixes for these are in: [If we can’t statically prove 32-bit indexing OK, only add guard if hint exists](https://github.com/pytorch/pytorch/pull/106004), [Provide a refined upper bound for nonzero when original numel is static](https://github.com/pytorch/pytorch/pull/105843) - **An initial plan for KeyedJaggedTensor.** After studying some of the models that use KJT and trying to get export working on them, here are some of the initial findings: - You can remove the list of integers from KJT before tracing a model, which will cause the model to perform a data-dependent access to populate these integers as unbacked integers. However, when we try to use these integers to do a tensor\_split, we immediately hit guards we cannot prove. The guards should be provable via sum(lengths) == values.shape[0] but our symbolic reasoning is not strong enough. These guards are for errors, so they should be bypassable by irrefutable guards (guards which, if they fail, imply you would have errored anyway. In this case you can convert the guard into a runtime test.) This is worth pursuing further. In any case, you expect to have 500 unbacked symints, symbolic reasoning must be fast enough to deal with it. - If you don’t remove the list of integers, you need some way to prevent them from 0/1 specializing. In export, you can simply require every sparse feature be populated to size 2 and hope it generalizes to 0/1. In eager, we probably will just have to specialize KJT to treat these integers specially. Big benefit to this strategy is you’re not hard-blocked on guards on unbacked SymInts, since there’s always a hint; don’t need any sum(lengths) reasoning since guards are discharged by checking the underlying values. Cannot actually do this in export because export does not support SymInt inputs–I plan to fix this. - Export with KJTs doesn’t work because KJTs are not a supported input. Direct fix [Add pytree support to KeyedJaggedTensor by ezyang · Pull Request #1287 · meta-pytorch/torchrec · GitHub](https://github.com/pytorch/torchrec/pull/1287); indirect fix is rewriting the export calling convention from pytree specs to a dictionary of “FQN” (Source.name()) really to Tensor. In that case, to pass a KJT named `id_list_features`, you would actually pass three tensors, `id_list_features._values`, etc. - More details at [Meta-only doc](https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#heading=h.nylqo2863cgh) (sorry, non-public due to details about Meta prod models). - **Translation validation bisection.** We had a case of hint disagreeing with sympy simplification in [internal](https://fb.workplace.com/groups/1075192433118967/posts/1279281309376744); we’ve also had instances of this in open source, see [[https://github.com/pytorch/pytorch/pull/101173](https://github.com/pytorch/pytorch/pull/101173)](integer and real equality). Yukio is thinking of implementing a bisection mechanism for translation validation, so we can find the first guard that actually caused a correctness problem. - **Export for QAT.** QAT wants to do whole-graph transformations on a pre-autograd FX graph. Export sort of supports this with `pre_dispatch` export. What is likely going to happen is this turns into the IR format that export is going to use. Pre-autograd functionalization is unlikely to happen; you only get some mild normalization. Still unresolved how to work this into the overall QAT workflow API, since export isn’t really keen on exposing this mid-point IR (which is kind of incoherent.) - **Notable bug fixes.** - [Change \_dynamo.export to be export(f)(\*args, \*\*kwargs)](https://github.com/pytorch/pytorch/pull/106109) and [Change \_dynamo.explain to be explain(f)(\*args, \*\*kwargs)](https://github.com/pytorch/pytorch/pull/106066) helps avoid ambiguity between user kwargs and export/explain kwargs. It is technically BC-breaking, when you exported a module with no arguments (quite rare!) - [Turn on capture\_dynamic\_output\_shape\_ops/capture\_scalar\_outputs by default for export](https://github.com/pytorch/pytorch/pull/105962). Not sure why we hadn’t done this before… - [Make \_CURRENT\_TRACING\_CONTEXT thread local](https://github.com/pytorch/pytorch/pull/105942). This occasionally caused a race that typically looked like “fake tensor mode mismatch.” - [Improve FakeTensor to work with mixed meta-cpu embedding bag arguments](https://github.com/pytorch/pytorch/pull/105924). This is for you reco system peeps using meta embedding tables with CPU inputs for testing. - [Tweak dynamic=False behavior](https://github.com/pytorch/pytorch/pull/105715) is in. - [Add missing evaluate\_expr for slice\_scatter, slight refactor](https://github.com/pytorch/pytorch/pull/105714); fixes slice\_scatter with SymInt start/end - [Support dynamic shapes in TritonTemplates by ipiszy · Pull Request #105295 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/105295) - responsible for decent TIMM improvement - **Notable new bugs.** - [[dynamo.export] symbolic\_shapes.GuardOnDataDependentSymNode](https://github.com/pytorch/pytorch/issues/106183) - lively discussion about irrefutable guards - [llama model failed for dynamic shape path](https://github.com/pytorch/pytorch/issues/106110) - this is cpu backend specifically - [Tensors always get 0/1 specialization guards, even if they’re not used](https://github.com/pytorch/pytorch/issues/106067) - discovered by Animesh - [GitHub · Where software is built](https://github.com/pytorch/pytorch/issues/105853) - not really sure what’s going on with this one **CI skips.** -3, -1, -1, -2 (no change). **Training dashboard (as of 1da4115702).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2015%20Jul%202023%2021%3A23%3A18%20GMT&stopTime=Sat%2C%2029%20Jul%202023%2021%3A23%3A18%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=1da41157028ee8224e456f6fab18bc22fa2637fe&rBranch=main&rCommit=0ad93a3d5684c2026bda8ff4ab7c72c6596a225b) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 92%, 59/64 | 96%, 44/46 | 98%, 59/60 | 100%, 8/8 | | Speedup | 1.54x → 1.56x | 1.69x | 1.28x → 1.35x | 1.97x → 2.04x | | Comptime | 81s | 107s → 108s | 142s | 38s → 39s | | Memory | 0.79x | 0.96x | 1.01x | 0.69x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/c57e925b073080140da113a66d6bff2736e6d31d.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/aaa34c84fffa7d8daf2f078a80d15714924de492.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/80217dab5c6f64e2c716390b2d960de5fd6c422b.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/80217dab5c6f64e2c716390b2d960de5fd6c422b.png) - TIMM models had the most change. Some of this is from cait\_m36\_384 which had its batch size changed. Some others are from [Support dynamic shapes in TritonTemplates by ipiszy · Pull Request #105295 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/105295) (ghostnet\_100). Some are across the board improvements (e.g., resmlp\_12\_224) **Inference dashboard (as of 1da4115702).** [This week on HUD](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2015%20Jul%202023%2021%3A34%3A00%20GMT&stopTime=Sat%2C%2029%20Jul%202023%2021%3A34%3A00%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=1da41157028ee8224e456f6fab18bc22fa2637fe&rBranch=main&rCommit=0ad93a3d5684c2026bda8ff4ab7c72c6596a225b) | Metric | Torchbench | Huggingface | TIMM models | Dynamic | | --- | --- | --- | --- | --- | | Passrate | 88%, 65/74 | 98%, 45/46 | 100%, 60/60 | 58%, 7/12 | | Speedup | 1.55x → 1.54x | 1.78x → 1.77x | 1.79x → 1.80x | 3.03x → 3.08x | | Comptime | 35s → 36s | 44s → 45s | 36s | 72s → 75s | | Memory | 0.68x | 1.11x | 0.84x → 0.85x | 0.87x | ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/c00fa8eccd2755d6ccb653ecdc6f0f6499b754f7.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/9/9031f0475feb2d88fc892c0000f3a0294843e941.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a0343ca4c70952f999d45ad171e93e401978f277.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/25b67fcfe75fd94ef3c7f23bcea548216c0c9e6d.png) Looks all within noise. ## What’s next - Rewriting export input/output spec flattening - Irrefutable guards - Generally more pushing on KJT stuff ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [August 7, 2023, 3:52am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/66 "2023-08-07T03:52:18Z")
# State of symbolic shapes: Aug 6, 2023 edition Previous update: [State of symbolic shapes branch - #65 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/65#state-of-symbolic-shapes-jul-29-2023-edition-1) ## Executive summary - **More on KJT/torchrec.** I had a nice discussion with Dennis van der Staay about torchrec and work on sparse arch. Some new information: (1) this workstream is almost certainly going to involve distributed later, because applying PT2 to post-torchrec sharded models is going to involve tracing past communication primitives–this also implies I’m going to want to get FakePG working on torchrec, (2) working on unit tests should be a pretty good idea, but there’s still some basic infra work to do (laid out last week), (3) not really expecting concrete performance improvements as sparse arch is going to be typically communication bound, so this is a mostly “we think this is promising, and the investment is not too big, because we’ve already done so much with dynamic shapes so far.”) - **Pre-dispatch export.** We’ve agreed to allow QAT to short-term publish a new export interface that produces a pre-dispatch FX graph with ATen operators which is suitable for graph transformations and training. The long term goal will to be have pre-dispatch functionalization which is the invariant the export team wants to allow this to be worked into torch.export proper. Pre-dispatch will generate an ExportedModule so that the APIs match. - **Fake export.** Export now supports exporting entirely fake modules/inputs. This means to export a model you don’t have to actually load its weights into memory; you can load it in a fake mode and still export it. This means we have some delicate code in Dynamo for dealing with two concurrent fake modes (but it’s not so bad: the outer fake mode is typically disabled while we do Dynamo analysis.) Only ONNX supports torch.load’ing models in fake mode at the moment. - **Improved user stacks in Dynamo.** `torch._guards.TracingContext.extract_stack()` now always accurately reports a user stack from anywhere in Dynamo, and we reliably use it for reporting real stacks for exceptions (previously, they used an entirely different mechanism.) - **Improved error messages for non-local inputs in export.** See [Improve error message when export encounters non-local input](https://github.com/pytorch/pytorch/pull/106403) for the details. This isn’t complete; follow through is also to make this work for outputs, and also work a little harder with the pytree representation (probably this week.) - **Dynamo change in attitude.** Many folks are concerned that Dynamo is just “endless” bugs. I pitched Animesh and Voz on a new attitude to fixing Dynamo bugs, which is that we should imagine the platonic ideal implementation of Dynamo as a faithful reimplementation of CPython in Python. Then, fixing a bug should not just be moving code around to fix a particular problem, but instead improving the local vicinity of code to bring it closer in line to this ideal. An example I used a lot explaining this was dict.keys support (bug fix is changing its type from tuple to set; real fix is to accurately model dict views.) To do this well, you need to regularly look at CPython code, and Dynamo may need to grow some new abstractions (perhaps a proper implementation of Python’s object model, Python traceable polyfills). - **Notable new bugs.** - [Case study of torch.compile / cpp inductor on CPU: min\_sum / mul\_sum with 1d / matmul-like with static / dynamic shapes](https://github.com/pytorch/pytorch/issues/106614) - one takeaway is that it’s difficult to compile an operator size chunk of code have exercise fine grained control on what dimensions should be dynamic/static (due to automatic dynamic) - [[dynamo] Unsupported to trace through Boolean Tensor indexing](https://github.com/pytorch/pytorch/issues/106580) ## Numbers As we’re not really doing much on performance numbers recently, I am simplifying this section. **Training.** [68cb854d73](https://hud.pytorch.org/pytorch/pytorch/commit/68cb854d73458a14684d584c25c22b17eb79dfca#inductor-a100-perf-nightly) [Dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2024%20Jul%202023%2003%3A44%3A56%20GMT&stopTime=Mon%2C%2007%20Aug%202023%2003%3A44%3A56%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=68cb854d73458a14684d584c25c22b17eb79dfca&rBranch=main&rCommit=1da41157028ee8224e456f6fab18bc22fa2637fe) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a29db494139c873275e7aa1563984152d06f6a09.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/00c2ef992e44b6af7550134f36ef88132776c2a4.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/5/581e11407f40612b1fb7aeaecafff4b630a23cf4.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/60ec42751776416ca70da1e2e640a9387298876e.png) Nothing much to report. **Inference.** [68cb854d73](https://hud.pytorch.org/pytorch/pytorch/commit/68cb854d73458a14684d584c25c22b17eb79dfca#inductor-a100-perf-nightly) [Dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Mon%2C%2024%20Jul%202023%2003%3A44%3A56%20GMT&stopTime=Mon%2C%2007%20Aug%202023%2003%3A44%3A56%20GMT&granularity=hour&suite=timm_models&mode=inference&dtype=bfloat16&lBranch=main&lCommit=68cb854d73458a14684d584c25c22b17eb79dfca&rBranch=main&rCommit=1da41157028ee8224e456f6fab18bc22fa2637fe) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/3/3fa8016cc0d8e0b083398e4fe48ae7163d0ddf51.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/8385bff8580c00b7732fccbd947a468daa71c8bf.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/3/326d1dd67a9d889009eadbc36b7d335260a8258e.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/23968553622d1883b403000e4c74ca6094537426.png) The big perf increase in torchbench is due to maml getting removed from the benchmark set (it slows down a lot under PT2 and was depressing the score). clip, hf\_Whisper, llama\_v2 are new models added thanks to @msaroufim ! ## What’s next? There are a lot of things that need doing - Finish overhauling export input/output pytree matching (probably not dumping the pytree in/out spec, but I think if we tree\_map into positional identifiers we can reliably detect KJT missing situations) - Make unbacked SymInts work in Inductor [gist:1293a41299604c44310341b7540eabcb · GitHub](https://gist.github.com/ezyang/1293a41299604c44310341b7540eabcb) (biggest problem is unbacked SymInt binding in wrapper codegen and the hinting logic) - Irrefutable guards - Write up the plan for sparse arch / KJT - Land pytree support for KJT/JT - 0/1 specialization suppression for list of int in KJT Stuff that probably can wait until later? - Host side torch.cond - DynTensor ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [August 12, 2023, 2:35pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/67 "2023-08-12T14:35:22Z")
# State of symbolic shapes: Aug 12, 2023 edition Previous update: [State of symbolic shapes branch - #66 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/66#state-of-symbolic-shapes-aug-6-2023-edition-1) ## Executive summary I’m trying something a little different, expanding the update to cover a wider variety of topics beyond dynamic shapes, mostly centered around things that I personally have involvement in (this is a lot of things, so you should be getting pretty good coverage this way!) ### Benchmarking - **Inductor CI/perf is upgraded to CUDA 12 / gcc 9.** This doesn’t seem to have any appreciable effect on perf, but we did it so we could do the next item. - **torchrec\_dlrm is back.** They were disabled a few months ago because of fbgemm nightly related flakiness. The flakiness has been resolved by building fbgemm/torchrec from source in the Docker image. These are now installed as part of the general torchbench installs, and should help some of the work we are doing on jagged tensors (since many important operators are currently implemented in fbgemm). - **Algorithmic efficiency.** Frank Schneider posted about how [PyTorch was slower than JAX](https://discuss.pytorch.org/t/struggling-to-get-pytorch-fast-enough-to-use-in-public-competition/186015) in their upcoming algorithmic-efficiency benchmark suite. A bunch of us, spearheaded by @msaroufim, jumped in to take a look at what was going on. Status updates at [https://docs.google.com/document/d/1okqKS32b0EhWQSFFoSV6IjGlYM4VhNYdxBPjdlFIw5w/edit](https://docs.google.com/document/d/1okqKS32b0EhWQSFFoSV6IjGlYM4VhNYdxBPjdlFIw5w/edit) (Meta-only). I personally have an interest in the dlrm side of things, since I’ve been working on sparse arch recently; after fixing some mild bugs, I was able to show parity on criteo1tb dlrm between PyTorch nightly and JAX on an A100x8 (PyTorch score: 7703.403180360794, JAX score: 7703.041719198227), although the number of evals varied, so I’m not sure if this a threat to validity. Unfortunately, this does not necessarily help their problem, which was an OOM. To make further progress on this, we may need some tools to help us understand why torch.compile memory usage is higher. Export - **Pre-dispatch export, part 2.** We had more discussion about pre-dispatch export in the Friday export meeting. @suo in particular was arguing that from a frontend perspective, it would make more sense to export pre-dispatch IR by default, and have the further post-dispatch lowerings be an extra pass on top that is opt-in by backends. One of the identified barriers to doing this is pre dispatch functionalization; the other is nondifferentiable decomps. nkaretnikov is going to take a look at `core_aten_decompositions` to see which of these are differentiable and which are not. In other news, torch.export is going platinum [https://github.com/pytorch/pytorch/pull/106904/](https://github.com/pytorch/pytorch/pull/106904/) - **dim order coming to Tensor.** We probably should have added this API a long time ago, but export really wants this on Tensor so in it goes. [https://github.com/pytorch/pytorch/pull/106835](https://github.com/pytorch/pytorch/pull/106835) Distributed - **Tracing FSDP.** @voz wrote a post [https://fb.workplace.com/groups/2917693311835451/](https://fb.workplace.com/groups/2917693311835451/) (Meta-only) about the state of tracing FSDP in Dynamo. The key info is that on a branch, he can trace everything through and get identical results on a single forward-backward to eager. There’s a lot of fixes that need to land to main; from his post: 1. The value of various small changes to FSDP to make this work vs adding fixes in dynamo (Pretty easy, preferring dynamo ofc but for some mostly no op shuffling, we do FSDP as well) 2. TypedStorage - is it tensor-like/tensor-associated enough to go in the graph? Do we need to add some ops for doing tensor typed storage data ptr comparison / checking free, etc? 3. Working through the cudastream story, in particular around wait\_stream and such 4. Lot’s of little bug fixes here and there 5. Coverage for missing comparisons, bytecode ops, general coverage gaps like attr access on FSDP modules, setting data on a tensor, etc. - **pytrees slow again for DTensor.** Junjie and Rodrigo have been trying to improve DTensor’s eager perf, and we spent the first half of composability sync talking about it. Rodrigo had a hack to pre-compile pytree applications into Python code but apparently this doesn’t help that much: [gist:5427cabfab6421d4e104905345f94a50 · GitHub](https://gist.github.com/kumpera/5427cabfab6421d4e104905345f94a50) . Another suggestion from the meeting was that after Brian’s subclass supports lands, maybe you could torch.compile each op individually with backend=“eager”. - **Data-dependent all2all.** Will Feng got all2all collective working in inductor [Add functional collective all\_to\_all\_single and support it in Inductor by yf225 · Pull Request #106655 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/106655/) This is notable because all2all collective has data-dependent output shape. It looks like unbacked symints worked here! Custom ops - **Custom ops.** Richard tells me he is going to add a class-based API for custom ops, to make it easier to define everything all in one place. More on this soon I assume! - **Custom op testing.** [https://github.com/pytorch/pytorch/pull/106903](https://github.com/pytorch/pytorch/pull/106903) is here to make it easier to retrofit pre-existing test suites to also test for important operator properties. Nested/jagged tensor - **SkolemSymNodeImpl.** @jw3468 is going to make size() work on jagged tensor by introducing a new concept to SymInt provisionally called SkolemSymNodeImpl. This is a special SymInt which is not symbolic (it can show up in eager mode) but only compares equal to itself (aka is a skolem variable). We will use this to represent jagged dimensions. All jagged tensors that have the same offsets tensor get assigned the same skolem variable, if you have different offsets tensors you can’t add them together because their skolem variables don’t match. More details at [https://docs.google.com/document/d/1e-R\_818YA4VlVTlozu5eyzRIV6TzyvSPDm9DMEw\_4xg/edit](https://docs.google.com/document/d/1e-R_818YA4VlVTlozu5eyzRIV6TzyvSPDm9DMEw_4xg/edit) (Meta-only) - **SAM single batch, vmap for nested tensor.** @jbschlosser has been working on integrating nested tensor with SAM, and one challenge with SAM is that it is written in a single-batch style, so the first problem is batchifying the model in the first place. Last week, an idea was to use vmap to automatically convert single-batch to multi-batch, and there is a PoC for this [Initial vmap + NT support with unbind fallback by jbschlosser · Pull Request #106786 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/106786) but there are still a number of spots in SAM which are not so easy to vmap across [https://docs.google.com/document/d/1\_yiHOBbaI4qFWqBfebjWPHOhkxKW3v-CHu3lj5apv1Y/edit](https://docs.google.com/document/d/1_yiHOBbaI4qFWqBfebjWPHOhkxKW3v-CHu3lj5apv1Y/edit) . Joel is going to try a few more days on this, and then pivot if it is still not looking promising. Dynamo - **Pivot on per-NN module caching.** @anijain2305 is working on having a separate code cache per NN module, but on Friday with the help of @voz we realized that you actually the problem is separable into two pieces: (1) an enhanced cache size limit policy that knows about NN modules [[RFC][dynamo] Separate cache sizes for nn module guard specialization by anijain2305 · Pull Request #107077 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/107077) and (2) improvements to cache lookup when there are a lot of cache entries (guard trees). - **Dynamo eager mode cond.** Yidi Wu: to support cond in eager mode, we plan to torch.compile the entire cond operator, manufacturing fresh code objects to ensure that the caches don’t interfere with each other. [https://docs.google.com/document/d/1esmHEa0fiktiSw1lvRsPmsbnTYxDSc0t3V9V5V0xK7I/edit#heading=h.pajqpbewbdg7](https://docs.google.com/document/d/1esmHEa0fiktiSw1lvRsPmsbnTYxDSc0t3V9V5V0xK7I/edit#heading=h.pajqpbewbdg7) (Meta-only) - **Time to get rid of functional VariableTracker?** VariableTracker in Dynamo is an immutable data structure: when a mutation happens, you allocate a fresh VariableTracker and then replace old VariableTrackers with the new one. This is because we have checkpointing functionality that is used to rewind old VariableTracker. However, this is a bit of pain from the modeling side, as every Python data structure has to be reimplemented to have purely functional operations. An alternate design is to allow direct mutation of VariableTrackers. To do checkpoints, we simply restart Dynamo analysis to “go back in time” by stopping execution at the point where we would have checkpointed (a deepcopy could also work, though I’m not a fan.) Speculate subgraph would be implemented by simply denying all mutations or doing some crazy thermometer continuation thing. This would help make Dynamo more metacircular and reduce the work needed to support new container types, of which we often need to support a lot. Dynamic shapes - **expect\_true irrefutable guards.** I talked through this in the last 20min of composability sync. Check [https://github.com/pytorch/pytorch/pull/106720](https://github.com/pytorch/pytorch/pull/106720) ; this is enough to make splits on unbacked SymInts work. - **Boolean masking, at last.** @yanboliang is looking into a pre-autograd FX transform that replaces boolean mask updates with torch.where calls. One annoying detail is how to deal with Dynamo tracing the boolean masks in the first place, when Inductor can’t deal with boolean masks if you can’t eliminate them? Our idea, in lieu of fixing Inductor to work with data-dependent shapes (which we are working on), is to attempt to eliminate all data-dependent ops in a pre-dispatch pass, and if it is not possible, restart Dynamo analysis saying “you need to graph break on this op next time.” - **Notable fixes.** - [SymInt’ify tile](https://github.com/pytorch/pytorch/pull/106933). This one needed for algorithmic-efficiency criteo1tb dlrm. - [[export] Refactor `constrain_as_value` and `constrain_as_size`](https://github.com/pytorch/pytorch/pull/106591) from Tugsuu (was bounced, needs relanding) - **Notable new bugs.** - [Dynamic shapes support for inductor foreach codegen](https://github.com/pytorch/pytorch/issues/107005) ## Numbers **Training.** [03414081ff Dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2005%20Aug%202023%2014%3A07%3A39%20GMT&stopTime=Sat%2C%2012%20Aug%202023%2014%3A07%3A39%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=03414081ff7ee011e17ee10f9ddb2584811bf965&rBranch=main&rCommit=68cb854d73458a14684d584c25c22b17eb79dfca) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/f3b0f267afe115a4cbb2a4d6a1e9f3174dcf1c50.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/68c3b5987f411b177a827591e0cfdab94d6dd8d5.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/0c83712103daf74995fa4efc94dfe2287be137aa.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/61072ca42b3020dc688583bce40ebae4d36c7d55.png) - Some accuracy regressions. torchbench: hf\_BigBird, vision\_maskrcnn (flaky). It’s not clear what broke hf\_BigBird; possibly the CUDA 12 upgrade. Need to investigate. AlbertForQuestionAnswering improved accuracy! - The huge perf improvement across the board is thanks to Peter Bell’s work [https://github.com/pytorch/pytorch/pull/106747](https://github.com/pytorch/pytorch/pull/106747) optimizing split reductions. This is not full runtime split reductions: instead Peter uses whatever the hint was at the time we compiled to plan the split reduction, and then we use it for all subsequent runs. This makes it more important to warm up Inductor with the “right” size hint to start; see also [Padded tensor subclass · Issue #105325 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/105325) ; there was also another user complaining about other cases where we made suboptimal decisions if the first kernel we compiled with wasn’t representative **Inference.** [Dashboard 03414081ff](https://hud.pytorch.org/benchmark/compilers?startTime=Sat%2C%2005%20Aug%202023%2014%3A28%3A16%20GMT&stopTime=Sat%2C%2012%20Aug%202023%2014%3A28%3A16%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=03414081ff7ee011e17ee10f9ddb2584811bf965&rBranch=main&rCommit=68cb854d73458a14684d584c25c22b17eb79dfca) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/d7cbe6667186dc52892519960ad1008afba575dd.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/9/9730476c1da0b52a94c939493f83217802171895.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/c37eb9228a03de943c546fc5e0e8c80570fb854f.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/1d36846ba4432e327d97b710392fa0cb43672c78.png) - A lot of change on the last day; some improvements and some regressions (but mostly regressions). Maybe CUDA 12 update related, need to check. hf\_BigBird also failing here too. RobertaForQuestionAnswering failing accuracy now ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [August 20, 2023, 11:55pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/68 "2023-08-20T23:55:35Z")
# State of PT2: Aug 20, 2023 edition Previous update: [State of symbolic shapes branch - #66 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/66#state-of-symbolic-shapes-aug-12-2023-edition-1) ## Executive summary Public service announcements - Trying to understand how to read PT2 logs? Check out [Logging docs - Google Docs](https://docs.google.com/document/d/1kpfPApUuwbfmN5mxKTw515LiZ02mmrMVS-WOnQgdiz0/edit#heading=h.e61gzbfwl1gu) for some quick orientation. (In other news, guards logging this week has been improved to show you which line of user code caused a guard to be added! Take a look and don’t be afraid to give feedback on it.) - Have you ever wanted to store lots of tracebacks for logging/debugging purposes, but were afraid to do so by default because it might be too expensive to do so? There is a new class in torch.utils.\_traceback called CapturedTraceback which makes it extremely fast to save a Python traceback (something like 20x faster than running a full `traceback.extract_stack()`), so it should change the calculation about whether or not you are willing to store tracebacks by default. We have already used this to keep fine-grained information about guard provenance to start. Note that CapturedTraceback DOES hold references to code objects, and these references can cause cycles (because of [co\_extra](https://github.com/pytorch/pytorch/issues/107469)), so good practice is to make sure you clear these traces once you know you no longer need them. - I spent some time debugging reference cycles this week (due to CapturedTraceback), and Alban pointed me at [objgraph](https://mg.pov.lt/objgraph/) for visualizing references/referents. It’s pretty cool, you should definitely check it out. Composability sync [https://www.youtube.com/watch?v=LmkFkOBwhks](https://www.youtube.com/watch?v=LmkFkOBwhks) - About modeling quantized dtypes: [https://www.threads.net/@ezyang00/post/CwFx1AEgXH4/?igshid=NTc4MTIwNjQ2YQ==](https://www.threads.net/@ezyang00/post/CwFx1AEgXH4/?igshid=NTc4MTIwNjQ2YQ==) - Brian Hirsh presented the steps to pre-dispatch functionalization [pre-dispatch functionalization - Google Docs](https://docs.google.com/document/d/1Ya1GW_8ErRDy6yPL91WOCSnpAvfqOJy5nxueRXEMONI/edit#heading=h.h1bbyxvt0r56) (Meta-only) Inside baseball - We are continuing to do a terrible job at not causing reference cycles in our compiler data structures. Folks have noticed that we [leak compiled models](https://github.com/pytorch/pytorch/issues/104095) even when the model objects are deleted. This generally emerges when a reference cycle goes through a non traversable object (C++ shared references or co\_extra on code objects); the result cannot be deallocated unless we explicitly break the reference cycle, which we generally do not do. Animesh is going to take a whack at the model finalization problem. On the bright side, Animesh did push a fix for a long standing bug [[dynamo][eval\_frame] Set destroy\_extra\_state deleter as part of co\_extra by anijain2305 · Pull Request #107117 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/107117) related to code object deallocation (admittedly rare, but it can happen); - Yidi Wu wanted to know if we could guard against backends changing [https://github.com/pytorch/pytorch/pull/107337](https://github.com/pytorch/pytorch/pull/107337) so you don’t have to call dynamo.reset() anymore. The motivation was to allow people to seamlessly use eager backend along side their compiled backend. Difficult! - Efficient transformer inference needs in-place scatter, being worked on at [https://github.com/pytorch/pytorch/pull/106192](https://github.com/pytorch/pytorch/pull/106192) This is not so easy to do because we generally don’t like dealing with mutation in Inductor but Horace thinks he has a plan. - When doing passes on FX graphs, it can be annoying to keep fake tensor metadata up-to-date. Horace is looking into some incremental rebuilding of the metadata, stopping re-propagation once you notice that the fake tensor lines up with the old values. Distributed - Handling backward hooks in Dynamo is kind of difficult. There is a discontinuity between hooks on inputs and hooks on intermediates; hooks on intermediates, in particular, have to somehow be reflected in whatever graph gets differentiated by autograd, but at the same time these hooks may have arbitrary Python bits that need handling by Dynamo. It seems the problem is made easier if we have combined forward-backward tracing in Dynamo, at which Dynamo knows enough about the backward structure to bypass AOTAutograd entirely. It might also be possible to just do cuts prior to going to AOTAutograd, which will impede optimization. It might be possible to bypass this problem for FSDP if hooks are only on parameters and outputs. Lots of difficulties… Dynamic shapes - KJT proddy stuff - Notable new dynamic shapes bugs: - a pile of cpu perf regressions somehow [[Inductor][cpu][dynamic shapes] some models perf regression on 2023\_08\_13 nightly release](https://github.com/pytorch/pytorch/issues/107364) [[Inductor][cpu][dynamic shapes] some models perf regression on 2023\_08\_14 nightly release](https://github.com/pytorch/pytorch/issues/107361) - [[inductor] [dynamic shape] 5 HF models fails with `Constraints violated` using transformers v4.31.0](https://github.com/pytorch/pytorch/issues/107200) - because HF did something specialization unfriendly - [Torch randn cannot take symbol shapes as shape argument.](https://github.com/pytorch/pytorch/issues/107170) - Some optimizer stuff [[Dynamo] Unable to Trace AdamW Optimizer when there is LR Scheduler](https://github.com/pytorch/pytorch/issues/107076) [Optimizers should use learning rates passed as tensors directly](https://github.com/pytorch/pytorch/issues/106802) # Numbers **Training.** [68b9bf9671 Dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Sun%2C%2006%20Aug%202023%2020%3A20%3A41%20GMT&stopTime=Sun%2C%2020%20Aug%202023%2020%3A20%3A41%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=68b9bf9671ffa4b580236f5bd436ce6c38d69a96&rBranch=main&rCommit=03414081ff7ee011e17ee10f9ddb2584811bf965) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/6df285e01966447a7d33be55e757ba400935a8cb.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/9/9ecc4a0da95792a81cd2f62ebfdea78d7b8fe98e.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/25cdf7431142df1cf6d988ddf961bca558a2a8a1.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/9/9fee986637f8a6ea9674473a73a99ab2f9a4ca18.png) Not too much action this week. However, a bit of flakiness on the inside; may need some follow up. torchrec\_dlrm added last week has shown up with dynamic shapes but is failing, so it still needs work. Early week TIMM improvement is from [[inductor] make thread order consistent with loop order by shunting314 · Pull Request #106827 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/106827) . Some of later week TIMM improvement may be from [https://github.com/pytorch/pytorch/pull/106911](https://github.com/pytorch/pytorch/pull/106911) (but stats were not run on that PR.) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/233a15b3d256039a6e89b8470b03b7004f5826c9.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/89b25dfebc56239237a172b5530b8c7ef6efadf4.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/09acf9abb9b3bca676be893f4cb95f8c4e8634f8.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/81bfe9bfe68b18c2680bc29c805c32fa7b8cf4a0.png) dlrm is now passing on dynamic shapes which is cool. RobertaForQuestionAnswering was fixed (not clear what fixed this; it’s in the 35cca799ff42182a1b7f1ee4d0225ee879b7c924..384e0d104fd077d31efafc564129660e9b7a0f25 range). Some other wins (and some regressions, most importantly sam) from [Unfuse bias add before pointwise ops by eellison · Pull Request #106912 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/106912), also some other unexplained changes like convnext\_base and jx\_next\_base in this same commit range (which sort of makes sense, @eellison landed a bunch of perf related changes) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [September 10, 2023, 7:10pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/69 "2023-09-10T19:10:36Z")
# State of PT2: Sep 8, 2023 edition Previous update: [State of symbolic shapes branch - #67 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/67#state-of-pt2-aug-20-2023-edition-1) We were on break for two weeks because I went on vacation, and I didn’t have time to do a report before/after vacation lol. ## Executive summary - **PyTorch 2.1 branch cut.** The cut was three weeks go (right when I went on vacation lol) and we’re reaching the end of the cherry-pick window. Track ongoing cherry picks at: [[v.2.1.0] Release Tracker · Issue #108055 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108055) - **Blueberries offsite was this week!** The blueberries workstream is focused on accelerating SOTA transformer models using PT2, quantization, sparsity and other techniques. Some highlights: [MFU is coming to the benchmark suite](https://github.com/pytorch/test-infra/pull/4558), some direct improvements to important models, int8 dynamic quantization with tensor subclasses. Many of these are not published yet, keep your eyes peeled at PTC! - **PyTorch conference registration** filling up fast. If you want to go and haven’t registered yet, you should register at [PyTorch Conference | LF Events](https://events.linuxfoundation.org/pytorch-conference/) Composability sync - Aug 24 [https://www.youtube.com/watch?v=H6EUSsvDmbw](https://www.youtube.com/watch?v=H6EUSsvDmbw) - we spent time going over recent KJT progress (to be reduxed below), and Voz reported progress on tracing FSDP with hooks (also to be reduxed below) - Aug 31 - not livestreamed publicly, I wasn’t there, but apparently there was some discussion about streams for tracing FSDP (no minutes alas) Distributed and PT2 - **Tracing FSDP** with Voz is deep in the weeds on backwards hooks support. We are attempting to implement hooks in a way that doesn’t require consolidated forward-backwards. The general strategy is (1) have Dynamo emit graphs that have `register_hook` calls on intermediates (`register_hook` calls on inputs must not go in the graph, they have to happen as part of residuals), (2) write these `register_hook` calls in such a way that when AOTAutograd runs, the actual hook code (which is arbitrary Python code and is not safe to run in tracing) is not run, but instead we run a meta function (which performs any needed metadata mutation) and then insert a call function to the original Python function (which will show up in backwards), (3) have compiled backwards take care of compiling this call function in the end. - **Per parameter FSDP** is looking pretty legit. Andrew Gu has been looking at the performance of per-parameter sharding (where parameters managed by FSDP aren’t shoved into a single flat buffer) and has found that we only really pay a penalty of 5% with per-parameter sharding but get better memory usage. Meta only: [https://fb.workplace.com/notes/618571790386318](https://fb.workplace.com/notes/618571790386318) - **DDP optimizer brittleness.** We currently support pipelining DDP code with PT2 by manually splitting graphs into multiple AOTAutograd functions so that backwards isn’t run too soon. The code here is kind of janky: I ran into two separate bugs that only happend when `optimize_ddp` was on: [[DDP PT2] TypeError: convert\_frame\_assert.\.\_convert\_frame\_assert() missing 2 required positional arguments: 'hooks' and 'frame\_state' · Issue #107637 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/107637) and [[optimize\_ddp] moco - NameError: name 's2' is not defined · Issue #108877 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108877) . Pritam has also been complaining about the graph break strategy: [torch.compile graph breaks should be independent of DDP buckets · Issue #108966 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108966) Will tells me that Chien-Chin is working on some new DDP strategy, but it appears to be centered around starting with a non-parallelized graph. Hopefully we can present it at composability this week. Note that DDP cannot be easily traced as it is implemented in C++. Dynamic shapes - Avik is proposing a change to the `dynamic_dim` API currently used to express dynamism in export API. Instead, they will adopt a Python typing generics style solution, where you bind generic variables for dynamic dimensions `batch = Dim("batch", max=64)` and then use this to annotate types on input tensors `x: TensorType[batch, K, N]`. This is very reminiscent of [[discussion] Expressing tensor dimension semantics / constraints through typing / constraints blocks. Constraints block could be scripted/traced and help for tracing/script execution and codegen · Issue #40373 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/40373) Meta only: [https://docs.google.com/presentation/d/168U7XK72C\_WSsZpGESP6Cho9udh193fi0gfjxCNcJ4E/edit](https://docs.google.com/presentation/d/168U7XK72C_WSsZpGESP6Cho9udh193fi0gfjxCNcJ4E/edit) - This is not really PT2 related, but there’s an interesting set of posts about the future of Sympy circulating around: [Towards a new SymPy: part 1 - Outline — blog documentation](https://oscarbenjamin.github.io/blog/czi/post1.html) Funnily enough, the part of Sympy which Oscar calls out as “overused” (the symbolic expression system) is precisely the part we actually care about. Maybe a good reason for us to figure out some way to note use this part (me, personally, I want a compact representation and hash consing.) - I discussed this in a bit of detail in composability three weeks ago, but work on supporting fine-grained KJTs is going very well. This week, I worked with Michael Suo to get APS sparse arch tracing all the way through. I managed to get it going all the way through (though it failed on some seemingly unrelated problem.) So fine-grained tracing definitely seems like it will work, even if we generate tons of crappy guards. My plan for next week is to make a serious attempt at tracing multi-node model parallel sharded torchrec\_dlrm. - This week, when I had spare time in the offsites, I worked on fixing a few one-off bugs. There were several that were pretty easy to nail: - [Don’t fastpath conj copy when conj/neg bit mismatch](https://github.com/pytorch/pytorch/pull/108881) - [Fix setitem with SymInt](https://github.com/pytorch/pytorch/pull/108873) - [Add support for symbolic repeat\_interleave](https://github.com/pytorch/pytorch/pull/108763) - [Add torch.\_check\_is\_size](https://github.com/pytorch/pytorch/pull/108685) - [Meta implementation for nms by ezyang · Pull Request #7944 · pytorch/vision · GitHub](https://github.com/pytorch/vision/pull/7944) - [Avoid creating a tensor of shape when not tracing by ezyang · Pull Request #7942 · pytorch/vision · GitHub](https://github.com/pytorch/vision/pull/7942) - While working on the meta implementation for nms I played around with Richard Zou’s opcheck testing: [Add `generate_opcheck_tests`, a PT2 crossref testing mechanism by zou3519 · Pull Request #106903 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/106903) It still needs some improvements ([Run only one pytest parametrization when generating optest by ezyang · Pull Request #108936 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/108936) [Make mutation test work with quantized tensors by ezyang · Pull Request #108935 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/108935) [optests improvements based on torchvision usage on nms by ezyang · Pull Request #108929 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/108929)) but I was able to get it to work end-to-end. Seems pretty promising! Inductor fun - Peter Bell is very close to landing inductor IR support for scan [https://github.com/pytorch/pytorch/pull/106581](https://github.com/pytorch/pytorch/pull/106581) which allows for native cumsum/cumprod support. Now all we need is for someone to add a higher order op that feeds into this and we will have torch.scan! - Someone should add a “realize” operator to PT2, which would force materializing a tensor rather than allowing fusions across it. Christian Puhrsch would find this useful for ensuring epilogue fusion occurs on int8 mm (today, regular fusion causes the pointwise operation to get fused into a later reduction, instead of fusing the pointwise into the matmul) - ABI compatibility for AOT Inductor is continuing to proceed apace slowly, but one agreement is that we’re probably going to also only have the ABI compatible codegen for OSS as well. Performance - Flash Attention 2 is close to landing: [Flash Attention v2 by drisspg · Pull Request #105602 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/105602) but it is currently stuck because it takes a lot of memory to compile, causing CI problems. - In the PT2 weekly meeting, we discussed H100 benchmarking. There are a lot of interlocking parts to this: we need to upgrade Triton to get their H100 improvements, and not everyone on the PyTorch team has access to an H100. Still looking for someone to sign up for this. - CUDA graph updates are a thing now: [CUDA C++ Programming Guide — CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#updating-instantiated-graphs) There may be some opportunities here. Elias says: “It mostly helps with eliding input copies. For the most part, removing input copies only really matters when you torch.compile only part of your model and leave the rest of the model in eager. This use case is pretty unlikely to train well anyway since you’ll still need to bifurcate the memory pool.” However, personally, I also think CUDA graph updates could be pretty useful for allowing you to deallocate the pool of memory needed by a CUDA graph, only reallocating it when it’s time to run the CUDA graph again. Dynamo - There was a pretty notable pytree API BC breakage which caused some internal problems: [https://github.com/pytorch/pytorch/pull/106116](https://github.com/pytorch/pytorch/pull/106116) - Some big refactors that are in progress: refactoring skipfiles / allowed functions (talk to Yanbo), refactoring guard trees (talk to Animesh) - A bunch of new contributors being onboarded to Dynamo: Quansight is working more on Dynamo issues, and Jack Cao from PyTorch XLA is looking to help us with consolidated forwards-backwards-optimizer support in Dynamo as it is essential for XLA Dynamo perf. Numbers is on break this week due to [A100 runners down: apt-get install nvidia-docker2, Could not get lock /var/lib/dpkg/lock-frontend · Issue #108862 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108862) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [September 17, 2023, 6:36pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/70 "2023-09-17T18:36:50Z")
# State of PT2: Sep 15, 2023 edition Previous update: [State of symbolic shapes branch - #69 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/69#state-of-pt2-sep-8-2023-edition-1) ## Executive summary Dynamo - KJT tracing updates: Tracing torchrec\_dlrm with distributed sharding manages to get to the wait on the sharded embedding table lookups, at which point we are stuck on a complicated custom autograd function. Voz to take a look after finishing up intermediate backward hooks. In other news, the production folks on the workstream have finished getting rid of layer splitting for disables only, so they’re now quite interested in compiling through as well. Lots of foundational work that still needs to be done; hoping for Q4 but is very aggressive! Meta only: [https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#heading=h.jknt1mqmztph](https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#heading=h.jknt1mqmztph) - Animesh is going to be working on improving guard evaluation overhead, but there is still some disagreement among Voz, Jason and Edward about two major things: (1) should we port guards to C++ and do the rest of the scheme all in one go, and (2) should we stay in the “one compiled function, one check function” regime, or go straight to Voz’s one shared check function for everything. - Some folks from the Cinder team came to the PT2 weekly to talk about some challenges of running PyTorch with lazy imports. One big problem is the way Dynamo implements skipfiles by traversing modules to find all identifiers attached to them; this plays poorly with lazy imports. Other trouble points include decorators which put identifiers into global state, and our dispatcher registration mechanism. - Horace is complaining about compile time still kinda slow while he’s been working on llama. Profiling shows pytree is still big culprit (20%); we also spend a lot of time doing map\_aggregate in FX (10%). Some discussion about reviving our fake tensor propagation rules caching idea. - Meta only: We’ve had a lot of PT2 related SEVs recently. There’s been some initial investigation classifying what happened [https://docs.google.com/document/d/1bMoQEoBlZ4vwsUztH1dEeEETHuNJXEj1uItQH8Cd7jo/edit#heading=h.dnms1ad3rdvu](https://docs.google.com/document/d/1bMoQEoBlZ4vwsUztH1dEeEETHuNJXEj1uItQH8Cd7jo/edit#heading=h.dnms1ad3rdvu) and some suggestions on what to do next [https://docs.google.com/document/d/1jhwgscFWe\_G8JDSSRbFpkZKy02cw8wipyuNYaHXF4Rg/edit](https://docs.google.com/document/d/1jhwgscFWe_G8JDSSRbFpkZKy02cw8wipyuNYaHXF4Rg/edit) . A lot of the problem stems from insufficient / flaky downstream testing. Michael Suo is leading the charge here. - Unrelatedly, there is also some external feedback (Meta only: [https://docs.google.com/document/d/1Ss3idfGSTV4GWElOe6pgw9JkJ\_b0p5AaEdkf-Yzil7M/edit](https://docs.google.com/document/d/1Ss3idfGSTV4GWElOe6pgw9JkJ_b0p5AaEdkf-Yzil7M/edit)) that PT2 speedups are promising but hard to actually work reliably. A lot of it has to do with distributed, e.g., [torch.compile/triton holding GIL during compilation and CompiledKernel call results in deadlocks during distributed training · Issue #109074 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/109074) and [TorchInductor workers use "fork" which doesn't work in a multithreaded environment · Issue #108586 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108586) Inductor - ABI compatible AOTInductor made a bit of progress this week, with [https://github.com/pytorch/pytorch/pull/109450](https://github.com/pytorch/pytorch/pull/109450) and [[inductor] Add a C shim layer for libtorch by desertfire · Pull Request #109391 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/109391) by Bin Bao. - Will Feng looking into improved item() and tolist() support in Inductor: [https://github.com/pytorch/pytorch/pull/109262](https://github.com/pytorch/pytorch/pull/109262) Composability sync hit a lot of topics this week. [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit) Topics that weren’t otherwise covered in this doc: - Elias told us about how SDPA pattern matches (and others; both inference and training patterns supported) are now compiled ahead of time, making it a lot cheaper to do lots of patterns. We took advantage of that to add a lot more patterns to match other SDPA variants. [Add Python serialization to Pattern Matcher patterns by eellison · Pull Request #108894 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/108894) - Chien-Chin told us about the new PT2 DDP plans. We cannot directly trace DDP because it is implemented in C++, and we cannot easily port it to Python because the implementation is complicated by bucketing. So the idea is to implement a Python non-bucketed DDP, and rely on compile to optimize it away. - Horace told us about developments in LLMs. One thing he wants is dequant primitives in PT2: a way to take int3/int4 packed values and unpack them into a larger tensor, with the idea that PT2 would compile away the memory traffic. In general he doesn’t think we should directly do this in PT, as there are so many quantization formats. Dynamic shapes - Last week I mentioned opcheck testing is usable, but Richard Zou is still evolving it on user feedback. A recent new change is to put the xfails into a JSON file so it can easily be automatically updated. However, there are still complaints from folks that it’s too hard to understand what goes wrong when a test crashes. Richard is going to investigate a two stage process now, where by we separate generation of test inputs and actually running the tests. To ensure generation of test inputs is kept up to date, we only need a single new test which runs all of the tests in the test file in one go and xrefs what tests are exercised with what we have recorded. - Horace wants a version of Tensor where some of the sizes are stored on device. This would allow you to perform a data-dependent operation without synchronizing; and you would still save on memory traffic because you would have kernels mask out memory loads when they go out of bounds of the dynamic shape. In some sense, this is a specialization of jagged tensor where everything in the jagged dimension has the same size. - Notable bug fixes: - [Add meta and OpInfo for \_embedding\_bag\_dense\_backward](https://github.com/pytorch/pytorch/pull/109211) - [Add torch.distributed get\_rank and get\_world\_size to constant\_fold\_functions](https://github.com/pytorch/pytorch/pull/109029) ## Numbers This is nearly a month worth of numbers! **Training.** [34ddf08f27 dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2018%20Aug%202023%2018%3A38%3A32%20GMT&stopTime=Sun%2C%2017%20Sep%202023%2018%3A38%3A32%20GMT&granularity=day&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=34ddf08f2737025fb1447070fae3087889b2e8bb&rBranch=main&rCommit=93f2a64d4d1af38904cb4d36e3509856c205615b) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/5/5b3d10a6dabf20cb3fde993e3291b72236808a20.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/126fc18505fef5d52db1adf11163394914488481.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/d/d37675b45c43a97ce705af7e78a23bf2c91c8967.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/6d20c714ce9e06835cafafb91fb6ec1be2363dad.png) - mobilevit\_s in timm models no longer runs, it looks like it’s due to flash attention, Elias will be fixing it along with the pattern matcher PRs. - Performance regression in torchbench between `0cfc5899f9bade72c7e18666e2006b003b5848bc..3a79621c9dce17f77fbddc06aab21f6bc477f313`. Testing [https://github.com/pytorch/pytorch/actions/runs/6215325516](https://github.com/pytorch/pytorch/actions/runs/6215325516) [https://github.com/pytorch/pytorch/actions/runs/6215347814](https://github.com/pytorch/pytorch/actions/runs/6215347814) - Hugging Face improvement from flash attention v2 landing again - Meaningful compile time regression everywhere, with no obvious culprit. Example model: ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/c/c7ae3b0888a6b5521e85c60f4c5101a5f5da36cf.png) **Inference.** [34ddf08f27 dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2018%20Aug%202023%2019%3A13%3A14%20GMT&stopTime=Sun%2C%2017%20Sep%202023%2019%3A13%3A14%20GMT&granularity=day&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=34ddf08f2737025fb1447070fae3087889b2e8bb&rBranch=main&rCommit=68b9bf9671ffa4b580236f5bd436ce6c38d69a96) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e85f94ce9cd2f954fae8f8495aa3b8cd186c1a76.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/b/b580bb3f84f456e24371ac293f2aeddcfbcdb91f.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/f3b6b1dda522f43c9f9cd074c6f1a93c5076a5b8.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/3/3a1240fcb3365b95bba41ab588a7fc894c5eeda2.png) - A lot of torchbench improvement: detectron2\_fcos\_r\_50\_fpn, doctr\_reco\_predictor, drq, llama, pyhpc\_turbulent\_kinetic\_energy all now pass accuracy. - cudagraphs freezing accuracy improvement in timm models, likely from some major bugfixes for freezing - pytorch\_stargan had huge perf improvement c2ac0da445cfe3d848342926f9cd4422bd35bfe2..781b7ebe912ec24cbd917cd548b748b1650ab6a2 - HuggingFace regression due to pin update [Problems hit when upgrading the version of HF used in CI · Issue #108145 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/108145) - Fairly large aot inductor regression due to ABI changes. ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [September 24, 2023, 8:34pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/71 "2023-09-24T20:34:30Z")
# State of PT2: Sep 23, 2023 edition Previous update: [State of symbolic shapes branch - #69 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/69#state-of-pt2-sep-15-2023-edition-1) ## Executive summary Dynamo - Yanbo has been making good progress on understanding the state of our skipfiles/allowlist situation. Here is my attempt to record what he described to me in our 1:1. - First, what do these things do? For any given frame, we can make one of three decisions on it: inline - the default decision; skip - we never start Dynamo on this frame, and we induce a graph break instead of inlining into it (BUT, skipped functions may have overrides in Dynamo that allow us to avoid a graph break); allow in graph - we don’t inline into the function, but instead directly put it into the graph (and run it to do fake tensor propagation.) Skipfiles and allowlist control whether or not we do something different from the default decision. - Yanbo’s theory is that **allowlist should be explicitly enumerated function-by-function.** This makes sense; there’s a fixed set of operations we can actually put in the graph (coinciding with Torch IR; see composability sync), and they have to be audited to ensure they don’t do naughty stuff like mutate Python state. - Suppose that we didn’t care about compile time / Dynamo bugs at all. In theory, it shouldn’t be necessary to have a skip list at all, because you’d expect Dynamo to independently work out that something couldn’t be compiled and graph break. There is a big nuance here though: the **torch module is skipped!** Most of the time, this skip is bypassed for other reasons, e.g., a torch function is allowed in graph, or a submodule is explicitly allowed for inlining. But by default we won’t actually compile anything in torch (and this can be quite surprising for PyTorch devs!) - Chesterton’s fence rules everything around me. Sometimes we have manual implementations of functions (like nn.Module.parameters) which are unnecessary, because they were added back when Dynamo’s Python language support was not so good, but now we can just inline into these functions, but some seemingly benign skip rules are load bearing and cause problems. So many of Yanbo’s initial refactors will be oriented around preserving semantics as much as possible, while improving code organization. - Jason, Edward, Animesh and Voz got together to discuss some design questions about guard trees raised last week. The conclusion was that we are NOT going to do combined guard tries, Animesh’s plan as original scoped as is. One interesting thing I learned from this discussion was that guards with source-based guard structure deal poorly with guards that mention two sources, but Jason proposed a way to deal with this case: instead of directly having a guard like `x.size(0) == x.size(1)`, instead have assignment statements like `let s0 = x.size(0)` and `let s1 = x.size(1)`, and then have an extra guard that only makes reference to this local scratch space `s0 == s1`. These extra guards get run immediately once you notice that all of its free variables have been assigned. Jason’s argument is that size guards can be very fast to run if we compile them, so it doesn’t matter if they get run unnecessarily early. Some very rough meeting notes: [Guard Refactor Discussion - Google Docs](https://docs.google.com/document/d/1EbrR9o7Loi_fU1MHNJAxxCOItn0dv44hLF3NB2pZ1nE/edit#heading=h.o7t8ttlom4nx) - Lazos suffered a bit from some bikeshedding about how he should write some new VariableTrackers, but hey, at least we got a doc out of it: [Which VariableTracker should I use - Google Docs](https://docs.google.com/document/d/1XDPNK3iNNShg07jRXDOrMk2V_i66u1hEbPltcsxE-3E/edit#heading=h.i6v7gqw5byv6) - PSA: when you print Dynamo logs, they come with a `[0/0]` marker that says what frame you are compiling, and which recompile of that frame you are on. `[19/0]` means you are compiling the 19th interesting frame for the first time, while `[1/20]` means you are recompiling the 1st frame for the 20th time (probably bad!) Occasionally, you will see `[0/0_1]`; this means that we restarted analysis on 0/0 for some reason, so this is the second (1 is zero-indexed) time around compiling it. - I mentioned to jansel that there are a number of dynamo refactors I’d kind of like to firm up: mutable variable trackers, a more accurate python object model, variable tracker cleanup (we have a big VariableTracker subclass hierarchy), more metacircularity (so that constant folding is a lot easier to implement.) Hopefully we can hit some of these during the offsite. Composability - We had a very spicy session at composability sync this week on Torch IR [https://youtu.be/FSPNXppkcjg](https://youtu.be/FSPNXppkcjg) [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit#heading=h.bazzea3046cp) [https://docs.google.com/document/d/17O1R57oOZp\_fK4dRf83UiM4fH6h6nblxjizTrbHP8BY/edit](https://docs.google.com/document/d/17O1R57oOZp_fK4dRf83UiM4fH6h6nblxjizTrbHP8BY/edit) The crux of the matter is what to do about “Torch IR”, which is conceptually a PyTorch program capture representation that is produced by fx.symbolic\_trace: an unnormalized format that contains precisely torch API operations that are part of PyTorch’s public API. It is a coherent concept that is used by folks today, but its denormalization makes it difficult to write sound analyses/passes on. Some argued that because it’s so difficult to use, we shouldn’t expose it, while others argued that the horse escaped from the barn. We were able to agree in the meeting what Torch IR is and what guarantees you should expect from it, but it’s still an ongoing discussion how this should relate to export. - Zachary DeVito’s been working on single controller distributed paradigm for PyTorch, where we issue commands of entire Dynamo traced graphs for external nodes to run. This is being done via a tensor subclass, but it is a bit unusual in that it doesn’t match other tensor subclass applications, where we don’t actually want to trace into the subclass itself, we just want to trace tensor operations on it “as if it were a normal tensor.” - Apparently optree [https://github.com/metaopt/optree](https://github.com/metaopt/optree) is slowly making progress into becoming an optional dependency for PyTorch: if it is installed, PyTorch pytree APIs will transparently make use of it instead, for hefty speedups. Pytrees are a big performance cost for PT2 compile time, so it will be nice for this to land; so nice that Richard Zou has been wondering if we shouldn’t actually just make this a required dependency. - COW tensor is alive again, thanks to Kurt Mohler accepting a mission to finish of Mikey Dagitses work. [Implement Copy-on-write (COW) tensors · Issue #109833 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/109833) - Discussions on internal reliability continue. It’s going to be some sort of multi pronged approach where we negotiate with PGs what testing we absolutely must do, while at the same time improving testing on poorly exercised parts of PT2 (e.g., PT2+FSDP) and working out periodic tests that are covered similarly to how we cover benchmark results in OSS. Dynamic shapes - PSA: We’ve covered this before, but someone asked me about this so I’ll repeat it: our plan for PT2 support for table-batched embeddings is that we will eventually support capturing embeddings both with and without the fusion preapplied (e.g., by module swapping). It’s your choice then whether or not to reuse preexisting optimization passes, or write a new PT2 based optimization . - PSA: When registering C++ implementations (meta or composite) that take SymInt, make sure to pass SymInt by value and not by const reference. This applies to composite types like `optional` too. Unfortunately, we don’t check this properly right now: [https://github.com/pytorch/pytorch/pull/109727](https://github.com/pytorch/pytorch/pull/109727) is wending its way in but I have to fix all the people who did it wrong first lol. - Adnan Akhundov has been working on PT2 compiling [https://docs.google.com/document/d/1q1Rccii\_A1xRsZETGPmrOebjKPOhbtWZ4bz9XrI6AdA/edit#heading=h.cxi2qmayvqhr](https://docs.google.com/document/d/1q1Rccii_A1xRsZETGPmrOebjKPOhbtWZ4bz9XrI6AdA/edit#heading=h.cxi2qmayvqhr) (Meta only) and triggering a lot of unbacked SymInt missing features that we’ve been meaning to patch in but haven’t gotten around to yet. We landed a lot of improvements this week driven by this workstream: - [Use constrain\_range\_as\_size for nonzero/repeat\_interleave](https://github.com/pytorch/pytorch/pull/109857) - [Allow inferring size-nature from sizes passed to empty constructor](https://github.com/pytorch/pytorch/pull/109720) - [Handle unbacked symints in Triton size hints](https://github.com/pytorch/pytorch/pull/109609) - [Handle unbacked symints in buffer reuse calculation](https://github.com/pytorch/pytorch/pull/109603) - In progress: [Add support for item() and nonzero() codegen in Inductor](https://github.com/pytorch/pytorch/pull/109893) - just needs deps - There’s still a lot more to do. There’s a lot of use of size hints in inductor which need to be rewritten to deal with the unbacked case when no hint is available. Another unusual thing about Adnan’s setup is he is using AOTInductor, so we’re also getting a lot of extra asserts from add\_runtime\_assertions\_for\_constraints\_pass.py which also don’t compile atm. - Jeffrey Wan has worked out that we should represent singleton symints (to be used to represent ragged dimensions) as sympy Atoms, which compare only equal to themselves. This is better than Symbol because they don’t show up as free symbols this way. - Notable bug fixes: - [Implement traceable torch.tensor when you have SymInt/SymFloat inputs](https://github.com/pytorch/pytorch/pull/109515) - [Make SymFloat behave symmetrically with float in torch.tensor](https://github.com/pytorch/pytorch/pull/109513) - Notable new bug reports: - [basic\_gnn\_gcn: ERROR:common:TypeError: object of type ‘GreaterThan’ has no len()](https://github.com/pytorch/pytorch/issues/109884) - [masked\_select for meta backend](https://github.com/pytorch/pytorch/issues/109871) - [Cannot use constrain\_as\_size from fake tensor implementations: RuntimeError: tried to get Int out of SymInt](https://github.com/pytorch/pytorch/issues/109861) - [add\_runtime\_assertions\_for\_constraints\_pass adds redundant asserts](https://github.com/pytorch/pytorch/issues/109852) - [[dynamo] torch.\_dynamo.exc.Unsupported: call\_function BuiltinVariable(float) [TensorVariable()] {}](https://github.com/pytorch/pytorch/issues/109538) - [[dynamo][jagged tensor] Slow compilation time for a helper function of jagged tensor](https://github.com/pytorch/pytorch/issues/109583) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [October 8, 2023, 8:16pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/72 "2023-10-08T20:16:29Z")
# State of PT2: Oct 8, 2023 edition Previous update: [State of symbolic shapes branch - #70 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/70#state-of-pt2-sep-23-2023-edition-1) We were on break last week as I was on vacation. ## Executive summary Compiler/distributed offsite was last week! PyTorch Conference talk slides are due to Linux Foundation **end of this week!** Dynamo - Our initial take on mutable variable trackers was “well, it is probably technically feasible, but it’d be a lot of work and the ROI is not obviously there.” It came up again this week, though,for perf reasons: [RFC / Discussion - Mutable Variable Trackers - Google Docs](https://docs.google.com/document/d/1fUGc-jLwk1_-fylH-qFuLJK20lKUj-YkQnOTOveREbs/edit?usp=sharing) from Voz - We discussed accurate python object model: definitely something we should do for user defined objects, maybe Fidget-Spinner will work on it. We have some Dynamo bugs recently relating to classes with nontrivial metaclasses (like abc) and multiple inheritance. - We have a proposal for guarding on Dynamo configuration, which should make it a lot easier to tweak config options: [Dynamo guard on global configuration · Issue #110682 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/110682) One notable choice we make is that outer-most torch.compile config wins; if this would be annoying for you please comment on the issue. Tracing FSDP - We talked about the relative importance of landing tracing FSDP quickly during the offsite. The general consensus was that, while this is an important capability, the more time pressing problems are optimizing tensor parallel compute (as it’s harder to manually get optimal overlapping in this regime) and tracing DDP (which Chien-Chin is working on.) - During the offsite, we came up with a full plan for Dynamo-level support for propagating hooks to backwards. The primary complication is that, in full generality, a backward hook installed in a Dynamo compiled region may be arbitrary Python code that would vary from run to run, but we emphatically do _not_ want to guard on it (nor can we, since we didn’t inline into the function.) In the simple case, the function is constant from iteration to iteration and we can bake it into the backwards graph (this is what is currently implemented); in the complicated case, Dynamo must construct the residual function, and then somehow pass it to AOTAutograd compiled function, so AOTAutograd knows to know that particular function is what should be invoked when backward rolls around. This can be done but it’s all quite fiddly. For FSDP we don’t need it in full generality because it’s a constant function. - More folks are collaborating on Voz’s experimental FSDP tracing branch: [https://github.com/pytorch/pytorch/tree/voz/fsdp\_autograd3](https://github.com/pytorch/pytorch/tree/voz/fsdp_autograd3) To run things on the branch just say `torchrun --standalone --nproc_per_node=2 fsdp.py` (will run with compiled forwards, but NOT compiled backwards). Current status is that compiled forwards works, compiled backwards does not. The problem is that compiled autograd has to do a pre-pass with fake tensors to construct the FX graph, but during this pre-pass it is unable to run hooks, and that means parameters aren’t the sizes it is expecting. - Not quite FSDP, but putting it here: on the subject of single controller, Haiping Zhao also looking at it this problem space, much more from the distributed side. He, Zach and Horace have been chatting. Core - We spent a bit of time talking about optimizer in the offsite. @janeyx99 summarized the discussion at Meta only: [https://fb.workplace.com/groups/pytorch.oss.dev/posts/1750253475399186/](https://fb.workplace.com/groups/pytorch.oss.dev/posts/1750253475399186/) and Meta only: [https://docs.google.com/document/d/1JJhRCl8F51nH\_Ke8Yd\_BV3scAmv5V4eSVnle5D8\_\_po/edit](https://docs.google.com/document/d/1JJhRCl8F51nH_Ke8Yd_BV3scAmv5V4eSVnle5D8__po/edit) My brief summary: we’re going to make optimizer support taking parameters in arbitrary pytree structure, rather than forcing just a list of parameters (which gives you the awful integer indexed structure where you have to reverse engineer which parameter is what.) It’s not BC-breaking, but people who use this API will have a much easier to work with state dictionary. - Composability sync this week was all about quantization [https://www.youtube.com/watch?v=7WhgpAIvxHU](https://www.youtube.com/watch?v=7WhgpAIvxHU) [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit#heading=h.8y2jwyieg2yh) The resolutions: - uint2/uint3/uint4 support in core to be prototyped as Python subclass by torchrec folks (lead by Ivan Kobzarev) - Decent chance we are going to get dequantize operators that can show up in export IR - We will have a pattern matcher that will let you compile regular Python calls in PyTorch IR into appropriate ATen matchers, mirroring how Inductor’s pattern match infra works. This will support just returning fx.Nodes to you so you can do arbitrary transformations, instead of just doing a replacement. - Richard Zou has been trying to convince people to use the new operator registration API, but he has been noticing that people _really_ like the old fashioned autograd.Function API, because it doesn’t require them to do work for things they don’t care about (e.g., supporting other transforms). Since we need to support this anyway, we are going to make sure Dynamo’s support in this regime is good. - AOTDispatch subclass PR is approved, close to landing! [https://github.com/pytorch/pytorch/pull/104483](https://github.com/pytorch/pytorch/pull/104483) Inductor - AOTInductor is currently working hard on GPU model support, but some folks have been poking at it for CPU, overhead sensitive workflows. There will likely be some work done in this area, cool increase in scope. - Inductor strategy will be presented to upper leadership soon. Meta-only slides: [https://docs.google.com/presentation/d/1M6W5YuXhfCkngmjPC8a\_EfdMs\_P6wXzsAbr\_LKkuisU/edit#slide=id.g2887f2cdaba\_0\_23](https://docs.google.com/presentation/d/1M6W5YuXhfCkngmjPC8a_EfdMs_P6wXzsAbr_LKkuisU/edit#slide=id.g2887f2cdaba_0_23) (They’re pretty interesting, I recommend reading them if you have access.) Export - Export input/output matching is being a problem again. This is the `AssertionError: traced result #1 () is not among graph-captured output`. Someone should rewrite this code. Dynamic shapes - ysiraichi is transitioning to PyTorch XLA, so he will have less time to work on dynamic shapes specifically. - There was a torchrec\_dlrm update last week; the main change is that I redid the torchrec changes assuming variable batches, and after fixing bugs it all worked out smoothly. Still blocked on complicated autograd.Function support. [https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#heading=h.34z2pradlobb](https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#heading=h.34z2pradlobb) - Notable bug fixes: - [Fix numel test to be \> 2](https://github.com/pytorch/pytorch/pull/110731) - [Change flash attention outputs to be SymInt instead of int](https://github.com/pytorch/pytorch/pull/110533) - [Add functional collective all\_to\_all\_single and support it in Inductor](https://github.com/pytorch/pytorch/pull/110195) - [Add masked\_select abstract impl](https://github.com/pytorch/pytorch/pull/110103) - [Add support for item() and nonzero() codegen in Inductor](https://github.com/pytorch/pytorch/pull/109893) landed - Notable new bug reports: - [Support using SymBool in arithmetics](https://github.com/pytorch/pytorch/issues/110738) - [Make torch.\_check work in Dynamo](https://github.com/pytorch/pytorch/issues/110719) - [variable type mismatch when trying compile](https://github.com/pytorch/pytorch/issues/110696) - [Unbacked SymInts get reallocated whenever you repropagate fake tensors](https://github.com/pytorch/pytorch/issues/110136) ## Numbers I guess I’m doing these monthly now. **Training.** [1b34238d67 dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2008%20Sep%202023%2019%3A57%3A52%20GMT&stopTime=Sun%2C%2008%20Oct%202023%2019%3A57%3A52%20GMT&granularity=day&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=1b34238d6799b4483508f2953e1ae1f0e862324d&rBranch=main&rCommit=34ddf08f2737025fb1447070fae3087889b2e8bb) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/0/0d48bbe4513293d8ecad84497b8e2fe035137eeb.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/17422386677c36f0090bab753e61d38d86abfa67.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/1/1cfe5549746fd7858eb529120a02b58b40eaa737.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/8/87810c16f53d0f1fe2672369bcf6714058476f25.png) - nanogpt, stable\_diffusion\_text\_encoder, stable\_diffusion\_unet newly added torchbench models - mobilevit\_s now passing timm\_models - Not sure what’s going on with blueberries readout lol. - Compile time does seem to have gotten worse. @Chillee has been complaining about compile time, although a lot of it is in Dynamo tracing. It is hard to see the effect of Dynamo in our current benchmark suite because it is heavily Inductor biased. Some improvement from guard hashing, but Horace says it’s only 1-2 seconds. - Unattributed speedup on timm\_efficientdet **Inference.** [1b34238d67 dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2008%20Sep%202023%2019%3A57%3A52%20GMT&stopTime=Sun%2C%2008%20Oct%202023%2019%3A57%3A52%20GMT&granularity=day&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=1b34238d6799b4483508f2953e1ae1f0e862324d&rBranch=main&rCommit=34ddf08f2737025fb1447070fae3087889b2e8bb) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/2e05b8aea00f95963ba9eb3b97b1596f5ce19cf5.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/f/f08bb1f3c4119796821acc8f45b22c882503ca8e.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/b/ba22d4d7966451dd9682e695af12f148bccd12be.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/7/7a91f5a5fad9b6686612b77176223cc4c4b70f82.png) - Lots of enablement in aot inductor, I like to see that pass rate go up - Some speedups are attributable to nanogpt being added - 1% improvement in HF, Horace updated some loading logic - HF inference: better flash attention matching at low inference +19%, some of this is also FlashAttention v2 - Some ups and downs with Yanbo’s for equiv invocation, letting us hit baddmm ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [November 6, 2023, 12:04am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/73 "2023-11-06T00:04:34Z")
# State of PT2: Nov 3, 2023 edition Previous update: [State of symbolic shapes branch - #71 by ezyang](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/71#state-of-pt2-oct-8-2023-edition-1) Sorry about the month’s delay! Between more vacation and PTC there wasn’t much time to do a writeup over the weekend. ## Executive summary Big tickets - PyTorch Conference happened! Thanks everyone who attended, there were lots of fun discussions. You can watch the talks at [https://www.youtube.com/watch?v=dR0lHxt3Tjo&list=PL\_lsbAsL\_o2BivkGLiDfHY9VqWlaNoZ2O](https://www.youtube.com/watch?v=dR0lHxt3Tjo&list=PL_lsbAsL_o2BivkGLiDfHY9VqWlaNoZ2O) . Some fun in person discussions that I had: (1) with Pierre Guilmin, you can now torch.compile complex tensors: [Add complex tensor with subclassing by pierreguilmin · Pull Request #48 · albanD/subclass\_zoo · GitHub](https://github.com/albanD/subclass_zoo/pull/48) This is actually going to be the preferred way to torch.compile complex numbers as you Triton doesn’t support interleaved layout and you’re not going to get efficient matrix multiply that way anyway (because the built-in instructions don’t support complex.) (2) Ho Young Jhoo and Nuno Lopes had some interesting work on automatically pipelining NNs, it was quite interesting. (3) Jack Cao and I sketched out what single step graph should look like in Dynamo, track progress at [https://github.com/pytorch/pytorch/pull/112296](https://github.com/pytorch/pytorch/pull/112296) - PyTorch 2.2 release is coming! The branch cut will be Dec 1. Dynamo - We’ve been having lots of discussions about what it will take to get Dynamo to the same level of stability as long running compiler projects like HHVM or LLVM. Some thoughts about refactoring pieces at [Refactoring Dynamo for stability - Google Docs](https://docs.google.com/document/d/1Kz02jiA45dGPDBnY9Xd3Z5S864evbMv75E_CZPOehJU/edit) As a smaller step, @voz has organized a weekly triage meeting for PT2 issues, separate from the regular PT2 weekly. - @suo is taking a serious look at getting torchbind to work on PT2. Some basic design notes at [PT2 torchbind - Google Docs](https://docs.google.com/document/d/19u6Ptc3_NeoB28OSrJ_wEis917UZLum-e3DkfUeA1T8/edit#heading=h.m1o9fvr5ibqz) ; we also discussed this at composability sync - In the land of tracing FSDP, there is currently some grunging about in PyTorch’s accumulate grad implementation. It is fairly complicated, including logic that checks the reference count to decide whether or not to reuse a buffer inplace or not. There is some debate about whether or not there should be an accumulate grad aten op (@jansel implemented one that lowers all the way to inductor), or it should be traced through by Dynamo. - Some work towards reducing the amount of guard administration is being made here [[export] Skip guard propagation for export only. by zhxchen17 · Pull Request #112685 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/112685) which should materially improve Dynamo tracing speeds - Some news about compiled optimizer from @mlazos [https://docs.google.com/document/d/1oHyw0RULF7UKZBCrOdVOzshlhiBFuTT7zKZd2xOGc3k/edit](https://docs.google.com/document/d/1oHyw0RULF7UKZBCrOdVOzshlhiBFuTT7zKZd2xOGc3k/edit) Core libraries - Ying Liu has been working on a tensor subclass for async execution. We discussed it in composability sync. The idea is that you can trigger an operation (typically communication) on a side stream, as well as some follow on operations, without having to literally move the follow on operations to the point where a sync happens. This also means that code in torchrec that has to be manually written as a pair of custom autograd functions for req/wait can be written in an intuitive, autograd style. We have a version that does this manually with callbacks (only queue kernels onto the stream at some known later point in time) and Ying is working on another version that uses streams only. One interesting thing we noticed that when you schedule allreduce in forwards first, backwards will naturally schedule it last, but you actually want the allreduce to happen ASAP! @albanD suggested we may be able to add an API to modify the priority order of autograd backwards, could be useful. - There will be a new repo [GitHub - pytorch/ao: PyTorch native quantization and sparsity for training and inference](https://github.com/pytorch-labs/ao) for some of the new quantization schemes we’re working on. We discussed this in composability sync. - I did a long overdue update to record\_stream docs at [Add a note about performant record\_stream use. by ezyang · Pull Request #112526 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/112526) after having some more discussions about it with @eellison who was trying to get cuda graph trees to work with record stream. - We’ve been talking about this with Vincent for a while, but there is now a proposed PR to add TensorDict to PyTorch core, check it out: [[RFC] Tensordict integration by vmoens · Pull Request #112441 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/112441) Dynamic shapes - repeat\_interleave dynamic shapes support was reverted due to S376879, this revert may itself have caused a sev S377088. It turns out that this diff was not related to the SEV, so we are relanding it. - torchrec dlrm with sharding and inductor works end-to-end, all the way through! Many of the changes have been merged upstream to torchrec/FBGEMM. Check [https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#bookmark=id.bpg458y8bfaa](https://docs.google.com/document/d/1VTGEh0MqadAsuRy0s5u39wQhNwMSVgCgYewivMcBbuU/edit#bookmark=id.bpg458y8bfaa) for status. We’re far enough along that the internal folks are going to try to do some enablement on their reco models - We had some discussion about supporting mark\_dynamic and automatic dynamic on tensor subclasses. Some of the complication is around the fact that you can have sizes that only occur in the outer tensor but not the inner tensor, and vice versa. Check for notes: [https://docs.google.com/document/d/1ipSxcTzEMMOAPvxP-YJlD5JBZZmIGgh8Q34ixtOUCRo/edit#heading=h.3px8g3br0skz](https://docs.google.com/document/d/1ipSxcTzEMMOAPvxP-YJlD5JBZZmIGgh8Q34ixtOUCRo/edit#heading=h.3px8g3br0skz) - Adnan has been running into a lot of “cannot guard on data dependent SymInts” and this has me wondering if we shouldn’t have a mode that automatically suggests what runtime asserts you ought to have. This lead to [Tracing mode for unbacked SymInts using real data · Issue #112749 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/112749) - If you need to force an input integer to be dynamic using mark\_dynamic, one way to hack it is to pass a 0xN tensor instead of an int and then project out the int again with size(1). - Notable new bugs - [AllenaiLongformerBase failing w/ dynamic shapes: “‘Pointwise’ object has no attribute ‘get\_stride’”](https://github.com/pytorch/pytorch/issues/112913) - [[dynamo] `.view([..., -1, ...])` fails on Tensors with unbacked SymInts in the shape](https://github.com/pytorch/pytorch/issues/112347) - the workaround we found was to manually replace i0 with i1 \* 12 so that the modulus checks work out - [pack\_padded\_sequence/pad\_packed\_sequence support in dynamo](https://github.com/pytorch/pytorch/issues/112044) - [Operators that return dynamic-shape outputs that require\_grad choke in AOTAutograd](https://github.com/pytorch/pytorch/issues/111950) - [torch2.1.0 DDP+compile+dynamic\_shape cause error](https://github.com/pytorch/pytorch/issues/111636) - workaround with `optimize_ddp = False` - [[inductor][dynamic] fused\_attention pattern could not be matched due to sym\_size](https://github.com/pytorch/pytorch/issues/111190) - Notable fixes - [Guarantee expr is a sympy.Expr before xreplace’ing it](https://github.com/pytorch/pytorch/pull/112619) - [Reland “Trigger specialization when you call size()/stride() from C++ (#111935)”](https://github.com/pytorch/pytorch/pull/112605) - [Refine replacements with equality tests on runtime asserts](https://github.com/pytorch/pytorch/pull/112156) - [Allow binary pointwise operations to cause refinement on unbacked SymInts](https://github.com/pytorch/pytorch/pull/112155) - [Use OpOverload instead of OpOverloadPacket for size/stride/etc slots](https://github.com/pytorch/pytorch/pull/112119) - [Convert evaluate\_expr GuardOnDataDependentSymNode into graph break](https://github.com/pytorch/pytorch/pull/111919) - [Don’t DCE unbacked SymInt if it is returned as shape constant buffer](https://github.com/pytorch/pytorch/pull/111803) - [SymIntify convolution](https://github.com/pytorch/pytorch/pull/111599) - [Don’t suppress original error message for data-dependent value](https://github.com/pytorch/pytorch/pull/111596) - [Allow SymInt to specialize to FLOAT](https://github.com/pytorch/pytorch/pull/111219) - [Force specialization on INT\_LIST](https://github.com/pytorch/pytorch/pull/111216) - [Don’t sympify reflection\_pad2d ranges](https://github.com/pytorch/pytorch/pull/111212) and [Improve reflection\_pad2d lowering for dynamic shapes](https://github.com/pytorch/pytorch/pull/110988) - [Use torch.\_check for cat error checking](https://github.com/pytorch/pytorch/pull/111035) - [Fix arange with dynamic end argument.](https://github.com/pytorch/pytorch/pull/110979) # Numbers **Training.** [64f326097b dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2006%20Oct%202023%2022%3A45%3A07%20GMT&stopTime=Sun%2C%2005%20Nov%202023%2023%3A45%3A07%20GMT&granularity=day&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=64f326097be8ac66ff057365f3bed2d64c697563&rBranch=main&rCommit=1b34238d6799b4483508f2953e1ae1f0e862324d) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/ac9a95dbad6bfcb95770c4513e20301963ef2797.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/ec350c7e1da6e66c2ed74059ca4f66e81c27c394.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/2/2776f16a7f247cb6c7b013f0149b8b37128600ca.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/69ea431cdfa6fe06936f08be788b19f34a1fcde6.png) - TIMM improvement is from channels last optimization **Inference.** [64f326097b dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Fri%2C%2006%20Oct%202023%2022%3A55%3A53%20GMT&stopTime=Sun%2C%2005%20Nov%202023%2023%3A55%3A53%20GMT&granularity=day&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=main&lCommit=64f326097be8ac66ff057365f3bed2d64c697563&rBranch=main&rCommit=1b34238d6799b4483508f2953e1ae1f0e862324d) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e24400fd1536fd9c62b030f11e50a8b3bf57006b.jpeg) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/a/a84997403f0ddf4bbd73989cbc022bcb432c72a3.jpeg) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/7/77ed108e5f3b05ff48e413c47d1c64c6d729bb4e.png) ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/6/610bb9026e97bdd46e1108184bc92fa3acf7be69.png) - 3% HF improvement from concat codgen on inference ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [January 13, 2024, 1:55pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/74 "2024-01-13T13:55:02Z")
# State of PT2: Jan 12, 2024 edition We’re back from holiday break. - Vibes: I’ve been back from recharge for only a week, and already it feels like an avalanche of bugs! Phew, lots to burn down. I asked Twitter for some time tracking advice: [https://twitter.com/ezyang/status/1745523916624330974](https://twitter.com/ezyang/status/1745523916624330974) - PyTorch Dev Podcast is back. [https://pytorch-dev-podcast.simplecast.com/episodes/dynamo-variabletracker](https://pytorch-dev-podcast.simplecast.com/episodes/dynamo-variabletracker) and more to come. Send me any topic requests. I had considerable decision paralysis about what to record, so for now I’m just going to randomly walk over topics that are salient in my mind. - Dtypes for uint1-7 are going to be a thing. PR at [https://github.com/pytorch/pytorch/pull/117208/](https://github.com/pytorch/pytorch/pull/117208/); the addition of these types is a revision of our dtype addition policy previously discussed at [Supporting new dtypes in PyTorch - Google Docs](https://docs.google.com/document/d/1O69_acetdC5QgXV5hXyy0oCDPJv1-qg5pc_Y0yMh5kg/edit#heading=h.w86jy6sdyvq5). Note that we recently added barebones support for uint16, uint32 and uint64 in PyTorch. - Jack Cao has a design doc for Dynamo single step graph capture [[RFC] Dynamo Single Step Graph · Issue #117394 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/117394) and we ratified it at composability sync. The way it uses compiled autograd to generate another Python program to inline into from Dynamo symbolic evaluate is pretty cool, check it out. - Did you know that mypy in daemon mode is way faster at type checking than lintrunner? - Landed stuff: - [Stop unconditionally applying hermetic mode](https://github.com/pytorch/pytorch/pull/116996) - hermetic mode prevented you from doing things like calling torch.compile or returning tensor subclasses from operator implementations registered to dispatcher. I’ve decided multipy is dead and so we can relax this restriction. - [Add AT\_DISPATCH\_V2](https://github.com/pytorch/pytorch/pull/116698) - this cool new macro lets you dispatch to multiple dtypes but without having to count how many extra dtypes you add. Check it out at Dispatch\_v2.h - [Prevent unbacked symbol reallocation by forcing unification for unbacked symbol def sites](https://github.com/pytorch/pytorch/pull/114368) makes progress on the we reallocate unbacked symbols problem. There’s still some other reallocation happening in AOTAutograd that needs to be nailed. - [Add `torch._lazy_clone` to create COW tensors](https://github.com/pytorch/pytorch/pull/117162) we have FINALLY landed a public API for making copy-on-write tensors, thanks @kurtamohler. Give it a try. - Some new bug fixes up for review: - [Properly preserve SymInt input invariant when splitting graphs](https://github.com/pytorch/pytorch/pull/117406) - fixes longstanding optimize\_ddp and dynamic shapes interaction bug. Was a lot simpler than I thought it would be. - [Avoid performing replacements when it would unrefine ranges](https://github.com/pytorch/pytorch/pull/117356) - this PR should actually fix a large class of guard on unbacked SymInt, in part because we now properly preserve ranges from Dynamo into AOTAutograd. ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [January 20, 2024, 10:00pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/75 "2024-01-20T22:00:21Z")
# State of PT2: Jan 20, 2024 edition - I did some live streamed bug fix sessions, which you can watch [on Youtube](https://www.youtube.com/edwardzyang). Check it out! - We had an internal SEV review. One thing that stood out to me was that two of the SEVs were compilation slowness stemming from dynamic shapes accidentally being turned on when it shouldn’t be. Dynamic shapes trouble. - Jack Cao and I got a design for dealing with saved for backwards intermediates, which is that we’re going to generate a pre-dispatch ATen FX graph into Dynamo, rather than directly represent torch.\*. We’re pretty sure this should work. Track Jack’s progress at [https://github.com/pytorch/pytorch/pull/112296](https://github.com/pytorch/pytorch/pull/112296) - Will Feng is getting close to finished with his prototype for lazy scheduler. It is a bit complicated, but it seems like it will work. [[RFC] LazyScheduler for operator reordering - Google Docs](https://docs.google.com/document/d/1vv0H5IMGwUMyzmJKnksJOnRSult1B4YlbBSs_MeAvXM/edit) - Richard Zou is back to designing a new, class-based custom ops API, analogous to autograd.Function. This is based on user feedback where the existing custom ops API is quite difficult to use for autograd. Meta only: [https://docs.google.com/document/d/1TVV3sDUv1E8ou1Hk0MeL7e1C6QPNl5UQesiP2MWhDSQ/edit](https://docs.google.com/document/d/1TVV3sDUv1E8ou1Hk0MeL7e1C6QPNl5UQesiP2MWhDSQ/edit) (hopefully public soon) - Richard Zou and co have been working on making the Dynamo CI tests less flaky. They’ve made a lot of progress. Right now, all of the tests in CI are reasonably hermetic (they reset before running) and they’ve clustered the failures. A lot of very simple stuff (e.g., error where Dynamo uses a variable that’s not defined) and then a huge long tail of niche failures. Not many accuracy failures. One big problem is many problems only repro in CI environment and not locally. - A number of new folks from PL&R are ramping up on Dynamo! The extra manpower is much appreciated. - Yanbo Liang is thinking about how to measure compile time in our prod workloads. The challenge is full E2E tests are quite difficult to run. But maybe simple components like Shampoo can be extracted out and tested on their own! - Simon Fan is working on improving testing of compiled autograd by turning it on our torchbench suite. Meta only status: [https://docs.google.com/spreadsheets/d/17aCEcAcif-1saHrdqALjfr-ybRMl\_uEq7mqje5whPZw/edit?usp=sharing](https://docs.google.com/spreadsheets/d/17aCEcAcif-1saHrdqALjfr-ybRMl_uEq7mqje5whPZw/edit?usp=sharing) the summary is some pass, some fail. \_cudnn\_rnn\_backward needs meta support, and there are some side/stride mismatch issues. Fire up the minifier! He’s also running torchbench with DDP. - New einops style library einx from the community: [Reddit - The heart of the internet](https://www.reddit.com/r/MachineLearning/comments/198yyzy/p_einx_tensor_operations_in_einsteininspired/?share_id=SICK8vkXXpD5rcku0hYEp&utm_content=2&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1) Seems pretty neat! This is my continued reminder that first class dimensions are also a thing in PyTorch too. - Landed stuff: - [Document and type torch.\_inductor.virtualized](https://github.com/pytorch/pytorch/pull/117658) - this is pretty helpful if you’re trying to understand how the global state in Inductor works - [Catch some missing unbacked symbol dependencies](https://github.com/pytorch/pytorch/pull/117650) - this bug typically manifests as we generate some code and it references i0 but the name doesn’t exist (we DCE’d too much) - Up for review: - [Rename unbacked SymInt prefix to u](https://github.com/pytorch/pytorch/pull/117859) - the current “i” prefix conflicts with indexing variables, leading to hilarious bugs! - [Fix several bugs related to unbacked SymInt codegen in inductor](https://github.com/pytorch/pytorch/pull/117862) - this fixes some long standing bugs when we accidentally reallocate unbacked SymInts - [Document OpsHandler protocol](https://github.com/pytorch/pytorch/pull/117790) - pretty cool doc PR about the ops namespace in Inductor, and what all of the operations mean. - [https://github.com/pytorch/pytorch/pull/117300](https://github.com/pytorch/pytorch/pull/117300) (from Oguz), a twisty story since we asked the Triton devs if we could do the analysis directly in Triton, they were not too interested. ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [January 29, 2024, 3:50am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/76 "2024-01-29T03:50:33Z")
# State of PT2: Jan 28, 2024 edition - Calibrations were this week. - Joel, Jeffrey, Alban, Brian and Edward convened to discuss subclass view fakeification again. Subclass view fakeification occurs when we are given a tensor subclass which is a view of another tensor (the canonical example is a nested tensor which is a view of a dense tensor of all the packed data), and we need to convert it into a faithful fake representation so we can simulate operations on it in Dynamo and AOT Autograd. Construction of views in fake tensors is traditionally done by fakeifying the base tensor, and then reapplying a recorded view function which specifies how to “replay” the view on an arbitrary new base. The problem of subclass view fakeification is that these view functions typically hard code size / tensors that are free variables of the view operation, but when fakeifying, these need to be swapped out with symbolic integers and corresponding fake tensors. Joel’s resolution after the meeting was to reify view functions so that this information can be swapped. Notes: [Subclass View Fake-ification in PT2 - Google Docs](https://docs.google.com/document/d/1C5taWiplmX7nKiURXDOAZG2W5VNJ2iV0fQFq92H0Cxw/edit#heading=h.vl1gidtprtoo) - Shampoo compile time is still a problem. I talked to some folks on Wednesday who were like “our job is stuck in produce\_guards” and it turned out to be the exact same tensors\_definitely\_do\_not\_overlap guard explosion that caused two other SEVs described at [Meta issue: Automatic dynamic shapes can cause compile-time performance cliffs · Issue #118213 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/118213). Brian is going to try to fix the tensors\_definitely\_do\_not\_overlap problem in a few weeks. I showed Yanbo how to navigate the logs to find the culprit (in this case, just searching for symbolic\_shapes logs was enough to identify this as the same problem.) There is some difficulty reliably turning of automatic dynamic shapes (which would help with this problem) that needs to be studied in more detail. - Two interesting new posts: [Micro-optimizations for the most micro of benchmarks](https://dev-discuss.pytorch.org/t/micro-optimizations-for-the-most-micro-of-benchmarks/1836) and [[RFC] New Python operator registration API](https://dev-discuss.pytorch.org/t/rfc-new-python-operator-registration-api/1838) which I highly recommend - I’ve been talking Elias and Mario through the plan to remove unsound 0/1 specialization [Eliminate compile time ranges for a simpler analysis · Issue #117361 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/117361) and people are on board, I am going to implement this next week. - A bit of chatter about what to do about backtraces in distributed interleaving each other. Chip Turner shared that if you pass appropriate arguments to torchrun like `torchrun --role mnist-trainer --log-dir /tmp/l -t 3 -r 3 mnist/main.py` good things happen. Unfortunately in internal prod we are still shoving everything to stderr but maybe we can change that. Meta only: [https://fb.workplace.com/groups/mast.users/posts/1451868658730526](https://fb.workplace.com/groups/mast.users/posts/1451868658730526) - We’re using dmypy instead of mypy for typechecking now in lintrunner. Typechecking is a lot faster! If you think there’s some weird cache problem, you can say `dmypy stop` to restart the daemon. - A lot of dynamic shapes bugs in Inductor specifically this week: [Inductor sizevars wrapper assignment DCE hazard · Issue #118385 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/118385) [assert isinstance(value, CppCSEVariable) and value.is\_vec · Issue #118379 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/118379) [Inductor mixed device operations not handled correctly, maybe buffer reuse problem? · Issue #118299 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/118299) . I’ve personally been mucking around a bit in Inductor recently! - Landed stuff: - [Realize inputs to DynamicScalar before unwrapping storage](https://github.com/pytorch/pytorch/pull/118125) - another “oops we fed the wrong thing to an extern kernel” bug - Landed from last week: [Fix several bugs related to unbacked SymInt codegen in inductor](https://github.com/pytorch/pytorch/pull/117862), [Rename unbacked SymInt prefix to u](https://github.com/pytorch/pytorch/pull/117859), [Document OpsHandler protocol](https://github.com/pytorch/pytorch/pull/117790) - Notable new bugs: - [Inductor: KeyError: ‘could not find u6’](https://github.com/pytorch/pytorch/issues/118218) - [Inductor sizevars wrapper assignment DCE hazard](https://github.com/pytorch/pytorch/issues/118385) - [Further simplifying guards](https://github.com/pytorch/pytorch/issues/118332) - - [Potential correctness problem with symbolic size deduplication leading to spurious dependence on tangents in functorch partitioner](https://github.com/pytorch/pytorch/issues/118224) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [February 5, 2024, 4:30am UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/77 "2024-02-05T04:30:30Z")
# State of PT2: Feb 4, 2024 edition - Alban working on a concept of “accelerator” for the PyTorch library, to better support non-standard backends. Some pain points including pinned memory (which is currently CUDA specific) and how to write library code like FSDP in a way that can handle multiple devices. Doc at [https://docs.google.com/document/d/1TySu95kPLc6kNzlOg1T8IRGHVkezv8cWyweEXYZaxlU/edit#heading=h.x2rqjkjdhxak](https://docs.google.com/document/d/1TySu95kPLc6kNzlOg1T8IRGHVkezv8cWyweEXYZaxlU/edit#heading=h.x2rqjkjdhxak) and we discussed it at composability meeting. - Jeffrey Wan has been working on [https://github.com/pytorch/pytorch/pull/117904](https://github.com/pytorch/pytorch/pull/117904) . Most notably, singleton SymInts are becoming a nested tensor specific concept, getting tensor stored directly on themselves, and are being supported being passed directly to a tensor constructor, so you can create a nested tensor directly from torch.empty so long as you appropriately pass in a nested int. The motivating reason for all of this work is so that we can reuse all autograd formulas with nested tensors; this is what necessitated making sizes well defined for nested tensor, despite the existence of a jagged dimension. - Brian: Has been working on a lot of internal driven fixes for DTensor and PT2. One particularly notable PR is a new API proposal [AOTDispatch: allow subclasses to correct when we guess metadata of tangents incorrectly by bdhirsh · Pull Request #118670 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/118670) for working around problems where the forward/backward have different tensor metadata when a subclass is involved. In the long term, Brian will likely take on fixing it properly (by having AOTAutograd recompiling when outputs show up and they mismatch what the existing compiled graph needs.) The difficulty here is mostly because partitioning decision has already been made at this point. - Composability sync [https://www.youtube.com/watch?v=kSOmyARCbyM](https://www.youtube.com/watch?v=kSOmyARCbyM) (minutes [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit#heading=h.8y2jwyieg2yh)) had a discussion about how exactly to handle AsyncCollectiveTensor in PT2. Broadly, our conclusion was to go the simple route: we wait on all input AsyncCollectiveTensors, output tensors from collectives don’t wait and instead produce AsyncCollectiveTensor from Inductor. Some debase about how some applications like FSDP do need fine grained stream control to control memory usage. - Michael Suo asked me, what are non urgent high impact things I (or others) can do? The big one that came to mind was improving the typing in PyTorch codebase. Typing serves as documentation and helps catch errors. It is a virtuous circle: the better typed we are, the more likely new code is to be typed. - Will Feng has picked up FSDP patch set from Voz. Most recent branch does not work end to end: the gradients are always zero. My advice to Will was to work on landing the individual fixes to main and trust that Voz has identified all the missing gaps. In the leads meeting there was some discussion about what to do about this workstream overall after Voz’s departure; jansel still supports this strategically and we are still committed to it. - Tugsuu has been working on pre-dispatch functionalization and non-strict export. I mentioned to him Jack Cao’s work at [https://github.com/pytorch/pytorch/pull/118155](https://github.com/pytorch/pytorch/pull/118155) - Richard was delighted to tell me that all flaky tests in Dynamo are fixed: keep an eye out for an official announcement. - Did you know that PyTorch actually has two sources of truth? Many internal oriented developers develop on top of fbcode to ensure fast lands into fbcode. This can result in desynchronization of fbcode and GitHub and it is necessary to reorder commits. But commits are not necessarily commutative; when they are not, this results in a “splitrace” ([https://docs.google.com/document/d/1LJFKwD\_lBudEt4kiUIMh\_LkYp0\_CFVaUSe64kIPglwo/edit#heading=h.j49wad13wr3z](https://docs.google.com/document/d/1LJFKwD_lBudEt4kiUIMh_LkYp0_CFVaUSe64kIPglwo/edit#heading=h.j49wad13wr3z)). I chatted with Ivan Zaitsev in case I had some good ideas for how to deal with it. One thing is that potential conflicts between diffs is unavoidable because we are not doing a bors-style landbot. So the name of the game is identifying if commits are likely to be safe to reorder. We have some evidence of commutativity from PRs themselves, as their final test is typically N commits behind their actual merge commit. In a decentralized system, it is best to first ask if you can make it centralized first: much simpler! - Elina Lobanova joining us to help with some observability logging stuff. She told me some good ideas: one that stuck out to me is how we should do feature flagging: we should read them out at training process start and then keep them the same until the job relaunches (Google flag style). This way, you can diff flags between jobs and see if they changed. - I popped into the Inductor weekly sync to get some info about topics. I pitched us rewriting our test suite to have one file per test and more standardization; there was lukewarm reception for this (although one person from the HHVM team was like “yeah, we have 15k test files in HHVM, what’s the problem?”) No one apparently helping anyone with problems internally due to inductor. Unbacked symints enablement workstreams are unblocked. Inductor workstream tracking at [https://docs.google.com/spreadsheets/d/1pwW5bRZqIzbi1026JrKo83P1P2Vbf8MG8CF09-48O9Q/edit#gid=0](https://docs.google.com/spreadsheets/d/1pwW5bRZqIzbi1026JrKo83P1P2Vbf8MG8CF09-48O9Q/edit#gid=0) - Notable new developer proposals: - [Automatic dynamic shapes should give up if a symbolic variable has too many guards on it](https://github.com/pytorch/pytorch/issues/118798) - [Selectively disable dynamic shapes in an inner compile region](https://github.com/pytorch/pytorch/issues/118758) - [Offline log analyzer/formatter](https://github.com/pytorch/pytorch/issues/119063) (thanks Elina for inspiring this) - [A way to designate a portion of torch.compile as a noinline block that is compiled/guarded separately, but less disruptive than a graph break (e.g., for loops)](https://github.com/pytorch/pytorch/issues/118966) - Notable new bugs: - [Dynamo fails on scalar bit shifts when `dynamic=True`](https://github.com/pytorch/pytorch/issues/119152); also some related interest due to power of two calculation (for block size, so maybe not real) - [torch.compile crashes when using DDP and dynamic shapes and torch.utils.checkpoint in 2.2.0](https://github.com/pytorch/pytorch/issues/118984) marked high prio - [`torch.compile(dynamic=True)` compiles forever](https://github.com/pytorch/pytorch/issues/118492); moral of story, don’t use dynamic=True - Landed stuff: - [Add documentation for meta device](https://github.com/pytorch/pytorch/pull/119119) - [Make torch.\_dynamo.mark\_static work inside graph](https://github.com/pytorch/pytorch/pull/118962) - live streamed at [https://www.youtube.com/watch?v=06HuwNR9-uI](https://www.youtube.com/watch?v=06HuwNR9-uI) - [Support symbolic min/max on unbacked SymInt](https://github.com/pytorch/pytorch/pull/118953) - [Add TORCHDYNAMO\_EXTENDED\_DEBUG\_GUARD\_ADDED](https://github.com/pytorch/pytorch/pull/118750) - useful debugging tool, get a backtrace only when a specific guard string shows up ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [February 11, 2024, 3:59pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/78 "2024-02-11T15:59:13Z")
# State of PT2: Feb 11, 2024 edition - Follow ups from last week: - Here’s Elina’s doc on PyTorch killswitches/justknobs: [https://docs.google.com/document/d/1Ukerh9\_42SeGh89J-tGtecpHBPwGlkQ043pddkKb3PU/edit#heading=h.hcoj4hguk6we](https://docs.google.com/document/d/1Ukerh9_42SeGh89J-tGtecpHBPwGlkQ043pddkKb3PU/edit#heading=h.hcoj4hguk6we) One thing that is new to me is that I previously thought maybe our PT2 configs should all have corresponding JK for controlling them, but this is not quite right; rather, the configs should stay configs, but their DEFAULTS should be controllable by JK. - Will Feng wrote a status update on FSDP, Meta only: [https://fb.workplace.com/groups/1096365031404923/posts/1100356211005805/](https://fb.workplace.com/groups/1096365031404923/posts/1100356211005805/) - I’ll be spending some time on [Offline log analyzer/formatter · Issue #119063 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/119063), follow along development at [GitHub - meta-pytorch/tlparse: TORCH\_LOGS parser for PT2](https://github.com/ezyang/tlparse) . In particular, check out the design notes at [tlparse/plan.txt at main · meta-pytorch/tlparse · GitHub](https://github.com/ezyang/tlparse/blob/main/plan.txt) Bonus: it’s written in Rust! - Ryan Tremblay reported a worse loss curve when torch.compile’ing a transformers model. This was very difficult to debug, but @bdhirsh eventually pinned it down to a stride mismatch problem in the meta implementation of flash\_att\_bw: [https://github.com/pytorch/pytorch/pull/119500](https://github.com/pytorch/pytorch/pull/119500) Great sleuthing! This can potentially affect all transformer models training with torch.compile, so if you are seeing bad loss please try out the one line fix. - Simon Fan published a very nice update on compiled autograd. The post is meta only: [https://fb.workplace.com/groups/1096365031404923/posts/1099790361062390/](https://fb.workplace.com/groups/1096365031404923/posts/1099790361062390/) but the summary: (1) compiled autograd is now being run on all [PT2 benchmark runs on dashboard](https://hud.pytorch.org/benchmark/compilers?startTime=Tue%2C+30+Jan+2024+18%3A58%3A02+GMT&stopTime=Tue%2C+06+Feb+2024+18%3A58%3A02+GMT&granularity=hour&mode=training&dtype=amp&lBranch=xmfan%2Fcompiled_autograd_perf_no_reuse&lCommit=5a4d9a52103c028f44bdaad675acc45fa9eb53d7&rBranch=main&rCommit=0245000be886a05ca490db8ae6cb4ed0e20a71f9), (2) some backward ops are missing meta implementations that we need to add e.g., aten.\_cudnn\_rnn\_backward, aten.\_embedding\_bag\_dense\_backward, (3) nicely, the speedup 3-11%, now that a refcounting problem in accumulate grad is resolved, (4) less nicely, peak memory usage is up, compile times doubled and CUDA graphs often does not work. On the subject of compiled autograd and DDP, Chien-Chin reports that nanogpt works, passes accuracy and (surprisingly!) performs better even though there is no bucketing yet. However, there’s a bug with zero\_grad where it forces Dynamo to recompile every iteration. Meta only: [https://fb.workplace.com/groups/1096365031404923/posts/1099785997729493/](https://fb.workplace.com/groups/1096365031404923/posts/1099785997729493/) - There’s a new doc on dealing with GuardOnDataDependentSymNode problems: [Dealing with GuardOnDataDependentSymNode errors - Google Docs](https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit#heading=h.44gwi83jepaj) . We also created a new Meta-only working group for support on these issues: [https://fb.workplace.com/groups/6829516587176185](https://fb.workplace.com/groups/6829516587176185) - From Yifu Wang, there’s a new, “native” implementation of functional collectives in Inductor IR and it’s getting close to the point where the old funcols (which are manually implemented with one IR node per collective) can be switched to them. Meta only: [https://fb.workplace.com/groups/1096365031404923/posts/1100109071030519/](https://fb.workplace.com/groups/1096365031404923/posts/1100109071030519/) - Two big happenings on the nested tensor front. First, we have agreed to stop using the term “nested tensor” as it is ambiguous, and instead use NJT (nested tensor with jagged layout) and NST (nested tensor with strided layout) to disambiguate between the two layout formats. Most development is happening on NJT. We devoted the first half of this week’s composability sync to working on nested ints; the final outcome was that (1) nested ints do NOT have device, (2) they test value equality on CPU tensor, and (3) they optionally can cache a CUDA tensor to avoid repeatedly resending it to device. Jeffrey Wan to follow up on this. - The other composability topic was on activation checkpointing. Check the minutes for some information [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit#heading=h.8y2jwyieg2yh) - PT2 and torchbind is in full swing, motivated by need of more complicateed bindings for preproc use cases. Richard Zou and Yidi Wu have been spearheading the effort. A design doc is here: [https://docs.google.com/document/d/179QyhicGzTXJ5jvTAoAosP\_Nzgf3PpgZwU\_E3VV9PlM/edit#heading=h.ykgiz8p1pud](https://docs.google.com/document/d/179QyhicGzTXJ5jvTAoAosP_Nzgf3PpgZwU_E3VV9PlM/edit#heading=h.ykgiz8p1pud) One thing that they think they need is a world token to maintain ordering. - NVIDIA TensorRT team is going to extend torch.\_dynamo.mark\_dynamic to support also specifying min/max constraints. - We historically focus a lot on CUDA performance. Nikita Shulga has been looking into performance of our LLM models against other implementations on CPU. [GitHub - malfet/llm\_experiments](https://github.com/malfet/llm_experiments) One thing he noticed is that our lower precision matmul implementation on ARM is really bad because we didn’t vectorize it. What should you do if you want to use PyTorch but in a deployment environment where you need no dependencies and low binary size? Probably something like Executorch runtime with a thin frontend pasted on top for dealing with KV cache / beam search / stuff that’s hard to export, and then AOTInductor exported blocks that you are pasting together. - PT-D has been on people’s minds. At high level, one of the things folks are most concerned about is composability: distributed involves a lot of complex features, all of which need to work together. I’m not too worried at a fundamental technology level, because many of the things we want to support are orthogonal and independently make sense, but ensuring we have adequate testing of all the combos is something that has persistently plagued our project even in simpler cases. - Some investigating from Richard Zou about how to productionize torchdim. Here, productionize means eliminating the monkey patching it is currently doing to add support for Dim to functions like torch.sum. One potential approach is to make torch function to support non-Tensor arguments having overrides, ala [torch\_function objects passed as non-Tensor args should trigger overrides](https://github.com/pytorch/pytorch/issues/119194) . Another potential is just to directly incorporate the modified logic from torchdim into our C binding code. Both of these are nontrivial work. - Some interesting proposals: - [Free-floating fake tensor without the mode (FakeTensorMode) · Issue #119589 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/119589) - apparently export wants to be able to serialize fake tensors, this is a way to make it make sense (serializing fake tensors with the mode is a lot more fraught) - Notable new bugs: - [If dynamic shapes runtime CUDA graphs never quiesces, should loudly warn / easy to diagnose](https://github.com/pytorch/pytorch/issues/119640) - [Memoize repeated calls to tolist()](https://github.com/pytorch/pytorch/issues/119477) - [Inability to represent size-like expressions causes us to choke when users use list of offsets representation](https://github.com/pytorch/pytorch/issues/119468) - [Compilation failing with name u0 not defined](https://github.com/pytorch/pytorch/issues/119414) - has fix posted - [[dynamic shapes] Symbolic guards with float values](https://github.com/pytorch/pytorch/issues/119338) - [Non-strict export forces specialization when you do tensor[symint:symint] (Dynamo also forces spec when you do narrow)](https://github.com/pytorch/pytorch/issues/119321) - Landed stuff: - [Improve TORCHDYNAMO\_EXTENDED\_DEBUG for GuardOnDataDependentSymNode](https://github.com/pytorch/pytorch/pull/119412) - this debug PR is highly recommended if you are dealing with GuardOnDataDependentSymNode errors, definitely worth rebasing past - [Add symbol\_guard\_limit\_before\_specialize](https://github.com/pytorch/pytorch/pull/119347) - we want to set some default for this, but empirically 100 is too low for TIMM models. More investigating needed. Some instrumentation to take a look at this at [Add symbol guard counts instrumentation](https://github.com/pytorch/pytorch/pull/119290) - [Don’t guard if there are unbacked SymInts](https://github.com/pytorch/pytorch/pull/119312) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [February 16, 2024, 7:57pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/79 "2024-02-16T19:57:35Z")
# State of PT2: Feb 16, 2024 edition Long weekend, so you get the update early this week! And good thing too, this week was jam packed. - We had our monthly execution check in for January. Meta only: [https://fb.workplace.com/groups/401294880339143/?multi\_permalinks=1802470840221533&hoisted\_section\_header\_type=recently\_seen](https://fb.workplace.com/groups/401294880339143/?multi_permalinks=1802470840221533&hoisted_section_header_type=recently_seen) [https://docs.google.com/document/d/15hmpQKSxtq7cmi4cYMikFWQUxm4vjea\_qEes3rE3hOE/edit#heading=h.laybrk1bqzr5](https://docs.google.com/document/d/15hmpQKSxtq7cmi4cYMikFWQUxm4vjea_qEes3rE3hOE/edit#heading=h.laybrk1bqzr5) Some highlights that I can share externally: - Executorch is starting to look at how to handle custom ops, e.g., those in torchvision. Mengwei’s initial thinking is to get people to write their custom ops in ET compatible style, and then adapt them to regular PyTorch dispatcher. [Improving ExecuTorch Custom Ops - Google Docs](https://docs.google.com/document/d/1Z-R31SHiPL_85U9P1pLKuGrn-Oa3ACmegTw1_pt0WWM/edit) - There’s been some agreement to unify on NJT, but Colin pointed out that there is not a completely solid commitment from torchrec side to actually move everything in torchrec to NJT, so there is still some ambiguity here. - Starting to get some coordination on speeding up PT2 compile time. Meta only: [https://docs.google.com/document/d/199LbkPiZdjn2CExEeRl8XDqO4unIPqMEwzyXJljGlY0/edit#heading=h.h2ok4ucbef8y](https://docs.google.com/document/d/199LbkPiZdjn2CExEeRl8XDqO4unIPqMEwzyXJljGlY0/edit#heading=h.h2ok4ucbef8y) ; notably, fake tensor caching has landed! - Flash Attention upgrade is in progress [Update flash\_attention kernel from 2.3.6 to 2.5.6 by drisspg · Pull Request #118935 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/118935) but blocked on CUTLASS upgrade - Ivan Kobzarev is working on torchrec / PT2 compatibility. Meta only: [https://docs.google.com/document/d/166uw-5GotLgwn1SXFyi1Umd2eAXerG\_ofXo2-n-D9Uw/edit](https://docs.google.com/document/d/166uw-5GotLgwn1SXFyi1Umd2eAXerG_ofXo2-n-D9Uw/edit) - Moto Hira has switched teams, so torchaudio is now ramping Ahmad to own the library - Some stuff I found out from 1:1s: - Oguz Oulgen: his team is ramping by fixing bugs in Dynamo, but they’re going to be moving to bigger things soon. The two big themes they’ll be tackling are Dynamo reliability and Dynamo compile time. Oguz is still trying to define what exactly the objectives under these themes will be. - Horace He is working on selective activation checkpointing. Meta only: [https://docs.google.com/document/d/1kNVs5vKkL-CNa2Ufvqex6VgwF8kAqF9aS6ca9pBuB\_8/edit](https://docs.google.com/document/d/1kNVs5vKkL-CNa2Ufvqex6VgwF8kAqF9aS6ca9pBuB_8/edit) but the idea is there will be a level 1 API where you explicitly specify what you want to checkpoint, and a level 2 API where you want to hit some memory budget within the region you’re checkpointing. The level 2 API is difficult to implement without a full graph, so it will likely not be supported in eager mode OR you can compile it, get the list of checkpointing decisions, and then you can feed that into eager mode. This work is based off of a cool prototype fmassa made. Horace hoping this doesn’t take too long, and also still planning to work on single controller, oriented at (1) fine tuning use case and (2) really big cluster use case. - Animesh Jain: should be able to wrap up C++ guards in the next few weeks, next on list is helping with subclass/Dynamo related interactions - Brian Hirsh: looked into how to do the non-overlapping test more efficiently (without doing quadratic pairwise checks) but this is actually kind of cursed. The problem is that if you imagine a 2D parameter buffer, the individual inner parameters correspond to non-overlapping tiles, which means that any single tile is NOT contiguous. Need some sort of multidimensional tiling algorithm. Has been working hard on DTensor / PT2 integration, lots of bugs! But it’s very real. - Avik C: had a chance to brief him on how symbolic shapes works. Looking forward to more collaboration between the eager mode online solver, and export’s offline solver. - Nikita Shulga: improving LLM model perf on CPU. Llama on ARM in fp16 in PyTorch is less than 1 tok/s (compared to 15 tok/s on llama2.cpp). Also looking into gguf compatibility, quantization subclasses, and checking if it works with AOTInductor. - Adnan Akhundov has knocking around supporting cond directly in Inductor, for host side, runtime control flow. WIP. - Composability sync - we sat down and worked out the accumulate grad problem, which is blocking compiled autograd x DDP. The new plan is to decompose accumulate grad before it gets to Inductor. Animesh to look at it. Check out [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit) for notes - Non-strict export is a thing! Non-strict export is an alternate implementation of torch.export.export which uses make\_fx tracing rather than Dynamo tracing. If you are suffering from missing Dynamo support, non-strict can help; but it’s not all beneficial, because make\_fx is unable to handle some patterns that Dynamo can handle transparently (e.g., x[s0:s1] forces specialization on s0 and s1; torch.tensor does not work). A lot of people are using it now. - Someone asked in leads meeting if tensor subclasses are “public” API surface. Well, they are… but you kind of have to be a PyTorch expert to use it. So we don’t necessarily expect them to be self service. - I’ve been working on improving logging inside Meta, with some implications for OSS too. In Meta, I’m trying to get us to stop muxing all our ranks together and write them to separate files [https://docs.google.com/document/d/1tf\_gJ3KlKFqjsTa39yVNymDSu57Lv3mHiWi9dzJJOUk/edit](https://docs.google.com/document/d/1tf_gJ3KlKFqjsTa39yVNymDSu57Lv3mHiWi9dzJJOUk/edit) (this requires some new API support in torchrun). I’ve also been working on a logparser called tlparse [GitHub - meta-pytorch/tlparse: TORCH\_LOGS parser for PT2](https://github.com/ezyang/tlparse) . I want to do a lot of things with it, but right now what it does is it looks at all the compilations that happened and gives you a nice overview of where they are, by rendering all their call stacks into a trie. Still rapidly evolving. It’s pretty fast too: 500MB/s, which is not the fastest but means a few GB logs is no big deal. - Notable new bugs in dynamic shapes - [PGO-style mode automatic dynamic shapes](https://github.com/pytorch/pytorch/issues/120080) - [Also attempt to match weaker conditions when replacing deferred runtime asserts](https://github.com/pytorch/pytorch/issues/119969) - [Complicated guards probably are being repeatedly issued](https://github.com/pytorch/pytorch/issues/119917) - [torch.compile doesn’t convert all input scalar types to symbolic values](https://github.com/pytorch/pytorch/issues/119778) - this is the same thing as converting float compute into tensor compute - [Deferred runtime assert is silently elided if unbacked symint gets replaced](https://github.com/pytorch/pytorch/issues/119689) - Landed stuff from Edward: - [Change default TORCH\_LOGS format to match Meta/glog standard](https://github.com/pytorch/pytorch/pull/119869) - [Rewrite maybe\_reduce more carefully for unbacked SymInt](https://github.com/pytorch/pytorch/pull/119562) - [Prevent DCE’ing unbacked SymInt for view outputs](https://github.com/pytorch/pytorch/pull/119552) ---
**Author:** ![ezyang](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ezyang/32/12_2.png) [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)\ **Post date:** [February 24, 2024, 10:26pm UTC](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777/80 "2024-02-24T22:26:06Z")
# State of PT2: Feb 24, 2024 edition - A lot of progress on torchrec and PT2 this week. - We got some clarity on how exactly to deal with the naive implementations on KJT embedding that torchrec uses by default. These are quite inefficient, and we do not actually need to trace them as post-export we will be unflattening modules and doing module swaps. We’ve come to the agreement that it’s not necessary to directly trace these (which is very slow); instead, we will somehow trace some higher level thing (either by tracing table batched code, or putting in dummy functions for the things to be module swapped.) Internal xref: [https://fb.workplace.com/groups/6829516587176185/posts/6880175418776968](https://fb.workplace.com/groups/6829516587176185/posts/6880175418776968) - Flavio has been working on fbgemm metas. These are annoying because the fused optimizers are all code generated in C++, so it’s most convenient to do the meta in C++. Good progress: [https://github.com/pytorch/FBGEMM/pull/2347](https://github.com/pytorch/FBGEMM/pull/2347) - Paul successfully used non-strict export on the model he was working on. Summary at (Meta-only) [https://docs.google.com/document/d/1dB\_8-RL3Mm9sD8wqW\_9dbHTQw2JVeeug9teU2F\_DCu4/edit#heading=h.5teu622jmqei](https://docs.google.com/document/d/1dB_8-RL3Mm9sD8wqW_9dbHTQw2JVeeug9teU2F_DCu4/edit#heading=h.5teu622jmqei) . Adding tolist support to non-strict export: [[PT2] tolist() support for FunctionalTensor by PaulZhang12 · Pull Request #120508 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/120508) Still some problems involving overspecialization to be looked into (internal xref: [https://fb.workplace.com/groups/6829516587176185/posts/6883282961799547](https://fb.workplace.com/groups/6829516587176185/posts/6883282961799547)). - Ivan has been working on tracing through torchrec async comms, rewriting them so they are in synchronous style without custom autograd function and therefore PT2 traceable. Interesting blocker: derivatives are missing on functional collectives; also some necessary primitives like reduce\_scatter\_v are missing - Colin is going to move into the dynamic shapes space per Adnan. - ghstack 0.9.0 is out. This release has a big new feature: you can now specify a subset of commits in your stack. Simply say `ghstack submit HEAD~` to push only HEAD~ (but not HEAD). Commit ranges are also supported. There’s also a number QoL improvements: we now no longer include the PR title inside our generated head/base commits, we no longer strip @ from email addresses in commit messages, and there’s a smoother GitHub auth token flow. Most open issues in our bug tracker were fixed. There’s also an experimental `--direct` feature which lets you generate PRs that merge directly into main, but it’s not tested to work with pytorchbot, this is mostly useful if you’re using ghstack on your own repository. Internal xref: [https://fb.workplace.com/groups/533197713799375/posts/1839730956479371/](https://fb.workplace.com/groups/533197713799375/posts/1839730956479371/) - Structured logging: - MVP is out at [Add structured trace logs by ezyang · Pull Request #120496 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/120496) ! After generating the logs, you can parse them with [GitHub - meta-pytorch/tlparse: TORCH\_LOGS parser for PT2](https://github.com/ezyang/tlparse) . It’s not a great choice for local development (you can use `TORCH_COMPILE_DEBUG` for that), but it’s really handy if you’re running remote jobs and the only way to conveniently get information out of them is via logging. - Meta-only: I’m waiting on Kurman for an alternate implementation in torchrun that will help us conveniently avoid interleaving multiple ranks of logs together. ETA Monday. - Composability sync was pretty spicy: Jason Ansel reopened the question of whether or not we want to be supporting a pre-dispatch serialization IR at all. [Composability meeting notes - Google Docs](https://docs.google.com/document/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/edit#heading=h.8y2jwyieg2yh) . - The composability minutes don’t have the full story: after the meeting, there was a bunch of back channeling with Horace and Jason, and we got back to “OK, I guess we need to support pre-dispatch IR”. One major thing that convinced Jason was that we don’t actually play on exporting FSDP-ized models (instead, the model will be unflattened post export and the FSDP applied at that point). For Horace, solving the org problem of needing to move your model around to different transforms and not at all in one go seemed insoluble without the export format. Horace to enumerate all the cases where we do side tables that you can’t export with pre dispatch: checkpointing and user defined Triton kernels. - On a side note, there was some late night discussion about what to do about subclasses and pre-dispatch export. My explanation to Michael Suo was that if a subclass can only be desugared after autograd, then your pre dispatch IR must include the subclass, and your target runtime must know how to deal with the subclass. In some sense, this is not surprising at all, because pre dispatch IR really hasn’t had any of PyTorch’s internal subsystems resolved ahead of time, so you are going to need, e.g., a full on autograd engine to actually run it (in practice, these export IRs are going to target PyTorch again). Sometimes, subclasses can be implemented before autograd, but you often give up quite a bit to do so. For example, for DTensor to be pre-autograd, would give up the capability to have a different sharding pattern between forwards and backwards; for NestedTensor to be pre-autograd, necessitates every operation nested tensor to have a separate ATen operator with its own derivative formula (as many nested tensor operations cannot be implemented just be desugaring). In some sense, this is not surprising: people are using ` __torch_dispatch__ ` because there are things you can’t do unless you’re below autograd! - Per-parameter FSDP is very serious business. With some upcoming training use cases that I cannot describe here in public, per-parameter FSDP is in serious contention for being the distribution mechanism. Will Feng has been focused on torch.compile’ing per-parameter FSDP (internal xref: [https://fb.workplace.com/groups/1096365031404923/posts/1108181883556571/](https://fb.workplace.com/groups/1096365031404923/posts/1108181883556571/)), and generating more AOTAutograd features that Brian has been helping consult with. - I had a chance to ask Brian what was going on with FP8 and DTensor composability. There is progress going on here (contrary to my impression), and it seems the current design problems revolve around cases where first DTensor then FP8 ordering wants to be violated. In particular, in some cases DTensor wants to do something special when a conversion to FP8 happens. The current thinking is that `to_fp8` will be a dedicated ATen operator that DTensor can override, and we just need to figure out how to dispatch this to the FP8 subclass (similar to backend select, we have a dispatch problem since no argument to the operator actually takes in an FP8 tensor) before we finally get to proxy tensor (since we don’t want `to_fp8` to show up in the final, desugared of subclasses graph.) Brian is consulting on this. - The state of NestedTensor has been on my mind. - Jeffrey Wan has a fairly major change to nested int out: [https://github.com/pytorch/pytorch/pull/119976](https://github.com/pytorch/pytorch/pull/119976) The implementation in the PR is actually different than the design Christian advocated for in the meeting, where sequences are canonically CPU but can be cached on CUDA. The main problem with Christian’s design is if you are given only a CUDA lengths tensor, Christian’s design as is forces an immediate sync to establish the CPU source of truth. More discussion necessary. - Joel Schlosser has subclass view-ification out [Subclass view fake-ification via reified ViewFuncs by jbschlosser · Pull Request #118405 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/118405) It seems pretty close, just needs some detail work. - Basil Hosmer’s old [fold](https://github.com/bhosmer/fold/) prototype was back in the news. The context is that I finally internalized something Michael Suo was telling me, which is that stock NJT cannot handle torchrec KJT, which requires two dimensions before jagged dim. How exactly should this be modeled with nested ints that Jeffrey is working on? If a nested int corresponds to a Seq dim in Basil’s formulation, fold tells us pretty directly how to model this: `(feature, batch, jagged, embedding)`, where jagged is a Seq dim. It is a little unfortunate that the stride note was never written, but with only a single jagged dimension, intuitive multiplication of Seq with int does what you want. Not sure if anyone else is on board with this, need to get more alignment. One thing fold doesn’t answer is how you should resolve degrees of freedom of if Seq’s data is stored on CPU or solely on CUDA. torchrec makes one particular choice, but there are others! Ivan tells me, however, that typically you don’t want to store stuff on CPU if you can avoid it. - Michael Suo has been pushing us to think more about the long term state of NJT and torchrec. This is not in the plans right now, but in the long term, we would ideally have NJT be the backend representation for JT and KJT. But torchrec has pretty stringent eager performance considerations, so it is not at all clear how you are ever going to actually manage this. This is somewhat reminiscent of the situation with complex tensors, where we have a C++ implementation, but for PT2 a Python implementation would be much preferable (but we can’t get rid of the C++ implementation because eager perf would suffer.) - Oguz/Chip thinking about super big jobs. When you have so many nodes, when one node fails you’re going to have to restart. This is going to happen a lot. This means warm start matters a lot, and the model is not changing. We’ve already got a memcache thing going on for compiled Triton kernels: think bigger. - Richard has been thinking more about custom ops; he has a new custom op API proposal [https://github.com/pytorch/pytorch/pull/120345/](https://github.com/pytorch/pytorch/pull/120345/) . One of the tender design questions is “why do people want to put plain PyTorch operations in their custom operator, the so-called traceable operator”? Reference this old document for some use cases: [PT2 black box escape hatch - Google Docs](https://docs.google.com/document/d/1rVg1tRInZcwWknsrVQwCxHsQhjOfxTgEzr4g9tJRMGI/edit#heading=h.40qxbvm41rpv) Another interesting idea that popped up: if someone puts a Triton kernel in their CUDA implementation of a custom op, how do we get this to Inductor in a way that it can understand and directly incorporate the Triton in, without generating an actual call to the custom op? One idea is for operators to have a more “structural” implementation, e.g., TritonKernel, FXKernel, where it’s not an opaque Python callable and you can poke at it from the compiler to get the important information. And you could automatically generate these by using Dynamo. Food for thought. - Torchbind fakeification is making progress by Yidi Wu at [https://github.com/pytorch/pytorch/pull/120045](https://github.com/pytorch/pytorch/pull/120045) . There’s some API bikeshedding on what exactly the API for getting the fake version of a torchbind object and testing its guard should be. - James March was complaining to me that OSS C++ logging has no timestamps on NCCL logs like “watchdog timeout”. This probably is not hard to fix, someone should check it out. - Animesh Jain has been steadily working on C++ guards, it’s a big stack of PRs that is slowly landing. He is planning to move into the accumulate grad question, which we had discussed in composability last week. We worked out some more implementation details: fixing .grad handling in Dynamo is probably the tall pole in the tent, desugaring of accumulateGrad will happen in Dynamo via a polyfill, we will never handle the refcount == 1 case. - Mengwei Liu has a proposal for letting you load inline Executorch kernels into regular PyTorch [Google Colab](https://colab.research.google.com/drive/1T_Q0G49MCSv-pklniM62Wrf2Ix4hOe0N#scrollTo=HqpSyLPpNz4c) - Chip’s been thinking about big training jobs! Some Meta only docs to look at: [https://docs.google.com/document/d/1gN8UmuqBxTX0MROSFkApjHFY0jzeE\_tMTbuRPoxgWHk/edit](https://docs.google.com/document/d/1gN8UmuqBxTX0MROSFkApjHFY0jzeE_tMTbuRPoxgWHk/edit) also [https://fburl.com/gdoc/yngic2wj](https://fburl.com/gdoc/yngic2wj) - There’s a Dynamo bug burn down coming up soon, organized by Jane Xu and Richard Zou. Meta only: [https://fb.workplace.com/groups/257735836456307/permalink/638928315003722/](https://fb.workplace.com/groups/257735836456307/permalink/638928315003722/) - Xiaodong is worried that DTensor is too focused on parallelism in llama, and there are other contexts which it is not well adapted to. - lucidrains will “be available in San Francisco for contracting, private tutoring, or full-time hire in March 2024”, per his website. - I didn’t pay much attention to the weekly compile time meeting but it seems like stuff is happening. Minutes at: [https://docs.google.com/document/d/199LbkPiZdjn2CExEeRl8XDqO4unIPqMEwzyXJljGlY0/edit?usp=sharing](https://docs.google.com/document/d/199LbkPiZdjn2CExEeRl8XDqO4unIPqMEwzyXJljGlY0/edit?usp=sharing) - Notable bugs everywhere: - [https://github.com/pytorch/pytorch/pull/119868](https://github.com/pytorch/pytorch/pull/119868) multiple folks ran into this (manifesting as making a fake tensor when you already have a fake tensor), thanks Laith for working on the fix! - [Dynamo is silently suppressing shape guards from asserts · Issue #118417 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/118417) this has been a long standing issue but someone complained about it again on GitHub so Tugsuu actually going to look into this - [Change default torch\_function behavior to be disabled when torch\_dispatch is defined by albanD · Pull Request #120539 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/120539) Alban is finally removing the “default torch function is not what you want with torch dispatch” footgun - Horace complaining TORCH\_COMPILE\_DEBUG doesn’t work reliably, if you have relevant bug reports send them in. - Brian has been looking into a strange SymNode tracing problem [[test fix] try to re-use cached symnodes across dynamo and AOTAutograd by bdhirsh · Pull Request #120090 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/120090) The logic here is all kind of cursed, not really clear what the right approach is. One idea is that sympy expressions actually could just always be constructed from scratch, removing dependence on tensors (which has caused problems before); this is similar to how Inductor IR does it. - Notable bugs in symbolic shapes: - [Tensors with multiple unbacked SymInt dims don’t work in Inductor](https://github.com/pytorch/pytorch/issues/120548) - [Second forward call of a compiled model (exact same input shapes, strides) is extremely slow due to cuda graphs](https://github.com/pytorch/pytorch/issues/120309) - this is not a bug per se, but a case where we do exactly what you asked us to do, which is sometimes not what you want (tracing each dynamic size separately for cuda graphs) - Some more user requests got synthesized into these potential improvements for symbolic reasoning: [Equivalent idea to size-oblivious guard for end of bounds on sizes](https://github.com/pytorch/pytorch/issues/120288) [Perhaps u1 - u0 should be size-like when u1 and u0 are size-like and u1 \>= u0](https://github.com/pytorch/pytorch/issues/120286) - Landed stuff from Edward - [[Dynamo] Handle guard\_size\_oblivious in user code](https://github.com/pytorch/pytorch/pull/120379) - [Properly trace into mark\_static](https://github.com/pytorch/pytorch/pull/120232) - this one was embarrassing, the original test was not written appropriately - [Fix missing right square bracket to match glog format](https://github.com/pytorch/pytorch/pull/119966) - ooops! [Previous page](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777.md?page=3) [Next page](https://dev-discuss.pytorch.org/t/state-of-symbolic-shapes-branch/777.md?page=5)