# Inductor updates

**URL:** <https://dev-discuss.pytorch.org/t/inductor-updates/1301>\
**Category:** compiler\
**Created:** [June 1, 2023, 2:58am UTC](https://dev-discuss.pytorch.org/t/inductor-updates/1301 "2023-06-01T02:58:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ngimel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/ngimel/32/73_2.png) [@ngimel](https://dev-discuss.pytorch.org/u/ngimel)\
**Post date:** [June 1, 2023, 2:58am UTC](https://dev-discuss.pytorch.org/t/inductor-updates/1301/1 "2023-06-01T02:58:54Z")

</div>

We started this year with approximately 1.5x speedup on HuggingFace with cudagraphs, and as of [today](https://hud.pytorch.org/benchmark/compilers?startTime=Tue%2C%2023%20May%202023%2021%3A34%3A35%20GMT&stopTime=Tue%2C%2030%20May%202023%2021%3A34%3A35%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=main&lCommit=999bae0f54108ffc5b7cf2524a02a83901554b16&rBranch=main&rCommit=bf059e3925a2565cb05a407f541cc11d762eef05) we stand at 1.70x. There have been several improvements to inductor over this timeframe, not focused solely on performance

1. Cudagraphs are now using a new implementation that brought cudagraph memory usage to be on par with non-cudagraphs, thus allowing us to enable cudagraphs much more readily

2. We have robust [utilities](https://docs.google.com/document/d/1LEzlLnnB_HHEKh0Epo_pkiwi3TYmHa6skmMxQxkKAMI/edit) that allows us to measure and analyze kernel performance

3. We have coordinate descent max autotuning algorithm that allows us to explore wider search space to compile faster kernels

4. We are now regularly updating timm and huggingface suites to make sure we are benchmarking the latest versions of the models. Upstream frequently updates model definitions, and in many cases those changes are beneficial for performance, e.g. they remove graph breaks or use fast implementation of scaled dot product from the core. These changes are now reflected in the benchmarks

5. We now [don’t break](https://github.com/pytorch/pytorch/pull/99483) on custom autograd functions (this is dynamo improvement, not inductor, but it’s critical for performance). This has been an often requested feature and this is bringing dramatic (2x) speed-ups to Deberta models

6. We are working on automatic application of a few optimizations that can be achieved by changes to user models, however, since users are usually reluctant to make those changes, out-of-the-box performance is worse than it could be

- We are working towards [automatic replacement](https://github.com/pytorch/pytorch/issues/101458) of attention patterns with efficient implementation from core

- [Automatic padding](https://github.com/pytorch/pytorch/pull/101915) of the matmul shapes brings approximately 3% across HF models. It utilizes the pattern matcher infrastructure mentioned above

- Automatic conversion to [channels-last](https://github.com/pytorch/pytorch/pull/99773) layout is expected to bring significant improvements to Timm benchmark suite

1. [Worked around](https://github.com/pytorch/pytorch/pull/102444) triton perf bug to improve HF performance by 4%

2. We’ve enabled checks on out-of-bounds accesses for indexing ops that previously resulted in either silent wrong results or IMAs. Since indexing ops are also widely used in decompositions of pooling/upsampling operations where indices are guaranteed to be in bounds, we make sure to not generate the checks for the ops that don’t require them to limit perf impact

3. Quansight contributors implemented flexible tl.reduce in triton which allowed us to implement min/max/argmin/argmax operations that match eager semantics, previously we were incorrectly propagating nans and not returning same values when there were multiple min/max candidates. 1-pass Welford variance calculation using tl.reduce is in the works

4. We’ve enabled codegen for [for\_each](https://github.com/pytorch/pytorch/pull/99975) ops that will allow us to generate fully fused optimizers. Currently only Adam and AdamW fully fused optimizers are supported in core, we will make this support much more flexible with torch.compile

5. We are improving capabilities of AOT inductor to support real models and dynamic shapes. Low latency AOT inductor will be very important for LLM inference, and I’m glad we are making progress here.