# Hy are AOTI packaged models faster than torch.compile models with the same inputs?

**URL:** <https://dev-discuss.pytorch.org/t/hy-are-aoti-packaged-models-faster-than-torch-compile-models-with-the-same-inputs/3240>\
**Category:** compiler\
**Created:** [October 2, 2025, 9:28pm UTC](https://dev-discuss.pytorch.org/t/hy-are-aoti-packaged-models-faster-than-torch-compile-models-with-the-same-inputs/3240 "2025-10-02T21:28:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![skv66](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/skv66/32/2885_2.png) [@skv66](https://dev-discuss.pytorch.org/u/skv66)\
**Post date:** [October 2, 2025, 9:28pm UTC](https://dev-discuss.pytorch.org/t/hy-are-aoti-packaged-models-faster-than-torch-compile-models-with-the-same-inputs/3240/1 "2025-10-02T21:28:38Z")

</div>

Hi all,

I’ve been experimenting with both `torch.compile` and AOTI compiled packaged models.  
Here’s my setup:

- Same model architectures (e.g., ResNet18, etc.)

- Same input shapes (e.g., `[1, 3, 224, 224]`)

- Both run in `eval()` mode, `torch.no_grad()`, CUDA device, and warmed up before timing

However, when I benchmark them:

- The **AOTI compiled and packaged models** (`torch._inductor.aoti_compile_and_package` + `torch._inductor.aoti_load_package`) run noticeably faster.

- The **`torch.compile` models** (using the Inductor backend) show higher per-iteration runtime, even though the kernels should be the same.

Questions:

- Why is there a performance gap between AOTI packaged models and `torch.compile` models when everything else (model, input, device) is identical?
- Is there a way to visualize the graph of compiled models?

Thanks,  
Shriraj

---

<div class="post-metadata">

**Author:** ![desertfire](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/desertfire/32/980_2.png) [@desertfire](https://dev-discuss.pytorch.org/u/desertfire)\
**Post date:** [October 3, 2025, 1:00am UTC](https://dev-discuss.pytorch.org/t/hy-are-aoti-packaged-models-faster-than-torch-compile-models-with-the-same-inputs/3240/2 "2025-10-03T01:00:57Z")

</div>

AOTI generates C++ wrapper code, which means it can get rid of Python overhead for the model code. ResNet18 is a small model and your input is small, so AOTI’s benefit tends to be more noticeable.

BTW, you can run with `TORCH_LOGS=output_code` to check the generated code from both, to double check if there is any disparity in the generated kernel code.

---

<div class="post-metadata">

**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\
**Post date:** [October 9, 2025, 5:24am UTC](https://dev-discuss.pytorch.org/t/hy-are-aoti-packaged-models-faster-than-torch-compile-models-with-the-same-inputs/3240/3 "2025-10-09T05:24:30Z")

</div>

The most likely culprit here is your model is too small to saturate the GPU, so it ends up being overhead bound (rather than compute bound like a larger model would be). AOTI has lower overhead, so it is faster – while if you scale up your model to be larger they will run the same speed.

I’d suggest enabling cudagraphs with `torch.compile(mode=”reduce-overhead”)`, which will help a lot for tiny models.

You could also try using a much larger batch size, so your model has more compute per kernel.