What's the difference between torch.export / torchserve / executorch / aotinductor?

youkaichao · November 7, 2023, 8:55am

I’m watching and learning from pytorch conference 2023, when I encountered these concepts. It seems they are all related to deploy models.

To the best of my understanding:

torch.export is a functionality in pytorch, to export a computation graph to some pre-defined format. This corresponds to the function torch.export.
torchserve is a framework to serve pytorch models at scale. It might use models from torch.export. This corresponds to a repo GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production .
executorch is another framework to run pytorch models for mobile/edge devices. It might use models from torch.export. This corresponds to a repo GitHub - pytorch/executorch: End-to-end solution for enabling on-device AI across mobile and edge devices for PyTorch models .
AOTInductor, from the talk https://www.youtube.com/watch?v=w7d4oWzwZ0c and slides , it seems to be an ahead-of-time version of inductor.

The most confusing thing to me is AOTInductor. It seems to be more of a concept. I don’t know if it is a new backend, or a new repo, or a new functionality.

Can anyone help me out? It would be great to provide some concrete code to show what is AOTInductor.

And any other help for understanding torch.export/torchserve/executorch would also be great!

youkaichao · November 7, 2023, 8:58am

Might be relevant to @desertfire .

gmagogsfm · November 13, 2023, 5:25pm

Hi Youkai, your understanding is pretty much all correct, only a few minor corrections.

torch.export is not a function, instead it is a Python module containing torch.export.export (I know …) and other utilities that export needs.
executorch always needs model from torch.export instead of “might use”
I am a bit behind on updates on AoTInductor. It is a prototype now and it is a new functionality to run TorchInductor in Ahead-of-time fashion. I don’t think it will be a new repo. @desertfire will have the definitive answer here.

desertfire · November 13, 2023, 6:20pm

Glad to see you are interested in AOTInductor. It is in a prototype stage and is being actively developed. AOTInductor: Ahead-Of-Time Compilation for Torch.Export-ed Models — PyTorch main documentation contains an example on how to use it. Please give it a try and let me know if you have any questions.

youkaichao · November 14, 2023, 1:16am

Thanks, that’s clear! So AOTInductor’s entry point is torch._export.aot_compile, and it is part of the export system to export the model as a standalone .so file.

jeromeku · November 29, 2023, 8:30pm

Thanks for the great talk at PyTorch Conf!

Quick question on the interplay between torch.script and AOTInductor – can you explain what the two tests here are testing? I.e., the difference between an AOT-compiled module that is scripted vs. one that is not, benefits / use-cases?