Thank you a lot for the detailed explanation. Having a libtorch delegate would be an extremely elegant way to preserve TorchScript’s device-agnostic strengths while making migration to an ExecuTorch workflow easier.
I assume some libtorch-compatible graph representation would need to be serialized with the ExecuTorch program to facilitate this. I’d be very interested to hear @mergennachin’s thoughts on what such a delegate and its serialization format might look like.
Torch export and the “Edge IR” being used there is actually already fully libtorch compatible. It is using a subset of the libtorch aten API.
You can see this in the “ATen mode” that partially exists in the current library.
I am not sure where it stands today (especially in the CMake-based build) and if the team would be happy to accept improvements for it.
That is interesting… But as far as I know there is not way for libtorch to consume torch.exported artifacts right now, correct? That would be the missing piece I guess.
PS: If I am not mistaken, Support loading and executing a ExportedProgram from torch.export in C++ environment · Issue #144663 · pytorch/pytorch · GitHub is the issue to track.
In this case, libtorch wouldn’t be consuming the exported artefact directly indeed. The ExecuTorch runtime is the one consuming such exported program, orchestrating running the model and it would rely on libtorch to provide the aten ops only.
This is for sure a tradeoff and not a perfect solution to your ask above. The main tradeoff being that the core team can focus on a single runtime consuming the exported artefact (hopefully making it more feature complete and reliable). But, some hard choices have to be made for that runtime (in particular how much flexibility is allowed and how much we specialize for slim runtime).
I see. So the libtorch libraries would be shipped along the ExecuTorch libs like e.g. onnxruntime libraries? And ExecuTorch would simply delegate ops to libtorch.
That is correct.
From checking with @digantdesai , it looks like it only works on the buck based build today though. But I think the team would be happy to support reasonable improvements there if you need!
That would be great! How can we contribute? We have an in-house database of 50+ models that are continuously exported with torchscript. If we could hook up the ExecuTorch / Libtorch workflow with this we should be able to provide meaningful reports.
The best place for reports and feedback is the issue tracker on the repo: Issues · pytorch/executorch · GitHub
That is also a good place to report missing documentation or onboarding material!
For faster pace discussion, you can send me an email via private message here and I can invite you to the PyTorch slack!