What is the correct, future-proof, way of deploying a pytorch python model in C++ for inference?

deployment
11

Hi @raphael10-collab thanks for giving ExecuTorch a try. The workflow is as follows:

  • Load your .safetensor into your BERT model (torch.nn.Module)
from safetensors.torch import load_model

load_model(model, "model.safetensors")
# Instead of model.load_state_dict(load_file("model.safetensors"))
  • Use torch.export() to export the torch.nn.Module into a ExportedProgram. You need to prepare the example input and dynamic shape information. See this wiki for instructions. Some code examples:
import torch

ep = torch.export.export(model, example_args)
  • Then you would need to use ExecuTorch APIs to export the ExportedProgram into .pte files. Please find the instruction here. You can look at the example in export_hf_util.py (may not directly apply to your case), and your code will look similar to this:
from executorch.exir import to_edge
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

program = to_edge(ep).to_backend(XnnpackPartitioner()).to_executorch() 
filename = "model.pte"
with open(filename, "wb") as f:
      program.write_to_file(f)
  • Once you have the model.pte you can run it using ./cmake-out/executor_runner:
./cmake-out/executor_runner --model_path model.pte

Hope this helps. You can also create issues in GitHub · Where software is built or join our discord channel!