# Using Python for Model Inference in Deep Learning

**Authors:** Zachary DeVito, Jason Ansel, Will Constable, Michael Suo, Ailing Zhang, Kim Hazelwood  
**arXiv:** [2104.00254](https://arxiv.org/abs/2104.00254)  
**Date:** 2021-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.00254v1/

## Key points

- Python is the preferred language for training complex neural networks due to its scientific computing libraries and tensor computation frameworks like PyTorch and TensorFlow.
- Transitioning from model training to inference can be challenging, requiring models to be extracted into formats like TensorFlow graphs or TorchScript programs.
- Researchers have proposed a solution for seamless model inference within Python by leveraging multiple Python interpreters in a single process.
- A new container format for models encapsulates native Python code and data, streamlining deployment without extraction steps.
- The innovative approach allows for scalable inference without sacrificing performance or packaging constraints.
- Rigorous evaluation on popular PyTorch models shows that the packaged Python models deliver performance comparable to TorchScript for larger models and maintain scalability even with some overhead for smaller models.
- This research simplifies the model deployment pipeline and enhances integration of performance optimizations within the Python ecosystem.

## Summary

In the realm of deep learning, Python has emerged as the go-to language for training complex neural networks. Its extensive array of scientific computing libraries combined with efficient tensor computation frameworks like PyTorch and TensorFlow make it a preferred choice for researchers and developers alike. However, transitioning from model training to inference can present challenges. One common issue is the need to extract models from Python into formats such as TensorFlow graphs or TorchScript programs in order to meet performance and packaging requirements. This extraction process can be cumbersome and time-consuming, hindering rapid prototyping efforts. To address this issue, a group of researchers led by Zachary DeVito, Jason Ansel, Will Constable, Michael Suo, Ailing Zhang, and Kim Hazelwood have proposed an innovative solution that allows for seamless model inference within Python itself. By leveraging multiple Python interpreters within a single process, they have devised a method to achieve scalable inference without sacrificing performance or packaging constraints. Central to their approach is the development of a new container format for models that encapsulates both native Python code and data. This novel format streamlines the model deployment process by eliminating the need for extraction steps while facilitating integration with existing performance-enhancing Python libraries. Through rigorous evaluation on a selection of popular PyTorch models sourced from Github, the team demonstrates how these models can be packaged in their unique inference format. Comparative analysis reveals that for larger models, their packaged Python models deliver performance on par with TorchScript. Moreover, even in scenarios where there is some overhead due to Python interpretation for smaller models, their multi-interpreter strategy ensures that inference remains scalable. Overall,this research not only simplifies the model deployment pipeline but also opens up avenues for enhanced integration of performance optimizations within the Python ecosystem.