# Improving Inference Performance of Machine Learning with the Divide-and-Conquer Principle

**Authors:** Alex Kogan  
**arXiv:** [2301.05099](https://arxiv.org/abs/2301.05099)  
**Date:** 2023-01-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.05099v1/

## Key points

- The paper addresses the issue of poor scalability of machine learning models when deployed on CPUs.
- The authors propose a novel approach based on the Divide-and-Conquer Principle to tackle this problem.
- Instead of allocating all available computing resources to the entire problem, they suggest breaking it into smaller chunks and letting the framework decide how computing resources should be allocated among those chunks.
- The proposed allocation mechanism is implemented in OnnxRuntime, a popular framework for training and inferencing ML models.
- The effectiveness of this approach is demonstrated with several use cases, including highly popular models for image processing (PaddleOCR) and NLP tasks (BERT).
- Section 2 elaborates on various reasons why inference commonly does not scale well on CPUs.
- In Section 3, the authors describe in detail the concept and implementation details of their proposed Divide-and-Conquer Principle as it applies to inference.
- Section 4 presents several use cases where this principle can be applied along with performance evaluation results demonstrating its benefits.
- Their approach allows efficient batching of inference requests of various sizes eliminating the need for padding and letting the framework allocate computing resources proportionally to the length of each sequence.
- Related work is discussed in Section 5 before concluding in Section 6.

## Summary

In this paper, the authors address the issue of poor scalability of machine learning models when deployed on CPUs. They propose a novel approach based on the Divide-and-Conquer Principle to tackle this problem. Instead of allocating all available computing resources to the entire problem, they suggest breaking it into smaller chunks and letting the framework decide how computing resources should be allocated among those chunks. The authors argue that in many use cases, such a division is natural and requires only trivial changes in user code. The proposed allocation mechanism is implemented in OnnxRuntime, a popular framework for training and inferencing ML models. The inference API is extended to allow user code to invoke parallel inference on multiple inputs. The effectiveness of this approach is demonstrated with several use cases, including highly popular models for image processing (PaddleOCR) and NLP tasks (BERT). In Section 2, the authors elaborate on various reasons why inference commonly does not scale well on CPUs. One reason is that the amount of computation required by a model during inference may not be "enough" for efficient parallelization. In Section 3, they describe in detail the concept and implementation details of their proposed Divide-and-Conquer Principle as it applies to inference. Section 4 presents several use cases where this principle can be applied along with performance evaluation results demonstrating its benefits. For instance, their approach allows efficient batching of inference requests of various sizes eliminating the need for padding and letting the framework allocate computing resources proportionally to the length of each sequence. In Section 5, related work is discussed before concluding in Section 6. Overall, this paper provides an insightful solution to address poor scalability issues faced by machine learning models when deployed on CPUs using a simple yet effective approach based on Divide-and-Conquer Principle.