# Serving DNNs like Clockwork: Performance Predictability from the Bottom Up

**Authors:** Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, Jonathan Mace  
**arXiv:** [2006.02464](https://arxiv.org/abs/2006.02464)  
**Date:** 2020-06-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2006.02464v2/

## Key points

- Machine learning inference is crucial for interactive web applications
- Model serving systems are needed to meet low latency targets
- Existing architectures struggle with tail latency caused by unpredictable execution times
- Deep Neural Network (DNN) models exhibit deterministic performance during inference
- Clockwork is a fully distributed model serving system designed for predictable end-to-end performance
- Clockwork supports thousands of models while meeting 100ms latency targets for 99.9999% of requests
- Clockwork leverages predictable execution times to achieve tight request-level service-level objectives (SLOs)
- The study was conducted by authors Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace

## Summary

Machine learning inference is becoming increasingly crucial for interactive web applications. This has led to the need for model serving systems that consistently meet low latency targets. While existing architectures use reactive techniques to address common sources of latency, they struggle to mitigate tail latency caused by unpredictable execution times. However, it has been observed that Deep Neural Network (DNN) models exhibit deterministic performance during inference. Leveraging this predictability, a principled design methodology was adopted to develop Clockwork - a fully distributed model serving system that ensures predictable end-to-end performance. Clockwork was evaluated using production trace workloads and demonstrated its ability to support thousands of models while meeting 100ms latency targets for 99.9999% of requests. Notably, Clockwork leverages predictable execution times to achieve tight request-level service-level objectives (SLOs) and provides a high degree of request-level performance isolation. This study was conducted by authors Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace as presented in their paper "Serving DNNs like Clockwork: Performance Predictability from the Bottom Up" at the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI '20).