# A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency

**Authors:** Tara N. Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, Chung-Cheng Chiu, David Garcia, Alex Gruenstein, Ke Hu, Minho Jin, Anjuli Kannan, Qiao Liang, Ian McGraw, Cal Peyser, Rohit Prabhavalkar  
**arXiv:** [2003.12710](https://arxiv.org/abs/2003.12710)  
**Date:** 2020-03-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2003.12710v2/

## Key points

- End-to-end (E2E) models have not surpassed conventional models in speech recognition quality and latency
- A recent study by Tara N. Sainath and her team developed an E2E model that surpasses conventional models in both aspects
- The researchers incorporated a large number of utterances from varied domains to increase acoustic diversity and vocabulary exposure
- They trained the model with accented English speech to make it more robust to different pronunciations
- Varied learning rate schedule was explored due to increased training data
- The team used end-of-sentence decision emitted by the RNN-T model to close the microphone and introduced various optimizations to improve LAS rescoring speed for latency improvement
- RNN-T+LAS model offers better tradeoff between word error rate (WER) and latency compared to conventional models, with an 8% relative improvement in WER while being over 400 times smaller in model size for the same latency.
- This study presents an on-device E2E speech recognition system that outperforms server-side conventional models in terms of both quality and latency, which could have significant implications for real-time applications such as virtual assistants or voice-controlled devices where low latency performance is crucial for user experience.

## Summary

In the field of speech recognition, end-to-end (E2E) models have not yet surpassed state-of-the-art conventional models in terms of both quality and latency. However, a recent study by Tara N. Sainath and her team has developed a first-pass Recurrent Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell (LAS) rescorer that surpasses conventional models in both aspects. To improve the quality of their model, the researchers incorporated a large number of utterances from varied domains to increase acoustic diversity and vocabulary exposure. They also trained the model with accented English speech to make it more robust to different pronunciations. Additionally, they explored using a varied learning rate schedule due to the increased amount of training data. On the latency front, the team explored using the end-of-sentence decision emitted by the RNN-T model to close the microphone and introduced various optimizations to improve LAS rescoring speed. The resulting RNN-T+LAS model offers a better tradeoff between word error rate (WER) and latency compared to conventional models. For example, for the same latency, RNN-T+LAS obtains an 8% relative improvement in WER while being over 400 times smaller in model size. Overall, this study presents an on-device E2E speech recognition system that outperforms server-side conventional models in terms of both quality and latency. This could have significant implications for real-time applications such as virtual assistants or voice controlled devices where low latency performance is crucial for user experience.