# Leveraging Large Language Models for Exploiting ASR Uncertainty

**Authors:** Pranay Dighe, Yi Su, Shangshang Zheng, Yunshu Liu, Vineet Garg, Xiaochuan Niu, Ahmed Tewfik  
**arXiv:** [2309.04842](https://arxiv.org/abs/2309.04842)  
**Date:** 2023-09-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.04842v1/

## Key points

- Large language models (LLMs) have shown great performance in natural language processing (NLP) tasks.
- LLMs face challenges in spoken language understanding (SLU) tasks due to accuracy limitations of automatic speech recognition (ASR) systems.
- This work aims to improve the accuracy of LLMs on SLU tasks by addressing high word-error-rate in ASR systems.
- The authors propose a novel approach of prompting the LLM with an n-best list of ASR hypotheses instead of relying solely on the 1-best hypothesis.
- Prompt-engineering techniques are used to explain the concept of n-best lists to the LLM.
- Low-Rank Adapters are introduced for fine-tuning on downstream tasks.
- By leveraging n-best lists and using Low-Rank Adapters, their approach enhances performance on device-directed speech detection and keyword spotting tasks.
- Systems that use n-best list prompts outperform those using only 1-best ASR hypotheses.
- This research demonstrates an efficient method to exploit ASR uncertainty via LLMs in speech-based applications.
- Significant improvements can be achieved without major modifications to existing ASR and LLM models.
- This approach has potential implications for multiple unrelated tasks that can benefit from shared ASR and LLM resources.

## Summary

In the field of natural language processing (NLP), large language models (LLMs) have shown great performance in various tasks. However, when it comes to spoken language understanding (SLU) tasks, LLMs face challenges due to the accuracy limitations of automatic speech recognition (ASR) systems used for transcription. This work focuses on improving the accuracy of LLMs on SLU tasks by addressing the issue of high word-error-rate in ASR systems, which can hinder the LLM's ability to understand spoken intent. Instead of designing complex architectures or specialized models that may incur high deployment costs, the authors aim to explore how far they can improve SLU performance without substantially changing the underlying ASR and LLM models. To achieve this, they propose a novel approach of prompting the LLM with an n-best list of ASR hypotheses instead of relying solely on the error-prone 1-best hypothesis. The concept of n-best lists is explained to the LLM through prompt-engineering techniques. The authors also introduce Low-Rank Adapters for fine-tuning on downstream tasks. By leveraging n-best lists and using Low-Rank Adapters, their approach proves effective in enhancing performance on device-directed speech detection and keyword spotting tasks. Systems that utilize n-best list prompts outperform those using only 1-best ASR hypotheses. This research paves the way for an efficient method to exploit ASR uncertainty via LLMs in speech-based applications. It demonstrates that by incorporating n-best lists into prompt engineering and utilizing Low-Rank Adapters, significant improvements can be achieved without major modifications to existing ASR and LLM models. This approach has potential implications for multiple unrelated tasks that can benefit from shared ASR and LLM resources. Overall, this study highlights a promising direction for leveraging large language models to overcome ASR uncertainties and enhance spoken language understanding capabilities in various applications.