# LLM in a flash: Efficient Large Language Model Inference with Limited Memory

**Authors:** Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar  
**arXiv:** [2312.11514](https://arxiv.org/abs/2312.11514)  
**Date:** 2023-12-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.11514v1/

## Key points

- Large language models (LLMs) are crucial for natural language processing and perform well in various tasks.
- LLMs have high computational and memory requirements, which pose challenges for devices with limited DRAM capacity.
- The authors propose a method to efficiently run LLMs exceeding the available DRAM capacity by storing model parameters on flash memory and bringing them on demand to DRAM.
- Their approach involves constructing an inference cost model that aligns with the behavior of flash memory.
- They introduce two techniques: "windowing" reduces data transfer by reusing previously activated neurons, while "row-column bundling" increases the size of data chunks read from flash memory.
- By combining these methods, they can run models up to twice the size of available DRAM, achieving significant improvements in inference speed on both CPU and GPU platforms.
- Additional features include integrating sparsity awareness and context-adaptive loading techniques into their design.
- Overall, this work presents an effective solution for running large language models on devices with limited memory resources.

## Summary

Large language models (LLMs) play a crucial role in natural language processing, delivering exceptional performance in various tasks. However, their computational and memory requirements pose challenges for devices with limited DRAM capacity. To address this issue, the authors propose a method that efficiently runs LLMs exceeding the available DRAM capacity by storing model parameters on flash memory and bringing them on demand to DRAM. The key idea behind their approach is to construct an inference cost model that aligns with the behavior of flash memory. This cost model guides optimization efforts in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this framework, they introduce two principal techniques: "windowing" and "row-column bundling". The first technique, windowing, strategically reduces data transfer by reusing previously activated neurons. By doing so, they minimize the amount of information that needs to be retrieved from flash memory. The second technique, row-column bundling, takes advantage of the sequential data access strengths of flash memory. It increases the size of data chunks read from flash memory, further improving efficiency. By combining these methods, the authors demonstrate that it is possible to run models up to twice the size of the available DRAM. They achieve significant improvements in inference speed compared to naive loading approaches on both CPU and GPU platforms. Specifically, they report a 4-5x increase in inference speed on CPU and a 20-25x increase on GPU. The authors also highlight additional features incorporated into their approach. They integrate sparsity awareness into their design to exploit any inherent sparsity present in LLMs. They also employ context-adaptive loading techniques to optimize resource usage based on specific contextual factors. Overall, this work presents an effective solution for running large language models on devices with limited memory resources. By leveraging flash memory and implementing optimized strategies for data transfer and retrieval, they achieve impressive gains in inference speed and enable the deployment of larger models on resource-constrained devices.