Running summarizing tools on a new article

Abstract: Large Language Models (LLMs) demonstrate exceptional general-purpose language processing capabilities, but they require extremely high memory and computational resources. Quantization techniques are crucial for deploying LLMs by improving memory efficiency and inference speed. Existing activation quantization methods are primarily dynamic, which, while achieving low quantization error, offer relatively limited inference acceleration. In contrast, static quantization, which provides faster inference, faces the challenge of significant quantization error. To address this, we propose AASQ-GPT, a novel technique that enables accurate activation static quantization without the need for retraining. Specifically, AASQ-GPT introduces a new quantization framework for activations, where after quantizing a batch of activations, the remaining unquantized activations are adjusted to minimize quantization error. Compared to traditional min-max quantization, AASQ-GPT significantly improves quantization perfor- mance without incurring any additional overhead. Experiments on LLMs of varying scales show that AASQ-GPT achieves state-of-the- art (SOTA) results in static quantization at the same bit width.