# Extending Llama-3's Context Ten-Fold Overnight

**Authors:** Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, Zhicheng Dou  
**arXiv:** [2404.19553](https://arxiv.org/abs/2404.19553)  
**Date:** 2024-04-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.19553v1/

## Key points

- Successfully extended context length of Llama-3-8B-Instruct from 8K to 80K through QLoRA fine-tuning
- Explored Multi-Detail QA tasks with homogeneous and heterogeneous contexts, as well as Biography Summarization tasks with context lengths between 64K to 80K
- Training dataset included question-answer pairs from multi-turn conversations and instances from RedPajama and LongAlpaca datasets
- Model fine-tuned using QLoRA with LoRA rank set to 32 and alpha to 16, achieving remarkable results on downstream long-context tasks
- Released resources, including model, training data, and code, are publicly available for further research in training long-context LLMs

## Summary

, , , , 

The team successfully extended the context length of Llama-3-8B-Instruct from 8K to 80K through QLoRA fine-tuning, resulting in superior performance across various evaluation tasks. They explored Multi-Detail QA tasks involving homogeneous and heterogeneous contexts, as well as Biography Summarization tasks with context lengths between 64K to 80K. The training dataset included question-answer pairs organized in multi-turn conversations and instances from RedPajama and LongAlpaca datasets. The model was fine-tuned using QLoRA with LoRA rank set to 32 and alpha to 16, achieving remarkable results on downstream long-context tasks. The released resources, including the model, training data, and code, are publicly available for further research in training long-context LLMs. Additionally, the model was evaluated on LongBench and InfiniteBench benchmarks, showcasing consistent outperformance compared to baselines except for code completion tasks. Further improvements may involve mixing more code data during training.