# Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs

**Authors:** Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, Sachin Dev Sharma  
**arXiv:** [2410.10739](https://arxiv.org/abs/2410.10739)  
**Date:** 2024-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.10739v1/

## Key points

- Continuous pre-training is crucial for optimal performance of Large Language Models (LLMs)
- LLMs are introduced as Base LLMs and instruction-refined LLMs
- Determining which model should undergo continuous pre-training is a challenge
- The study explores the relationship between continuous pre-training and instruction fine-tuning of LLMs
- Instruction fine-tuning requires hand-annotated examples and is computationally intensive
- The goal is to identify a compute-efficient strategy to improve instruction-following capabilities without specific instruction data
- Datasets used for tuning include LLaMa, Qwen, and other state-of-the-art (SoTA) LLMs
- Evaluation dataset categorization includes language understanding, reasoning, problem-solving, truthfulness assessment, etc.
- A comprehensive test dataset from EleutherAI ensures unbiased evaluation results and reproducibility
- Manual scraping of new articles was conducted to avoid data contamination concerns
- The study provides empirical evidence on the impact of continuous pre-training on maintaining up-to-date knowledge and improving instruction-following capabilities in LLMs

## Summary

In the realm of Large Language Models (LLMs), the need for continuous pre-training is crucial for optimal performance. This ensures that models stay up-to-date with the latest data and have accurate instruction-following capabilities. LLMs are typically introduced in two versions: Base LLMs, which are pre-trained on a diverse set of data, and instruction-refined LLMs, which undergo additional training with specific instructions to enhance their instruction-following abilities. The challenge lies in determining which model should undergo continuous pre-training to maintain both its instruction-following skills and stay current with new data. This study delves into the intricate relationship between continuous pre-training and instruction fine-tuning of LLMs. It aims to investigate how continuous pre-training impacts the instruction-following abilities of both base models and their instruction-finetuned counterparts. Instruction fine-tuning is computationally intensive and requires a significant number of hand-annotated examples for effective learning. The goal is to identify the most compute-efficient strategy to acquire up-to-date knowledge and improve instruction-following capabilities without relying on specific instruction data and fine-tuning. To address this challenge, the study explores datasets used for tuning LLaMa, Qwen, and other state-of-the-art (SoTA) LLMs. It's important to note that these instruction fine-tuning datasets are not publicly shared due to confidentiality reasons. In this work, "instruction following capabilities" and "instruction capabilities" are used interchangeably. The evaluation dataset categorization includes various subcategories such as language understanding, reasoning and problem-solving, truthfulness assessment, factual knowledge evaluation through benchmarks like IFEval, MMLU-Pro, GSM8K, Winogrande among others. To ensure unbiased evaluation results and reproducibility,a comprehensive test dataset is utilized using an evaluation harness framework from EleutherAI. This dataset focuses on assessing different capabilities including Instruction Following Evaluation (IFEval), which consists of verifiable instructions aimed at testing natural language instruction following abilities of LLMs across various metrics. Furthermore, in order to continuously pre-train models without data contamination concerns from existing datasets as highlighted by previous studies like Jiang et al., manual scraping of approximately 2 million articles was conducted using a static news crawler called FUNDUS. These articles were specifically selected based on being new to LLaMa 3.1 models within a specified date range from December 2023 to September 2024. Overall, this study provides empirical evidence on the impact of continuous pre-training on maintaining up-to-date knowledge and improving instruction-following capabilities in LLMs across different settings and sizes of pre-training data corpus for base models as well as their refined counterparts.