# How Many Instructions Can LLMs Follow at Once?

**Authors:** Daniel Jaroslawicz, Brendan Whiting, Parth Shah, Karime Maamari  
**arXiv:** [2507.11538](https://arxiv.org/abs/2507.11538)  
**Date:** 2025-07-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.11538v1/

## Key points

- Study focused on performance of language models in adhering to multiple instructions simultaneously
- Introduced IFScale benchmark with 500 keyword-inclusion instructions for business report writing task
- Tested 20 state-of-the-art models from seven major providers, best models achieved only 68% accuracy at maximum instruction density
- Some models maintained coherence as instruction density increased, while outliers like o3 and o4-mini showed marked decreases in coherence
- Model output token counts influenced coherence levels, smaller outputs led to decreased coherence
- Identified distinct degradation patterns in model performance based on instruction density
- Emphasized importance of selecting models based on application requirements and reliability considerations for mission-critical applications
- Research sheds light on challenges and trade-offs of using language models in high-density instruction environments

## Summary

Our study delved into the performance of language models in adhering to multiple instructions simultaneously. This is a crucial aspect for production-grade systems and we introduced IFScale, a benchmark consisting of 500 keyword-inclusion instructions for a business report writing task. Our analysis involved testing 20 state-of-the-art models from seven major providers and found that even the best models achieved only 68% accuracy at the maximum instruction density. Further investigation into core task performance revealed interesting insights. While most models maintained coherence or showed only slight declines as instruction density increased, outliers like o3 and o4-mini exhibited marked decreases in coherence. These findings suggest that certain models may struggle with core task performance when focused on instruction adherence. Additionally, model output token counts played a significant role in coherence levels, with smaller outputs leading to decreased coherence. Our study also identified distinct degradation patterns in model performance based on instruction density, providing valuable information for deploying language models in instruction-heavy scenarios. We highlighted the importance of selecting models based on application requirements and reliability considerations, emphasizing the need for consistent performers in mission-critical applications. In conclusion, our research sheds light on the challenges and trade-offs associated with using language models in high-density instruction environments. By understanding these limitations and insights, practitioners can make informed decisions when implementing LLMs for real-world applications and work towards targeted improvements in model performance.