# Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity

**Authors:** Germán Kruszewski, Pierre Erbacher, Jos Rozen, Marc Dymetman  
**arXiv:** [2512.05962](https://arxiv.org/abs/2512.05962)  
**Date:** 2025-12-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2512.05962v1/

## Key points

- Reinforcement Learning (RL) used for fine-tuning Large Language Models (LLMs)
- Drawback of RL models: reduction in diversity due to mode-seeking KL divergence
- Proposed solution: starting from an explicit target distribution derived by filtering incorrect answers while preserving relative probabilities of correct ones
- Approach involves approximating refined target distribution using $α$-divergence family for control over precision and diversity
- New methodology tested on Lean theorem-proving benchmark, showcasing state-of-the-art performance along coverage-precision Pareto frontier

## Summary

, , , , 

In the realm of artificial intelligence, Reinforcement Learning (RL) has emerged as the go-to technique for fine-tuning Large Language Models (LLMs) to tackle tasks that require reasoning abilities. However, recent findings have indicated a notable drawback associated with models trained using RL – a significant reduction in diversity. This phenomenon is attributed to RL's implicit optimization of the "mode-seeking" or "zero-forcing" Reverse Kullback-Leibler (KL) divergence towards a target distribution. As a result, the model tends to concentrate its focus on specific high-probability regions of the target distribution while neglecting others. To address this issue, a novel approach is proposed in this study. Instead of relying on RL and its inherent biases, the researchers advocate for starting from an explicit target distribution. This distribution is derived by filtering out incorrect answers while preserving the relative probabilities of correct ones. Leveraging a pre-trained LLM as the foundation, the team approximates this refined target distribution using the $α$-divergence family. This family of divergences offers a unified framework that allows for direct control over the trade-off between precision and diversity by smoothly transitioning between mode-seeking and mass-covering divergences. The efficacy of this new methodology is put to the test on a Lean theorem-proving benchmark. The results speak volumes about its success, showcasing state-of-the-art performance along the coverage-precision Pareto frontier. Notably, this approach surpasses all previous methods when evaluated on the coverage axis, demonstrating superior capabilities in balancing both coverage and precision in reasoning tasks. Authored by Germán Kruszewski, Pierre Erbacher, Jos Rozen, and Marc Dymetman, this research paper titled "Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity" sheds light on an innovative strategy to enhance diversity and performance in LLMs through targeted filtering techniques and divergence-based optimization strategies.