# The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction

**Authors:** Pratyusha Sharma, Jordan T. Ash, Dipendra Misra  
**arXiv:** [2312.13558](https://arxiv.org/abs/2312.13558)  
**Date:** 2023-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.13558v1/

## Key points

- Transformer-based Large Language Models (LLMs) are popular in machine learning
- Selectively removing higher-order components from weight matrices can enhance LLM performance
- This intervention is known as LAyer-SElective Rank reduction (LASER)
- LASER can be applied after model training without additional parameters or data
- Extensive experiments validate LASER's effectiveness in improving LLM performance
- LASER challenges the conventional approach of increasing model size and training data for better performance
- Reducing higher-order components can lead to significant improvements in LLMs
- These findings have important implications for language modeling advancements and optimizing LLMs for better reasoning capabilities.

## Summary

, , , , 

In the field of machine learning, Transformer-based Large Language Models (LLMs) have gained popularity due to their ability to process large amounts of data and continuous improvement by researchers. However, a recent study has revealed a surprising finding - selectively removing higher-order components from weight matrices can enhance LLM performance. This intervention, known as LAyer-SElective Rank reduction (LASER), can be applied after model training without additional parameters or data. Extensive experiments across various language models and datasets validate LASER's effectiveness in improving LLM performance. The results also provide insights into when LASER is most effective and its underlying mechanism. This challenges the conventional approach of increasing model size and training data for better LLM performance, suggesting that reducing higher-order components can lead to significant improvements. These findings have important implications for language modeling advancements and optimizing LLMs for better reasoning capabilities.