# System 2 Attention (is something you might need too)

**Authors:** Jason Weston, Sainbayar Sukhbaatar  
**arXiv:** [2311.11829](https://arxiv.org/abs/2311.11829)  
**Date:** 2023-11-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.11829v1/

## Key points

- Soft attention in Transformer-based Large Language Models (LLMs) allows models to focus on different parts of the input context when generating responses.
- Soft attention is prone to incorporating irrelevant information into latent representations, negatively impacting next token generations.
- System 2 Attention (S2A) is a new attention mechanism proposed to address this issue.
- S2A leverages reasoning and language understanding capabilities of LLMs to determine relevant parts of the context for attention.
- S2A regenerates the input context by selecting only relevant portions before applying attention, ensuring meaningful information is considered during response generation.
- S2A outperforms traditional attention mechanisms in opinion or irrelevant information detection, question answering (QA), math word problems, and longform generation tasks.
- S2A improves factuality and objectivity in generated responses while reducing sycophancy.
- S2A helps LLMs produce more accurate and objective outputs by filtering out irrelevant or biased information from the context.

## Summary

Soft attention in Transformer-based Large Language Models (LLMs) is a technique that allows the models to focus on different parts of the input context when generating responses. However, this approach is prone to incorporating irrelevant information into its latent representations, which can negatively impact the quality of next token generations. To address this issue, the authors propose a new attention mechanism called System 2 Attention (S2A). S2A leverages the reasoning and language understanding capabilities of LLMs to determine what parts of the context are relevant and should be attended to. The key idea behind S2A is to regenerate the input context by selecting only the relevant portions before applying attention. By doing so, S2A ensures that only meaningful information is considered during response generation. In their experiments, the authors compare S2A with standard attention-based LLMs on three different tasks: opinion or irrelevant information detection, question answering (QA), math word problems, and longform generation. The results show that S2A outperforms traditional attention mechanisms in all these tasks. Specifically, S2A improves factuality and objectivity in generated responses while reducing sycophancy (excessive praise or flattery). This indicates that S2A helps LLMs produce more accurate and objective outputs by filtering out irrelevant or biased information from the context. Overall, this study introduces System 2 Attention as an effective solution for addressing issues related to soft attention in Transformer-based Large Language Models. By leveraging natural language reasoning and instruction-following abilities of LLMs, S2A enables better control over what information is attended to during response generation. The experimental results demonstrate its superiority over standard attention mechanisms across various tasks involving opinion detection, QA, math word problems, and longform generation.