# Learning without training: The implicit dynamics of in-context learning

**Authors:** Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, Javier Gonzalvo  
**arXiv:** [2507.16003](https://arxiv.org/abs/2507.16003)  
**Date:** 2025-07-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.16003v1/

## Key points

- Study focuses on in-context learning in Large Language Models (LLMs)
- Incorporates self-attention layer with a Multi-Layer Perceptron (MLP) within a transformer block
- Demonstrates that MLP layer weights can be modified according to context
- Enables LLMs to adapt and learn new patterns in context without additional weight updates at inference time
- Finetuning the transformer model using stochastic gradient descent with specific order of examples processed in-context, updating only the weight matrix of the MLP layer
- Convergence analysis shows relative change in weights converges to zero as more context is processed
- Highlights effectiveness of leveraging context for enhancing learning capabilities of LLMs beyond traditional training methods
- Acknowledges limitations and suggests areas for further research

## Summary

The study delves into the implicit dynamics of in-context learning in Large Language Models (LLMs), focusing on their ability to learn new patterns without additional weight updates at inference time. By incorporating a self-attention layer with a Multi-Layer Perceptron (MLP) within a transformer block, the researchers demonstrate that the weights of the MLP layer can be modified according to the context. This mechanism enables LLMs to adapt and learn in context even when presented with patterns not seen during training. Through theoretical analysis and experimentation, the study suggests that this simple yet effective mechanism allows LLMs to transform context into low-rank weight updates of the MLP layer. By finetuning the transformer model using stochastic gradient descent with a specific order of examples processed in-context and only updating the weight matrix of the MLP layer, the researchers show how LLMs can dynamically adjust their weights based on contextual information. The convergence analysis reveals that as more context is processed, the relative change in weights converges to zero, indicating that LLMs can effectively learn and adapt to new patterns presented in-context. In conclusion, this study highlights the effectiveness of leveraging context for enhancing learning capabilities beyond traditional training methods for Large Language Models. However, it also acknowledges limitations and areas for further research. Overall, this work sheds light on how LLMs can utilize contextual information for improved performance and showcases its potential for future advancements.