# In-Context Learning Creates Task Vectors

**Authors:** Roee Hendel, Mor Geva, Amir Globerson  
**arXiv:** [2310.15916](https://arxiv.org/abs/2310.15916)  
**Date:** 2023-10-24  
**Summary page:** https://www.summarizepaper.com/fr/paper/2310.15916v1/

## Abstract

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set $S$ to find a best-fitting function $f(x)$ in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query $x$ and a single "task vector" calculated from the training set. Thus, ICL can be seen as compressing $S$ into a single task vector $\boldsymbol{\theta}(S)$ and then using this task vector to modulate the transformer to produce the output. We support the above claim via comprehensive experiments across a range of models and tasks.