# Transformers as Support Vector Machines

**Authors:** Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, Samet Oymak  
**arXiv:** [2308.16898](https://arxiv.org/abs/2308.16898)  
**Date:** 2023-08-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.16898v1/

## Key points

- Transformer architecture has revolutionized natural language processing (NLP)
- Attention layer allows input tokens to interact through pairwise similarities
- Study establishes a formal equivalence between self-attention and Support Vector Machine (SVM) problem
- Optimizing attention layer converges towards an SVM solution that minimizes nuclear norm of combined parameter $W=KQ^\top$
- Gradient descent exhibits local/global directional convergence under suitable geometric conditions
- Over-parameterization plays a crucial role in catalyzing global convergence
- Findings applicable to diverse datasets and verified through experiments
- Comprehensive optimization-theoretic analysis of the attention model
- Max-margin equivalence provides fundamental insights into optimization geometry of transformers
- Open problems and research directions for learning rate selection, initialization, and optimization methods
- Understanding implicit bias of gradient descent under margin constraints presents exciting research direction

## Summary

In recent years, the transformer architecture has revolutionized natural language processing (NLP) since its introduction in "Attention Is All You Need". The attention layer within the transformer allows input tokens to interact through pairwise similarities computed using softmax$(XQK^\top X^\top)$, where $(K,Q)$ are trainable key-query parameters. In this study, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin Support Vector Machine (SVM) problem. By characterizing the implicit bias of 1-layer transformers optimized with gradient descent, we make several important findings. Firstly, when optimizing the attention layer with vanishing regularization parameterized by $(K,Q)$, it converges towards an SVM solution that minimizes the nuclear norm of the combined parameter $W=KQ^\top$. On the other hand, directly parameterizing by $W$ minimizes a Frobenius norm objective. We highlight that this convergence can occur towards locally-optimal directions rather than global ones. Secondly, we prove that gradient descent under suitable geometric conditions exhibits local/global directional convergence. Over-parameterization plays a crucial role in catalyzing global convergence by ensuring feasibility of the SVM problem and guaranteeing an optimization landscape devoid of stationary points. While our theory primarily applies to linear prediction heads, we propose a more general SVM equivalence that predicts implicit bias with nonlinear heads. Our findings are applicable to diverse datasets and have been verified through experiments. This study contributes to existing research on transformers by offering a comprehensive optimization-theoretic analysis of the attention model. Unlike previous works that rely on strict statistical assumptions about data, our findings encompass different problem geometries and datasets. The max-margin equivalence provides fundamental insights into the optimization geometry of transformers and serves as a framework for future research endeavors. Moving forward, there are several open problems and research directions to explore which can provide valuable insights into learning rate selection, initialization and optimization methods for further exploration. Additionally, understanding the implicit bias of gradient descent under margin constraints presents an exciting research direction in formalizing this relationship between transformers and hard-margin SVM which can facilitate generalization analysis in traditional deep learning similar to kernel/NTK analysis for deep neural networks enabling generalization analysis in traditional deep learning . Overall, our study deepens our understanding of transformers and language models by providing insights into their underlying principles which opens up new avenues for research and paves way for further advancements in this field.