# Towards General Text Embeddings with Multi-stage Contrastive Learning

**Authors:** Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang  
**arXiv:** [2308.03281](https://arxiv.org/abs/2308.03281)  
**Date:** 2023-08-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.03281v1/

## Key points

- GTE is a general-purpose text embedding model trained using multi-stage contrastive learning.
- The authors aim to unify various NLP tasks into a single format by training a unified text embedding model through contrastive learning over diverse datasets from multiple sources.
- By increasing the amount of training data during unsupervised pre-training and supervised fine-tuning stages, the authors achieve substantial performance improvements over existing embedding models.
- GTE$_\text{base}$ outperforms OpenAI's black-box embedding API and larger text embedding models by 10 times on the massive text embedding benchmark, even with a modest parameter count of 110M.
- GTE outperforms previous best code retrievers of similar size without additional fine-tuning on each programming language individually, highlighting its capability to handle code-related tasks effectively.
- Multi-stage contrastive learning is the key factor behind these impressive results.
- GTE offers a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.

## Summary

The paper introduces GTE, a general-purpose text embedding model that is trained using multi-stage contrastive learning. The authors aim to unify various NLP tasks into a single format by training a unified text embedding model through contrastive learning over diverse datasets from multiple sources. By significantly increasing the amount of training data during both unsupervised pre-training and supervised fine-tuning stages, the authors achieve substantial performance improvements over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTE$_\text{base}$ outperforms OpenAI's black-box embedding API and even surpasses larger text embedding models by 10 times on the massive text embedding benchmark. Moreover, without additional fine-tuning on each programming language individually, GTE outperforms previous best code retrievers of similar size by treating code as text. This highlights the model's capability to handle code-related tasks effectively. The key factor behind these impressive results is the effective utilization of multi-stage contrastive learning. By harnessing this approach, GTE offers a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks. In conclusion, the paper presents GTE as an advanced text embedding model that achieves state-of-the-art performance by leveraging multi-stage contrastive learning. Its ability to handle diverse NLP tasks and perform well in code-related tasks makes it a valuable tool for various applications in natural language processing and programming.