# Text and Code Embeddings by Contrastive Pre-Training

**Authors:** Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson  
**arXiv:** [2201.10005](https://arxiv.org/abs/2201.10005)  
**Date:** 2022-01-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.10005v1/

## Key points

- Deep unsupervised learning with generative and embedding models has shown significant success
- Models have reduced the need for labeled training datasets and benefited various downstream applications
- Text and code embeddings achieved through contrastive pre-training on unsupervised data at scale have demonstrated state-of-the-art results in linear-probe classification, text search, and code search tasks
- Utilizing large batch sizes during training has led to high-quality vector representations of text and code
- Potential avenues to offset computational costs include providing safe public access to pre-trained language models and implementing efficient training pipelines
- Addressing biased representations is crucial to mitigate risks of representational harm
- Contrastive pre-training on unsupervised data generates high-quality text and code embeddings, though underperformance was observed in sentence similarity tasks
- Ethical considerations are important when developing AI technologies

## Summary

, , , , 

In recent years, deep unsupervised learning with generative and embedding models has shown significant success. These models have been instrumental in reducing the need for labeled training datasets and have benefited various downstream applications. In this work, we focus on text and code embeddings achieved through contrastive pre-training on unsupervised data at scale. The developed models have demonstrated state-of-the-art results in linear-probe classification, text search, and code search tasks by maximizing the likelihood of observed data and distinguishing it from noise. By utilizing large batch sizes during training, high-quality vector representations of text and code have been obtained. Despite the computational resources required for training these embedding models, there are potential avenues to offset these costs while still allowing users to benefit from their capabilities. One approach is providing safe public access to large pre-trained language models and implementing efficient training pipelines that leverage improved model architectures and training schemes. This research also highlights the importance of addressing issues related to biased representations that could influence resource allocation and opportunities for individuals. It emphasizes the need for further exploration and implementation efforts in these areas to mitigate any potential risks of representational harm. Overall, this study underscores the impact of contrastive pre-training on unsupervised data in generating high-quality text and code embeddings. While achieving remarkable results in various tasks such as linear-probe classification and semantic search, there were observations of underperformance on sentence similarity tasks. The broader societal implications of this work are also discussed, emphasizing the importance of considering ethical considerations when developing AI technologies.