# CodeBERT: A Pre-Trained Model for Programming and Natural Languages

**Authors:** Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou  
**arXiv:** [2002.08155](https://arxiv.org/abs/2002.08155)  
**Date:** 2020-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2002.08155v1/

## Key points

- CodeBERT is a pre-trained model for programming language (PL) and natural language (NL)
- It can be used in various NL-PL applications, such as code search and code documentation generation
- CodeBERT uses a Transformer-based neural architecture and a hybrid objective function
- It can identify plausible alternatives generated by other models through replaced token detection
- CodeBERT can utilize both bimodal data (NL-PL pairs) and unimodal data to improve performance
- It achieves state-of-the-art results in natural language code search and code documentation generation tasks
- CodeBERT outperforms previous pre-trained models in NL-PL probing tasks
- It offers general purpose representations that excel in various NL-PL applications
- CodeBERT is valuable for developers and researchers in the field of programming language and natural language tasks.

## Summary

CodeBERT is a pre-trained model designed to handle both programming language (PL) and natural language (NL). It aims to learn versatile representations that can be used in various NL-PL applications, including code search and code documentation generation. The model is built using a Transformer-based neural architecture and trained with a hybrid objective function which incorporates the task of replaced token detection, allowing it to identify plausible alternatives generated by other models. One of the key advantages of CodeBERT is its ability to utilize both bimodal data (NL-PL pairs) and unimodal data. The bimodal data provides input tokens for training the model, while the unimodal data helps improve the performance of the generators. By combining these two types of data, CodeBERT achieves state-of-the-art results in natural language code search and code documentation generation tasks. To gain insights into the knowledge learned by CodeBERT, researchers construct a dataset for NL-PL probing and evaluate its performance in a zero-shot setting where parameters are fixed. The results demonstrate that CodeBERT outperforms previous pre-trained models in NL-PL probing. Overall, CodeBERT offers a powerful solution for handling both programming language and natural language tasks due to its general purpose representations which enable it to excel in various NL-PL applications, making it an invaluable tool for developers and researchers in the field.