# Magicoder: Source Code Is All You Need

**Authors:** Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang  
**arXiv:** [2312.02120](https://arxiv.org/abs/2312.02120)  
**Date:** 2023-12-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.02120v1/

## Key points

- Introduces Magicoder, a series of fully open-source Large Language Models (LLMs) for code
- Aim to close the gap with top code models
- Models have no more than 7B parameters and are trained on 75K synthetic instruction data using OSS-Instruct
- OSS-Instruct incorporates open-source code snippets to generate high-quality instruction data for code
- Motivation is to address bias in synthetic data generated by LLMs by providing them with open-source references
- Magicoder and MagicoderS outperform state-of-the-art code models across various coding benchmarks
- Benchmarks include Python text-to-code generation, multilingual coding, and data science program completion
- MagicoderS based on CodeLlama surpasses ChatGPT on HumanEval+ in terms of pass@1 score (66.5 vs 65.9)
- OSS-Instruct enables low bias and high-quality instruction tuning using abundant open source references

## Summary

The paper introduces Magicoder, a series of fully open-source Large Language Models (LLMs) for code that aim to close the gap with top code models. These models have no more than 7B parameters and are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach that incorporates open-source code snippets to generate high-quality instruction data for code. The main motivation behind this work is to address the inherent bias in synthetic data generated by LLMs by providing them with a wealth of open-source references, resulting in more diverse, realistic, and controllable data. The authors highlight the orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct, which enables the development of an enhanced version called MagicoderS. Both Magicoder and MagicoderS outperform state-of-the-art code models with similar or larger sizes across various coding benchmarks. These benchmarks include Python text-to-code generation, multilingual coding, and data science program completion. Notably, MagicoderS based on CodeLlama even surpasses ChatGPT on HumanEval+ in terms of pass@1 score (66.5 vs 65.9). Overall, OSS-Instruct opens up new possibilities for low bias and high quality instruction tuning using abundant open source references. The authors of the paper are Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding and Lingming Zhang.