# A Lightweight Framework for High-Quality Code Generation

**Authors:** Mohammed Latif Siddiq, Beatrice Casey, Joanna C. S. Santos  
**arXiv:** [2307.08220](https://arxiv.org/abs/2307.08220)  
**Date:** 2023-07-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.08220v1/

## Key points

- Increase in use of automated source code generation using transformer-based generative models
- Generated source code can contain vulnerabilities and quality issues
- FRANC is a lightweight framework for recommending secure and high-quality source code
- FRANC includes a static filter to ensure compilability and a quality-aware ranker for sorting code snippets
- Prompt engineering techniques are used to fix persistent quality issues
- Evaluation results show improvements in compilability and ranking of code snippets
- FRANC does not require retraining or fine-tuning of language models, reducing costs associated with modifying existing models
- Previous studies have focused on fine-tuning or modifying the model itself, while FRANC offers a lightweight solution without these requirements.

## Summary

In recent years, there has been an increase in the use of automated source code generation using transformer-based generative models. These models are capable of generating functional code based on developers' requirements. However, recent research has shown that the automatically generated source code can contain vulnerabilities and other quality issues. Despite efforts to improve code generation models, retraining and fine-tuning large language models is time-consuming and resource-intensive. To address these challenges, the researchers propose FRANC, a lightweight framework for recommending more secure and high-quality source code derived from transformer-based code generation models. FRANC includes a static filter that ensures the generated code is compilable by applying heuristics. It also incorporates a quality-aware ranker that sorts the code snippets based on their quality scores. Additionally, prompt engineering techniques are used to fix persistent quality issues. The framework was evaluated using five Python and Java code generation models and six prompt datasets, including a newly created one called SOEval. The results showed that the static filter improved Java suggestions by 9% to 46% regarding compilability and Python suggestions by 10% to 43%. The average improvement over the NDCG@10 score for the ranking system was 0.0763, indicating better ranking of high-quality code snippets. The repairing techniques were able to fix the highest 80% of prompts. One notable aspect of FRANC is that it does not require retraining or fine-tuning of language models. Instead, it filters out vulnerable and low-quality code from the model's output without modifying the model itself. This approach reduces costs associated with modifying existing models while still addressing quality issues. In related work, previous studies have focused on improving code generation models through various methods such as property-specific continuous vectors or incorporating human feedback during training. However, these approaches either require fine-tuning or modification of the model itself. In contrast, FRANC provides a lightweight solution that filters out vulnerable and low-quality code without the need for retraining or modifying the original model. In conclusion, FRANC offers a novel framework for enhancing the quality of source code generated by transformer-based models. It provides a static filter, quality-aware ranking system, and prompt engineering techniques to address vulnerabilities and other quality issues. The evaluation results demonstrate significant improvements in compilability and ranking of code snippets.