Results of the summarizing process for the arXiv paper: 2306.09896v1

Large Language Models (LLMs) have shown impressive capabilities in code generation, but they still struggle with challenging programming tasks. To address this issue, self-repair has emerged as a popular technique to boost performance. However, limited studies exist on how and when self-repair works effectively. This paper analyzes GPT-3.5 and GPT-4's ability to perform self-repair on APPS, a challenging dataset consisting of diverse coding challenges. The authors establish a new evaluation strategy called pass@t that measures the pass rate of tasks against the total number of tokens sampled from the model, enabling a fair comparison to purely sampling-based approaches. They find that only GPT-4 is capable of carrying out self-repair on challenging coding tasks effectively. The effectiveness of self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback on programs generated by GPT-3.5 and using expert human programmers to give feedback on programs generated by GPT-4 unlocks significant performance gains. The paper also discusses broader impacts related to productivity improvements for both legitimate software development and malicious intent. Additionally, research on LLMs comes at a high environmental cost due to their computational requirements. This work contributes to program synthesis with large language models by evaluating models from the perspective of minimizing the number of samples needed instead of raw accuracy or pass@k metric commonly used in prior literature. It assumes access to full input-output examples, unlike some prior work that distinguishes between public and private tests for filtering purposes. Finally, this work differs from most prior research in code repair by using textual feedback provided by the model itself rather than relying solely on statistical or learning based techniques for repairing human written code. The authors demonstrate that replacing GPT-4's self generated feedback with feedback provided by an experienced programmer increases the number of repaired programs passing all unit tests significantly. In conclusion, this paper presents valuable insights into the effectiveness of self repair in LLMs for challenging coding tasks and highlights broader impacts related to productivity improvements and environmental costs.

• Large Language Models (LLMs) are impressive in code generation but struggle with challenging programming tasks.
• Self-repair has emerged as a popular technique to boost performance, but limited studies exist on its effectiveness.
• GPT-3.5 and GPT-4 were analyzed for their ability to perform self-repair on APPS dataset, which consists of diverse coding challenges.
• A new evaluation strategy called pass@t was established to measure the pass rate of tasks against the total number of tokens sampled from the model.
• Only GPT-4 is capable of carrying out self-repair on challenging coding tasks effectively.
• The effectiveness of self-repair is bottlenecked by the feedback stage; using expert human programmers to give feedback unlocks significant performance gains.
• Research on LLMs comes at a high environmental cost due to their computational requirements.
• This work contributes to program synthesis with large language models by evaluating models from the perspective of minimizing the number of samples needed instead of raw accuracy or pass@k metric commonly used in prior literature.
• The authors demonstrate that replacing GPT-4's self-generated feedback with feedback provided by an experienced programmer increases the number of repaired programs passing all unit tests significantly.
• This paper presents valuable insights into the effectiveness of self-repair in LLMs for challenging coding tasks and highlights broader impacts related to productivity improvements and environmental costs.

Exploring the Effectiveness of Self-Repair in Large Language Models for Challenging Coding Tasks

Large language models (LLMs) have become increasingly popular for code generation due to their impressive capabilities. However, LLMs still struggle with challenging programming tasks and self-repair has emerged as a popular technique to boost performance. Despite its popularity, limited studies exist on how and when self-repair works effectively. This paper analyzes GPT-3.5 and GPT-4's ability to perform self-repair on APPS, a challenging dataset consisting of diverse coding challenges.

Evaluation Strategy

The authors establish a new evaluation strategy called pass@t that measures the pass rate of tasks against the total number of tokens sampled from the model, enabling a fair comparison to purely sampling based approaches. They find that only GPT-4 is capable of carrying out self repair on challenging coding tasks effectively. The effectiveness of self repair is bottlenecked by the feedback stage; using GPT-4 to give feedback on programs generated by GPT-3.5 and using expert human programmers to give feedback on programs generated by GPT 4 unlocks significant performance gains.

Broader Impacts

The paper also discusses broader impacts related to productivity improvements for both legitimate software development and malicious intent as well as environmental costs associated with research into LLMs due to their computational requirements.

Contributions

This work contributes to program synthesis with large language models by evaluating models from the perspective of minimizing the number of samples needed instead of raw accuracy or pass@k metric commonly used in prior literature. It assumes access to full input output examples unlike some prior work that distinguishes between public and private tests for filtering purposes. Finally, this work differs from most prior research in code repair by using textual feedback provided by the model itself rather than relying solely on statistical or learning based techniques for repairing human written code; it demonstrates that replacing GPT 4's self generated feedback with feedback provided by an experienced programmer increases the number of repaired programs passing all unit tests significantly..

Conclusion

In conclusion, this paper presents valuable insights into the effectiveness of self repair in LLMs for challenging coding tasks and highlights broader impacts related to productivity improvements and environmental costs associated with such research efforts