Results of the summarizing process for the arXiv paper: 2401.03676v1

In their paper "Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education," authors Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo and Mei Kuan Lim address the growing concern among educators regarding the use of Large Language Models (LLMs) like ChatGPT in programming education. The authors specifically focus on the potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct. To investigate this issue, they conduct an empirical study where they examine how well AIGC Detectors can detect code generated by LLMs. To gather data for their study, the authors generate code using different variants in response to coding problems and collect a dataset consisting of 5,069 samples. Each sample includes a textual description of a coding problem and its corresponding human-written Python solution code from various sources such as Quescol, Kaggle and LeetCode. Using this dataset, they create 13 sets of code problem variant prompts to instruct ChatGPT to generate outputs. They then assess the performance of five AIGC detectors in distinguishing between human-written code and AI-generated code. The results reveal that existing AIGC Detectors perform poorly in accurately identifying AI-generated code compared to human-written code. This finding highlights the need for further improvements in AIGC detection technology to ensure its effectiveness in detecting academic misconduct involving LLMs like ChatGPT. Overall,this research sheds light on the challenges posed by LLMs in programming education and emphasizes the importance of developing robust AIGC Detectors to maintain academic integrity. AI Detectors; Identifying AI-Generated Code; Education; Academic Misconduct; Large Language Models.

• - Authors address the concern of educators regarding the use of Large Language Models (LLMs) like ChatGPT in programming education
• - Focus on potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct
• - Conducted an empirical study to examine how well AIGC Detectors can detect code generated by LLMs
• - Generated code using different variants and collected a dataset consisting of 5,069 samples
• - Created 13 sets of code problem variant prompts to instruct ChatGPT to generate outputs
• - Assessed the performance of five AIGC detectors in distinguishing between human-written code and AI-generated code
• - Existing AIGC Detectors perform poorly in accurately identifying AI-generated code compared to human-written code
• - Highlights the need for further improvements in AIGC detection technology
• - Emphasizes the challenges posed by LLMs in programming education
• - Importance of developing robust AIGC Detectors to maintain academic integrity.

The authors of a study talked about how teachers are worried about using big computer programs to help teach coding. They also looked at how these programs can be used in bad ways to cheat in school. They did an experiment to see if the programs could tell if code was written by a person or by the program itself. They made different versions of the program and collected a lot of examples. They tested five different detectors to see if they could tell the difference between human-written code and code made by the program, and found that they didn't do very well. This means we need better detectors to make sure people don't cheat in school." Definitions- Educators: Teachers or people who teach others - Large Language Models (LLMs): Big computer programs that can understand and generate human-like text - Artificial Intelligence Generated Content (AIGC) Detectors: Programs that can check if something was made by a person or an AI program - Academic misconduct: Cheating or doing something wrong in school - Empirical study: A type of research where experiments are done to collect data - Dataset: A collection of information or examples used for studying something - Variants: Different versions or types - Prompts: Instructions given to get a response from someone or something - Distinguishing: Telling the difference between two things

Introduction

The use of Artificial Intelligence (AI) in education has been gaining momentum in recent years. With the rise of Large Language Models (LLMs) like ChatGPT, educators have started to explore their potential for teaching programming languages. These LLMs are trained on large datasets and can generate human-like text responses based on a given prompt. However, this technology also raises concerns about academic misconduct, as students may use these models to generate code solutions instead of writing them themselves. In response to this concern, a group of researchers from Nanyang Technological University conducted an empirical study titled "Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education." The paper aims to assess the effectiveness of existing AI Generated Content (AIGC) detectors in identifying code generated by LLMs like ChatGPT. The authors also discuss the implications of their findings for programming education and academic integrity.

The Study

To investigate the issue at hand, the authors created a dataset consisting of 5,069 samples. Each sample included a textual description of a coding problem and its corresponding human-written Python solution code from various sources such as Quescol, Kaggle, and LeetCode. This dataset was used to train five AIGC detectors - MOSS, JPlag, Sherlock Holmes Plagiarism Detection System (SH), Simian Clone Detector (Simian), and YAP3 - on distinguishing between human-written code and AI-generated code. Next, the authors generated 13 sets of code problem variant prompts to instruct ChatGPT to produce outputs. These prompts were designed to mimic real-world coding problems that students might encounter in their coursework or assessments. The resulting outputs were then added to the dataset as AI-generated samples. After training the AIGC detectors on this augmented dataset, they were tested on their ability to identify AI-generated code. The results showed that the existing AIGC detectors performed poorly in accurately identifying AI-generated code compared to human-written code. This finding highlights the need for further improvements in AIGC detection technology to ensure its effectiveness in detecting academic misconduct involving LLMs like ChatGPT.

Implications for Education

The study's findings have significant implications for programming education, particularly in terms of maintaining academic integrity. As LLMs become more prevalent and accessible, there is a growing concern that students may use them to generate code solutions instead of writing them themselves. This not only undermines the learning process but also raises ethical concerns about plagiarism and cheating. To address this issue, educators must be aware of the potential exploitation of imperfections in AIGC detectors by students looking to cheat. They should also consider incorporating assessments that require students to explain their thought processes and reasoning behind their code solutions rather than just submitting the final output. This approach can help detect cases where students have used LLMs without fully understanding the underlying concepts. Moreover, as highlighted by this study, there is a pressing need for further research and development of robust AIGC detection technology. With advancements in AI technology, it is crucial to continuously improve these detectors' accuracy and effectiveness to keep up with evolving methods of academic misconduct.

Conclusion

In conclusion, "Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education" sheds light on the challenges posed by LLMs like ChatGPT in programming education. The study's findings highlight the need for improved AIGC detection technology to maintain academic integrity and prevent cheating using these models. As educators continue to explore the potential uses of AI technology in education, it is essential to also address its potential drawbacks and take necessary measures to mitigate them. Further research on this topic will be crucial as we navigate through this rapidly evolving landscape of AI in education.