# Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches

**Authors:** Wannapon Suraworachet, Jennifer Seon, Mutlu Cukurova  
**arXiv:** [2401.01692](https://arxiv.org/abs/2401.01692)  
**Date:** 2024-01-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.01692v1/

## Key points

- Effective collaboration is important in overcoming challenges faced by groups
- External support is needed to regulate group interactions due to varying perceptions of challenges
- Three natural language processing models were investigated: expert knowledge rule-based model, supervised machine learning (ML) model, and Large Language Model (LLM)
- Objective was to detect and identify different dimensions of challenges from student discourse (cognitive, metacognitive, emotional, technical/other)
- Results show that supervised ML model and LLM approach perform well in detecting challenges and identifying their dimensions
- Rule-based approach relies heavily on engineered features by experts and is less effective
- Discussion on the performance of these approaches in automating detection and support of students' challenge moments during collaborative learning activities
- LLMs offer advantages but have limitations related to reliability, validity evaluation, privacy, and confabulation
- Additional considerations like model transparency are discussed for meaningful analytical feedback for students and educators using LLMs
- Study emphasizes the need for careful evaluation and consideration when implementing natural language processing models in educational contexts.

## Summary

The study titled "Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches" explores the importance of effective collaboration in overcoming challenges faced by groups. It highlights the need for external support in regulating group interactions due to varying perceptions of challenges. To address this issue, the researchers investigate three different natural language processing models: an expert knowledge rule-based model, a supervised machine learning (ML) model, and a Large Language Model (LLM). The objective is to detect and identify different dimensions of challenges (cognitive, metacognitive, emotional, and technical/other) from student discourse. The results show that both the supervised ML model and the LLM approach perform significantly well in detecting challenges and identifying their dimensions. In contrast, the rule-based approach heavily relies on engineered features by experts and proves less effective. The paper extensively discusses the performance of these three approaches in automating the detection and support of students' challenge moments during collaborative learning activities. While LLMs offer numerous advantages such as their ability to process large amounts of data, they are not considered a complete solution for issues related to detection and feedback provision in socially shared regulation of learning. Concerns regarding reliability, validity evaluation, privacy, and confabulation limit their effectiveness. The paper concludes with a discussion on additional considerations like model transparency to explore meaningful analytical feedback for students and educators using LLMs. Overall,this study sheds light on the potential benefits and limitations of leveraging natural language processing models for predicting challenge moments in collaborative learning settings. It emphasizes the need for careful evaluation and consideration of various factors when implementing such models in educational contexts.