# Investigating Automatic Scoring and Feedback using Large Language Models

**Authors:** Gloria Ashiya Katuka, Alexander Gain, Yen-Yun Yu  
**arXiv:** [2405.00602](https://arxiv.org/abs/2405.00602)  
**Date:** 2024-05-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.00602v1/

## Key points

- Automatic grading and feedback systems have traditionally utilized machine learning and deep learning techniques, with recent focus on large language models (LLMs) like LLaMA-2.
- LLMs require significant computational resources for fine-tuning, leading to the development of Parameter Efficient Fine-tuning (PEFT) methods such as LoRA and QLoRA to reduce memory and computational requirements.
- PEFT-based quantized models have been effective in fine-tuning LLMs for assigning grades to short answers and essays, as well as generating feedback with high accuracy.
- Experiments showed that 4-bit quantized LLaMA-2 13B models outperformed base models in grade score predictions and feedback generation, achieving high similarity with expert feedback based on BLEU and ROUGE scores.
- The study highlights the potential impact of using quantization approaches to fine-tune LLMs for automatic grading and feedback generation tasks at reduced costs and latency, benefiting educators in online education environments.

## Summary

In the realm of automatic grading and feedback systems, there has been a longstanding exploration of traditional machine learning and deep learning techniques utilizing language models. The recent availability of high-performing large language models (LLMs) like LLaMA-2 has opened up avenues for investigating their application in automatic grading and feedback generation. However, these LLMs demand significant computational resources for fine-tuning and specific adjustments to optimize their performance for such tasks. To address these challenges, Parameter Efficient Fine-tuning (PEFT) methods such as LoRA and QLoRA have been implemented to reduce memory and computational requirements during model fine-tuning. This study delves into the effectiveness of PEFT-based quantized models with classification or regression heads in fine-tuning LLMs for automatically assigning continuous numerical grades to short answers and essays, as well as generating corresponding feedback. Experiments were conducted on both proprietary and open-source datasets, revealing that grade score predictions through finetuned LLMs are highly accurate with an average error rate of less than 3% in grade percentage. Moreover, when providing graded feedback, fine-tuned 4-bit quantized LLaMA-2 13B models outperformed competitive base models by achieving high similarity with subject matter expert feedback based on BLEU and ROUGE scores. Qualitatively, the feedback generated by these models also demonstrated a high level of alignment with expert feedback. The findings from this study shed light on the potential impact of using quantization approaches to fine-tune LLMs for various downstream tasks like automatic short answer scoring and feedback generation at reduced costs and latency. With advancements in AI-powered grading systems leveraging natural language processing technologies, educators can benefit from time-saving advantages while offering personalized feedback to students in settings where instructor resources may be limited, such as online education environments.