# ChaTA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

**Authors:** Yann Hicke, Anmol Agarwal, Qianou Ma, Paul Denny  
**arXiv:** [2311.02775](https://arxiv.org/abs/2311.02775)  
**Date:** 2023-11-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.02775v1/

## Key points

- Study addresses challenges of scalable and intelligent question-answering (QA)
- Leveraging open-source Large Language Models (LLMs)
- Pipeline combines retrieval augmented generation (RAG), supervised fine-tuning (SFT), and an alternative to reinforcement learning with human feedback (RLHF)
- Enhancing LLMs from the LLaMA-2 family
- Experiments conducted on a Piazza dataset from an introductory CS course
- Dataset consists of 10k QA pairs and 1.5k pairs of preferences data
- Data privacy ensured
- Utilizing adaptability of LLMs to offer versatile query responses
- Comprehensive evaluation shows pipeline improves answer quality by 33%
- RAG particularly impactful
- Work lays foundation for ChaTA, an intelligent QA assistant customizable for courses with online QA platform
- Effective fine-tuning of LMs on instruction data and human preferences data to improve task completion and response quality highlighted in related work
- Challenges and future directions in utilizing machine learning for QA workflows discussed

## Summary

In this study, we address the challenges of scalable and intelligent question-answering (QA) by leveraging open-source Large Language Models (LLMs). Our pipeline combines retrieval augmented generation (RAG), supervised fine-tuning (SFT), and an alternative to reinforcement learning with human feedback (RLHF) to enhance LLMs from the LLaMA-2 family. We conduct experiments on a Piazza dataset from an introductory CS course, consisting of 10k QA pairs and 1.5k pairs of preferences data, while ensuring data privacy. To overcome limitations, we utilize the adaptability of LLMs to offer versatile query responses. Our comprehensive evaluation using both LLM-based and rubric-based human evaluations shows that our pipeline improves answer quality by 33%, with RAG being particularly impactful. This work lays the foundation for ChaTA, an intelligent QA assistant customizable for courses with an online QA platform. In related work, we highlight the effectiveness of fine-tuning LMs on instruction data and human preferences data to improve task completion and response quality. We also discuss challenges and future directions in utilizing machine learning for QA workflows.