# Evaluating Quality of Chatbots and Intelligent Conversational Agents

**Authors:** Nicole M. Radziwill, Morgan C. Benton  
**arXiv:** [1704.04579](https://arxiv.org/abs/1704.04579)  
**Date:** 2017-04-15  
**Summary page:** https://www.summarizepaper.com/en/paper/1704.04579v1/

## Key points

- Chatbots are intelligent software agents that respond to natural language input
- They have become easier to train and implement due to open source code availability, development platforms, and SaaS options
- Chatbots can enhance customer experiences and support learning, but they can also be used for social harm
- The paper explores quality issues related to chatbot development and implementation
- It examines attributes such as usability, accuracy, reliability, security/privacy concerns
- Different approaches for assessing chatbot quality are reviewed, including surveys or interviews with users or experts in the field
- The paper proposes a quality assessment method using the Analytic Hierarchy Process (AHP)
- The proposed method aims to provide a comprehensive evaluation framework for assessing chatbot quality.

## Summary

The paper titled "Evaluating Quality of Chatbots and Intelligent Conversational Agents" by Nicole M. Radziwill and Morgan C. Benton explores the quality issues and attributes associated with chatbot development and implementation. Chatbots are intelligent software agents that respond to natural language input, such as text or voice, providing conversational output and executing tasks when commanded. While chatbot technologies have been around since the 1960s and have influenced user interface development in games since the early 1980s, they have become easier to train and implement due to open source code availability, development platforms, and Software as a Service (SaaS) options. In addition to enhancing customer experiences and supporting learning, chatbots can also be used for social harm such as spreading rumors or misinformation or attacking individuals for expressing their thoughts online. The paper presents a literature review of quality issues related to chatbot development and implementation. It examines various attributes that contribute to chatbot quality including usability, accuracy, reliability, security/privacy concerns etc. Furthermore, the paper reviews different approaches for assessing chatbot quality such as surveys or interviews with users or experts in the field. It proposes a quality assessment method based on these attributes using the Analytic Hierarchy Process (AHP). The proposed method aims to provide a comprehensive evaluation framework for assessing the quality of chatbots. Overall, this paper provides valuable insights into the contemporary issue of chatbot development and implementation by discussing quality issues, attributes, and proposing a novel approach for evaluating chatbot quality using AHP.