AI summary · arXiv: 2304.08979v1 · cs.CR
In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
The essentials
- ChatGPT retrieves knowledge from the model itself and generates answers for users
- Concerns have been raised regarding its reliability
- The authors perform the first large-scale measurement of ChatGPT's reliability in generic question-answering scenarios with a carefully curated set of 5,695 questions across ten datasets and eight domains
- ChatGPT's reliability varies across different domains, especially underperforming in law and science questions
- System roles can impact ChatGPT's reliability and it is vulnerable to adversarial examples
- Thematic analysis was used to categorize samples based on their similarity in terms of semantics and domains, facilitating meaningful comparisons
- The final codebook includes eight codes/domains: history, law, general works, medicine, social science, science, technology and recreation
- The study provides valuable insights into ChatGPT's reliability
- There is a need for strengthening the reliability and security of large language models (LLMs)
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Why Does ChatGPT Fall Short in Answering Questions Faithfully?Shen Zheng, Jie Huang, Kevin Chen-Chuan Chang · 2023 · Computation and Language · cited 85 times · AI summary available
Abstract
Recent advancements in Large Language Models, such as ChatGPT, have demonstrated significant potential to impact various aspects of human life. However, ChatGPT still faces challenges in aspects like faithfulness. Taking question answering as a representative application, we seek to understand why ChatGPT falls short in answering questions faithfully. To address this question, we attempt to analyze the failures of ChatGPT in complex open-domain question answering and identifies the abilities under the failures. Specifically, we categorize ChatGPT's failures into four types: comprehension, factualness, specificity, and inference. We further pinpoint three critical abilities associated with QA…
-
ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking aboutAman Rangapur, Haoran Wang · 2023 · Computation and Language · preprint, not peer reviewed · cited 4 times · AI summary available
Abstract
Large language models have gained considerable interest for their impressive performance on various tasks. Among these models, ChatGPT developed by OpenAI has become extremely popular among early adopters who even regard it as a disruptive technology in many fields like customer service, education, healthcare, and finance. It is essential to comprehend the opinions of these initial users as it can provide valuable insights into the potential strengths, weaknesses, and success or failure of the technology in different areas. This research examines the responses generated by ChatGPT from different Conversational QA corpora. The study employed BERT similarity scores to compare these responses w…
-
Is ChatGPT Trustworthy Enough? A ReviewGuoliang Zhou, Yijia Liu, Zheng Yan et al. · 2025 · IEEE Consumer Electronics Magazine · cited 1 time · Not summarized yet
Abstract
ChatGPT, as an advanced model that seamlessly integrates into diverse digital interactions, shows great potential to enhance the performance of consumer technology and reshape its landscape. The critical question of its trustworthiness and associated challenges becomes increasingly prominent. However, the literature still lacks a thorough review to study its trust. This comprehensive review explores whether ChatGPT is trustworthy enough by navigating the multifaceted realm of large language models, placing a significant focus on its exemplar model, ChatGPT. Our exploration traverses the complex interplay of factors impacting user trust, including trust in ChatGPT itself and trust in its util…
-
ChatGPT as a tool for User Story Quality Evaluation: Trustworthy Out of the Box?Krishna Ronanki, Beatriz Cabrero-Daniel, Christian Berger · 2023 · Software Engineering · cited 43 times · AI summary available
Abstract
In Agile software development, user stories play a vital role in capturing and conveying end-user needs, prioritizing features, and facilitating communication and collaboration within development teams. However, automated methods for evaluating user stories require training in NLP tools and can be time-consuming to develop and integrate. This study explores using ChatGPT for user story quality evaluation and compares its performance with an existing benchmark. Our study shows that ChatGPT's evaluation aligns well with human evaluation, and we propose a ``best of three'' strategy to improve its output stability. We also discuss the concept of trustworthiness in AI and its implications for non…
-
Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and FaithfulnessBo Li, Gexiang Fang, Yang Yang et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 224 times · AI summary available
Abstract
The capability of Large Language Models (LLMs) like ChatGPT to comprehend user intent and provide reasonable responses has made them extremely popular lately. In this paper, we focus on assessing the overall ability of ChatGPT using 7 fine-grained information extraction (IE) tasks. Specially, we present the systematically analysis by measuring ChatGPT's performance, explainability, calibration, and faithfulness, and resulting in 15 keys from either the ChatGPT or domain experts. Our findings reveal that ChatGPT's performance in Standard-IE setting is poor, but it surprisingly exhibits excellent performance in the OpenIE setting, as evidenced by human evaluation. In addition, our research ind…
-
Do You Trust ChatGPT? - Perceived Credibility of Human and AI-Generated ContentMartin Huschens, Martin Briesch, D. Sobania et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 44 times · Not summarized yet
Abstract
This paper examines how individuals perceive the credibility of content originating from human authors versus content generated by large language models, like the GPT language model family that powers ChatGPT, in different user interface versions. Surprisingly, our results demonstrate that regardless of the user interface presentation, participants tend to attribute similar levels of credibility. While participants also do not report any different perceptions of competence and trustworthiness between human and AI-generated content, they rate AI-generated content as being clearer and more engaging. The findings from this study serve as a call for a more discerning approach to evaluating infor…
-
Consistency Analysis of ChatGPTMyeongjun Jang, Thomas Lukasiewicz · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 88 times · Not summarized yet
Abstract
ChatGPT has gained a huge popularity since its introduction. Its positive aspects have been reported through many media platforms, and some analyses even showed that ChatGPT achieved a decent grade in professional exams, adding extra support to the claim that AI can now assist and even replace humans in industrial fields. Others, however, doubt its reliability and trustworthiness. This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour, focusing specifically on semantic consistency and the properties of negation, symmetric, and transitive consistency. Our findings suggest that while both models appear to show an enhanced language understandin…
-
Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and ErrorsRidong Han, Tao Peng, Chaohao Yang et al. · 2023 · Computation and Language · cited 84 times · AI summary available
Abstract
ChatGPT has stimulated the research boom in the field of large language models. In this paper, we assess the capabilities of ChatGPT from four perspectives including Performance, Evaluation Criteria, Robustness and Error Types. Specifically, we first evaluate ChatGPT's performance on 17 datasets with 14 IE sub-tasks under the zero-shot, few-shot and chain-of-thought scenarios, and find a huge performance gap between ChatGPT and SOTA results. Next, we rethink this gap and propose a soft-matching strategy for evaluation to more accurately reflect ChatGPT's performance. Then, we analyze the robustness of ChatGPT on 14 IE sub-tasks, and find that: 1) ChatGPT rarely outputs invalid responses; 2) …
Prior work it builds on
-
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTQihuang Zhong, Liang Ding, Juhua Liu et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 331 times · AI summary available
Abstract
Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achi…
-
ChatGPT: The End of Online Exam Integrity?Teo Sušnjak, Timothy R. McIntosh · 2022 · Education sciences · cited 626 times · Not summarized yet
Abstract
This study addresses the significant challenge posed by the use of Large Language Models (LLMs) such as ChatGPT on the integrity of online examinations, focusing on how these models can undermine academic honesty by demonstrating their latent and advanced reasoning capabilities. An iterative self-reflective strategy was developed for invoking critical thinking and higher-order reasoning in LLMs when responding to complex multimodal exam questions involving both visual and textual data. The proposed strategy was demonstrated and evaluated on real exam questions by subject experts and the performance of ChatGPT (GPT-4) with vision was estimated on an additional dataset of 600 text descriptions…
-
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and InteractivityYejin Bang, Samuel Cahyawijaya, Nayeon Lee et al. · 2023 · Computation and Language · cited 1,858 times · AI summary available
Abstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an interme…
-
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution PerspectiveJin-Dong Wang, Xixu Hu, Wenxin Hou et al. · 2023 · IEEE Data Engineering Bulletin · cited 325 times · Not summarized yet
Abstract
ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of ChatGPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select severa…
-
Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnessGuido Zuccon, Bevan Koopman · 2023 · Computation and Language · preprint, not peer reviewed · cited 54 times · AI summary available
Abstract
Generative pre-trained language models (GPLMs) like ChatGPT encode in the model's parameters knowledge the models observe during the pre-training phase. This knowledge is then used at inference to address the task specified by the user in their prompt. For example, for the question-answering task, the GPLMs leverage the knowledge and linguistic patterns learned at training to produce an answer to a user question. Aside from the knowledge encoded in the model itself, answers produced by GPLMs can also leverage knowledge provided in the prompts. For example, a GPLM can be integrated into a retrieve-then-generate paradigm where a search engine is used to retrieve documents relevant to the quest…
Papers that cite it
-
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark DatasetsMd Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 261 times · Not summarized yet
Abstract
The development of large language models (LLMs) such as ChatGPT has brought a lot of attention recently. However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. In this paper, we aim to present a thorough evaluation of ChatGPT's performance on diverse academic datasets, covering tasks like question-answering, text summarization, code generation, commonsense reasoning, mathematical problem-solving, machine translation, bias detection, and ethical considerations. Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in thes…
-
Accuracy and Repeatability of ChatGPT Based on a Set of Multiple-Choice Questions on Objective Tests of HearingK. Kochanek, H. Skarżyński, W. Jędrzejczak · 2024 · Cureus · cited 37 times · Not summarized yet
Abstract
Introduction: ChatGPT has been tested in many disciplines, but only a few have involved hearing diagnosis and none to physiology or audiology more generally. The consistency of the chatbot’s responses to the same question posed multiple times has not been well investigated either. This study aimed to assess the accuracy and repeatability of ChatGPT 3.5 and 4 on test questions concerning objective measures of hearing. Of particular interest was the short-term repeatability of responses which was here tested on four separate days extended over one week. Methods: We used 30 single-answer, multiple-choice exam questions from a one-year course on objective methods of testing hearing. The question…
-
Evaluating ChatGPT's effectiveness and tendencies in Japanese internal medicine.Yudai Kaneda, Akari Tayuinosho, Rika Tomoyose et al. · 2024 · Journal of Evaluation In Clinical Practice · cited 3 times · Not summarized yet
Abstract
INTRODUCTION ChatGPT, a large-scale language model, is a notable example of AI's potential in health care. However, its effectiveness in clinical settings, especially when compared to human physicians, is not fully understood. This study evaluates ChatGPT's capabilities and limitations in answering questions for Japanese internal medicine specialists, aiming to clarify its accuracy and tendencies in both correct and incorrect responses. METHODS We utilized ChatGPT's answers on four sets of self-training questions for internal medicine specialists in Japan from 2020 to 2023. We ran three trials for each set to evaluate its overall accuracy and performance on nonimage questions. Subsequently, …
-
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via DebateBoshi Wang, Xiang Yue, Huan Sun · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 161 times · Not summarized yet
Abstract
Large language models (LLMs) such as ChatGPT and GPT-4 have shown impressive performance in complex reasoning tasks. However, it is difficult to know whether the models are reasoning based on deep understandings of truth and logic, or leveraging their memorized patterns in a relatively superficial way. In this work, we explore testing LLMs' reasoning by engaging with them in a debate-like conversation, where given a question, the LLM and the user need to discuss to make the correct decision starting from opposing arguments. Upon mitigating the Clever Hans effect, our task requires the LLM to not only achieve the correct answer on its own, but also be able to hold and defend its belief instea…
-
AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial TopicsVahid Ghafouri, Vibhor Agarwal, Yong Zhang et al. · 2023 · International Conference on Information and Knowledge Management · cited 37 times · Not summarized yet
Abstract
The introduction of ChatGPT and the subsequent improvement of Large Language Models (LLMs) have prompted more and more individuals to turn to the use of ChatBots, both for information and assistance with decision-making. However, the information the user is after is often not formulated by these ChatBots objectively enough to be provided with a definite, globally accepted answer. Controversial topics, such as "religion", "gender identity", "freedom of speech", and "equality", among others, can be a source of conflict as partisan or biased answers can reinforce preconceived notions or promote disinformation. By exposing ChatGPT to such debatable questions, we aim to understand its level of aw…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 6 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.