# Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors

**Authors:** Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, Xiang Wan  
**arXiv:** [2305.14450](https://arxiv.org/abs/2305.14450)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14450v1/

## Key points

- The paper evaluates ChatGPT's capabilities in information extraction tasks from four perspectives: performance, evaluation criteria, robustness, and error types.
- ChatGPT's performance is assessed on 17 datasets with 14 information extraction sub-tasks under zero-shot, few-shot, and chain-of-thought scenarios. However, there is a significant performance gap between ChatGPT and state-of-the-art results.
- To address this gap, the authors propose a soft-matching strategy for evaluation to more accurately reflect ChatGPT's performance.
- The robustness of ChatGPT is analyzed on 14 IE sub-tasks. It rarely outputs invalid responses but struggles with irrelevant context and long-tail target types. Moreover, it has difficulty understanding subject-object relationships in relation extraction tasks.
- The authors analyze the errors of ChatGPT and identify "unannotated spans" as the most dominant error type. This raises concerns about the quality of annotated data used to train models like ChatGPT but also suggests that it may be possible to annotate data using such models.
- The paper provides a comprehensive analysis of ChatGPT's capabilities in information extraction tasks from multiple perspectives.
- The authors release their data and code on Github for further research in this field.

## Summary

In their paper "Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors," authors Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan evaluate the capabilities of ChatGPT from four perspectives: performance, evaluation criteria, robustness and error types. They assess ChatGPT's performance on 17 datasets with 14 information extraction (IE) sub-tasks under zero-shot, few-shot and chain-of-thought scenarios. The authors find a significant performance gap between ChatGPT and state-of-the-art (SOTA) results. To address this gap they propose a soft-matching strategy for evaluation to more accurately reflect ChatGPT's performance. The authors analyze the robustness of ChatGPT on 14 IE sub-tasks and find that it rarely outputs invalid responses. However irrelevant context and long-tail target types greatly affect its performance. Additionally they note that ChatGPT struggles to understand subject-object relationships in relation extraction tasks. Finally the authors analyze the errors of ChatGPT and identify "unannotated spans" as the most dominant error type. This raises concerns about the quality of annotated data used to train models like ChatGPT but also suggests that it may be possible to annotate data using such models. Overall this paper provides a comprehensive analysis of ChatGPT's capabilities in information extraction tasks from multiple perspectives. The authors release their data and code on Github for further research in this field.