# Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks

**Authors:** Agam Shah, Sudheer Chava  
**arXiv:** [2305.16633](https://arxiv.org/abs/2305.16633)  
**Date:** 2023-05-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.16633v1/

## Key points

- This paper explores the effectiveness of zero-shot large language models (LLMs) in the financial domain, specifically focusing on ChatGPT.
- The authors compare the performance of ChatGPT with open-source generative LLMs and RoBERTa fine-tuned on annotated data.
- Three research questions are addressed: data annotation, performance gaps, and feasibility of using generative models in finance.
- LLMs like ChatGPT have shown impressive performance without labeled data, but fine-tuned models generally outperform ChatGPT.
- Annotating with generative models is time-intensive.
- Four financial NLP tasks are used to benchmark different models.
- Zero-shot ChatGPT performs impressively across all tasks without labeled data, but doesn't outperform fine-tuned PLMs.
- Performance gap between fine-tuned PLMs and ChatGPT is larger when datasets are not publicly available yet.
- Fully open source LLMs perform significantly lower than ChatGPT for financial tasks.
- Using generative LLMs for labeling data can be 1000 times more time consuming compared to fine tuned PLMs.
- The paper discusses various datasets used in the study related to hawkish dovish sequence classification, financial sentiment analysis, financial numerical claim detection, and named entity recognition.
- Overall, this paper provides insights into how well ChatGPT performs with zero shot on various NLP tasks in the financial domain and compares it with other generative LLMs and fine-tuned PLMs. It also highlights performance gaps, feasibility of using generative models, and time required for data annotation in finance research projects.

## Summary

This paper explores the effectiveness of zero-shot large language models (LLMs) in the financial domain, specifically focusing on ChatGPT. The authors compare the performance of ChatGPT with open-source generative LLMs and RoBERTa fine-tuned on annotated data. They address three research questions related to data annotation, performance gaps, and the feasibility of using generative models in finance. The authors mention that LLMs like ChatGPT have shown impressive performance on various natural language processing tasks without any labeled data. However, fine-tuned models generally outperform ChatGPT. The research also highlights the time-intensive nature of annotating with generative models. To answer their research questions, the authors use four financial NLP tasks and benchmark different models. They employ RoBERTa-base and RoBERTa-large for fine-tuning benchmarks, while using ChatGPT-3.5-Turbo, Dolly-V2-12B, and H2O-12B as zero-shot models. Key insights from this study include: 1) While zero-shot ChatGPT fails to outperform fine-tuned PLMs (Pre-trained Language Models), it still performs impressively across all tasks without access to labeled data; 2) The performance gap between fine-tuned PLMs and ChatGPT is larger when datasets are not publicly available yet; 3) Fully open source LLMs perform significantly lower than ChatGPT for financial tasks; 4) Using generative LLMs for labeling data can be 1000 times more time consuming compared to fine tuned PLMs. The paper also discusses the datasets used in the study including hawkish dovish sequence classification, financial sentiment analysis, financial numerical claim detection and named entity recognition datasets. In summary, this paper is one of the first studies to investigate how well ChatGPT performs with zero shot on various NLP tasks in the financial domain. It compares ChatGPT with other open source generative LLMs and fine tuned PLMs providing insights into the performance gaps feasibility of using generative models and time required for data annotation in finance research projects.