# Uncovering ChatGPT's Capabilities in Recommender Systems

**Authors:** Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, Jun Xu  
**arXiv:** [2305.02182](https://arxiv.org/abs/2305.02182)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02182v1/

## Key points

- ChatGPT's debut has received significant attention from the NLP community and beyond.
- The study focuses on analyzing ChatGPT's recommendation ability from an IR perspective.
- Three recommendation policies (point-wise, pair-wise, and list-wise ranking) were reformulated into a domain-specific prompt format.
- Extensive experiments were conducted on four datasets to evaluate ChatGPT's performance compared to other large language models in all three ranking policies.
- ChatGPT outperformed other models in all scenarios.
- List-wise ranking achieved the best trade-off between cost and performance, according to unit cost improvements analysis.
- ChatGPT shows potential for mitigating the cold start problem and enabling interpretable recommendation systems.
- The authors have made their code and detailed results openly available on GitHub at https://github.com/rainym00d/LLM4RS.

## Summary

The debut of ChatGPT has garnered significant attention from the natural language processing (NLP) community and beyond. While previous studies have demonstrated its effectiveness in various downstream NLP tasks, its capabilities and limitations in terms of recommendations have remained unclear. To address this gap, the authors of this study conducted an empirical analysis of ChatGPT's recommendation ability from an Information Retrieval (IR) perspective, specifically focusing on point-wise, pair-wise, and list-wise ranking. To achieve their goal, the researchers reformulated the three recommendation policies into a domain-specific prompt format. They then performed extensive experiments on four datasets from different domains to evaluate ChatGPT's performance compared to other large language models across all three ranking policies. The results showed that ChatGPT outperformed other models in all three scenarios. Furthermore, through an analysis of unit cost improvements, the authors identified that ChatGPT with list-wise ranking achieved the best trade-off between cost and performance when compared to point-wise and pair-wise ranking. This finding highlights the potential of using ChatGPT for mitigating the cold start problem and enabling interpretable recommendation systems. To facilitate further exploration in this area, the authors have made their full code and detailed original results openly available on GitHub at https://github.com/rainym00d/LLM4RS. In summary, this study provides valuable insights into ChatGPT's recommendation abilities from an IR perspective. The findings demonstrate its superiority over other large language models across different ranking policies and highlight its potential for addressing challenges such as cold start problems in recommender systems.