# Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction

**Authors:** Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, Derek Zhiyuan Cheng  
**arXiv:** [2305.06474](https://arxiv.org/abs/2305.06474)  
**Date:** 2023-05-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.06474v1/

## Key points

- Large Language Models (LLMs) are effective in text generation, translation, and summarization
- LLMs require less data than Collaborative Filtering (CF) for user preference comprehension
- A study was conducted on CF and LLMs for user rating prediction
- Zero-shot LLMs perform worse than traditional recommender models with user interaction data
- Fine-tuning LLMs with limited training data can achieve comparable or better performance than traditional models
- LLMs have access to real-world information that can be used for answering questions and creative writing
- Previous studies have explored using BERT and GPT-2 for recommendation problems but not achieving good results compared to well-tuned baselines like GRU4Rec.
- The study highlights the potential benefits of fine-tuning LLMs with limited training data for efficient recommendation systems.

## Summary

Large Language Models (LLMs) have proven to be highly effective in handling a wide range of tasks such as text generation, translation, and summarization. However, their ability to comprehend user preferences based on their previous behavior remains an emerging research question. Collaborative Filtering (CF) has traditionally been the most effective method for these tasks, relying heavily on extensive rating data. In contrast, LLMs require considerably less data while maintaining exhaustive world knowledge about each item such as movies or products. In this paper submitted to ACM, the authors conduct a thorough examination of both CF and LLMs within the classic task of user rating prediction. The task involves predicting a user's rating for a candidate item based on their past ratings. The authors investigate various LLMs in different sizes ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. The study reveals that zero-shot LLMs lag behind traditional recommender models that have access to user interaction data, indicating the importance of such data. However, through fine-tuning with only a small fraction of training data, LLMs achieve comparable or even better performance than traditional models demonstrating their potential through data efficiency. Furthermore, the authors highlight that LLMs are trained on enormous datasets of text providing access to real-world information which can be converted into knowledge used for answering questions and creative writing like poems and articles. Previous studies have explored formulating recommendation problems as natural language tasks using BERT and GPT-2 but not achieving results as good as well-tuned baselines like GRU4Rec. Overall, this study contributes valuable insights into evaluating LLMs' understanding of user preferences in comparison with traditional recommender models utilizing human interaction data. It highlights the potential benefits of fine-tuning LLMs with limited training data for efficient recommendation systems.