# Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

**Authors:** Chenglei Si, Diyi Yang, Tatsunori Hashimoto  
**arXiv:** [2409.04109](https://arxiv.org/abs/2409.04109)  
**Date:** 2024-09-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.04109v1/

## Key points

- Recent advancements in large language models (LLMs) have potential to accelerate scientific discovery by autonomously generating and validating new ideas
- Lack of evidence showing LLM systems can produce novel, expert-level ideas or complete entire research process
- Experimental design established to evaluate research idea generation, comparing expert NLP researchers and LLM ideation agent
- Over 100 NLP researchers recruited to write novel ideas, subject to blind reviews alongside LLM-generated ideas
- Results showed LLM-generated ideas deemed more novel than human expert ideas, slightly weaker in feasibility
- Challenges highlighted in building and evaluating research agents include shortcomings in LLM self-evaluation and lack of diversity in idea generation
- Template inspired by grant submission guidelines introduced to structure idea proposals from both human participants and LLM agent
- Style normalization module developed to standardize writing styles without altering content
- Review form based on AI conference reviewing practices designed to assess novelty, excitement, feasibility, and expected effectiveness of research ideas
- Blind review evaluation compared ideas from three conditions: Human Ideas; AI Ideas generated by LLM agent; AI Ideas + Human Rerank selected manually from LLM agent's output
- Differences in novelty scores among the three conditions observed, with AI-generated ideas ranking higher than human ones
- Further studies planned to explore how novelty and feasibility judgments impact research outcomes by recruiting researchers to execute proposed ideas into full projects

## Summary

Recent advancements in large language models (LLMs) have generated excitement about their potential to accelerate scientific discovery by autonomously generating and validating new ideas. However, there has been a lack of evidence showing that LLM systems can produce novel, expert-level ideas or complete the entire research process. To address this gap, an experimental design was established to evaluate research idea generation while controlling for confounders and conducting a head-to-head comparison between expert NLP researchers and an LLM ideation agent. Over 100 NLP researchers were recruited to write novel ideas, which were then subject to blind reviews alongside ideas generated by the LLM agent. The results revealed that LLM-generated ideas were deemed more novel than human expert ideas, although slightly weaker in feasibility. This study highlighted challenges in building and evaluating research agents, such as shortcomings in LLM self-evaluation and lack of diversity in idea generation. To ensure a fair evaluation process, a template inspired by grant submission guidelines was introduced to structure idea proposals from both human participants and the LLM agent. Additionally, a style normalization module was developed to standardize writing styles without altering content. A review form based on AI conference reviewing practices was designed to assess novelty, excitement, feasibility, and expected effectiveness of research ideas. The blind review evaluation compared ideas from three conditions: Human Ideas written by expert researchers; AI Ideas generated by the LLM agent; and AI Ideas + Human Rerank selected manually from the LLM agent's output. The results showed differences in novelty scores among the three conditions, with AI-generated ideas ranking higher than human ones. Moving forward, further studies will explore how these novelty and feasibility judgments impact research outcomes by recruiting researchers to execute the proposed ideas into full projects. This comprehensive approach aims to shed light on the capabilities of current LLM systems for research ideation and address challenges in evaluating AI-generated ideas effectively.