# AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities

**Authors:** Fabrizio Davide, Pietro Torre, Andrea Gaggioli  
**arXiv:** [2412.09385](https://arxiv.org/abs/2412.09385)  
**Date:** 2024-12-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.09385v1/

## Key points

- Researchers Fabrizio Davide, Pietro Torre, and Andrea Gaggioli introduce a novel assessment methodology to evaluate Large Language Models (LLMs).
- LLMs are sophisticated AI systems with impressive capabilities in natural language understanding and problem-solving tasks.
- Traditional evaluation methods based on task-specific benchmarks may not fully capture LLMs' complex reasoning abilities.
- The researchers combine estimating the likelihood of Artificial General Intelligence (AGI) emerging by 2030 with implementing an automated peer review process (LLM-PR).
- Significant variation in estimates provided by different LLMs is observed, with Pplx-70b-online as the top performer and Gemini-1.5-pro-api ranking lowest.
- The reliability of the LLM-PR process is demonstrated through a high Intraclass Correlation Coefficient (ICC = 0.79).
- Cross-comparisons with external benchmarks show consistent rankings but suggest existing benchmarks may not fully capture skills relevant for AGI prediction.
- Weighting schemes based on external benchmarks are explored to optimize alignment between LLM predictions and human expert forecasts, leading to the development of a new 'AGI benchmark'.
- The study offers insights into LLMs' capabilities in speculative forecasting tasks and highlights the need for innovative evaluation frameworks to assess AI performance effectively.

## Summary

In their paper "AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities," researchers Fabrizio Davide, Pietro Torre, and Andrea Gaggioli address the challenges in evaluating Large Language Models (LLMs) by introducing a novel assessment methodology. LLMs are sophisticated artificial intelligence systems that have shown impressive capabilities in natural language understanding and problem-solving tasks. However, traditional evaluation methods based on task-specific benchmarks may not fully capture their complex reasoning abilities. To overcome this limitation, the researchers combine two key tasks - estimating the likelihood of Artificial General Intelligence (AGI) emerging by 2030 and implementing an automated peer review process (LLM-PR). The outcomes of this study reveal significant variation in estimates provided by different LLMs, with Pplx-70b-online emerging as the top performer and Gemini-1.5-pro-api ranking lowest. The reliability of the LLM-PR process is demonstrated through a high Intraclass Correlation Coefficient (ICC = 0.79). Cross-comparisons with external benchmarks highlight consistent rankings but also suggest that existing benchmarks may not fully capture skills relevant for AGI prediction. The researchers further explore weighting schemes based on external benchmarks to optimize alignment between LLM predictions and human expert forecasts, leading to the development of a new 'AGI benchmark'. This study offers valuable insights into LLMs' capabilities in speculative forecasting tasks and underscores the need for innovative evaluation frameworks to assess AI performance effectively. , this paper provides a comprehensive analysis of current challenges in evaluating LLMs before detailing the AGI forecasting task submitted to a panel of models and analyzing outcomes. The methodology for the LLM peer review process is explained alongside findings before comparing results with expert survey data and introducing a new benchmark related to AGI forecasting.