# Approaching Human-Level Forecasting with Language Models

**Authors:** Danny Halawi, Fred Zhang, Chen Yueh-Han, Jacob Steinhardt  
**arXiv:** [2402.18563](https://arxiv.org/abs/2402.18563)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18563v1/

## Key points

- Researchers focus on optimizing a retrieval-augmented language model system for accurate forecasting of future events
- Aim to determine if language models can match the performance of competitive human forecasters
- Process involves fine-tuning a reasoning model by collecting a large dataset of forecasts and selecting subsets where the model outperforms human crowds
- Data generation for fine-tuning involves running the system at each retrieval date in the schedule with multiple configurations for data augmentation
- Optimization procedure includes generating candidate outputs per input, selecting best reasoning-prediction pairs, and fine-tuning the model on strong forecasts
- Fine-tuning data structure consists of inputs containing questions, descriptions, resolution criteria, and summarized articles; target outputs comprise reasoning and predictions
- Acknowledgments to individuals who contributed helpful discussions and feedback on an early draft of the paper; support from various institutions is also acknowledged
- Retrieval system detailed in four steps: search query generation, news retrieval using APIs, relevance filtering and re-ranking, text summarization
- Goal is to gather historical articles relevant to forecasting tasks
- Researchers strive to enhance system's performance for accurate forecasting at scale through rigorous optimization procedures and data collection efforts

## Summary

The researchers in this study focus on optimizing a retrieval-augmented language model system for accurate forecasting of future events. They aim to determine if language models can match the performance of competitive human forecasters. The process involves fine-tuning a reasoning model by collecting a large dataset of forecasts and selecting subsets where the model outperforms human crowds. To generate data for fine-tuning, the system runs at each retrieval date in the schedule and on each question in the training set with multiple configurations for data augmentation. The optimization procedure includes generating candidate outputs per input by trying different scratchpad prompts, selecting the best reasoning-prediction pairs, and fine-tuning the model on strong forecasts. The fine-tuning data structure consists of inputs containing questions, descriptions, resolution criteria, and summarized articles; and target outputs comprising reasoning and predictions. This process aims to teach the model which reasoning to apply in specific contexts. Acknowledgments are extended to individuals who contributed helpful discussions and feedback on an early draft of the paper. Support from various institutions is also acknowledged. The retrieval system is detailed in four steps: search query generation, news retrieval using APIs, relevance filtering and re-ranking, and text summarization. The goal is to gather historical articles relevant to forecasting tasks. Through rigorous optimization procedures and data collection efforts , the researchers strive to enhance their system's performance for accurate forecasting at scale.