# Preventing the Forecaster's Evaluation Dilemma

**Authors:** Malte C. Tichy  
**arXiv:** [2312.13720](https://arxiv.org/abs/2312.13720)  
**Date:** 2023-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.13720v1/

## Key points

- The paper addresses the issue of bias in forecast evaluation and its impact on forecast quality.
- Grouping data by outcome value can introduce bias in forecast evaluation, leading to exaggerated or extreme forecasts.
- This phenomenon is referred to as the "Forecaster's Dilemma."
- Anticipatory adjustments in forecasts jeopardize their calibration and worsen their quality.
- The author proposes a solution where forecast evaluators group pairs of forecasts and outcomes based on predicted values to assess bias more accurately.
- The paper provides a mathematical treatment using the number of items sold in a supermarket as an example.
- Biases induced by grouping data by outcome should be considered when evaluating forecasts.
- Alternative evaluation procedures that account for bias across velocities can enhance forecast calibration and ensure more accurate predictions.

## Summary

In the paper titled "Preventing the Forecaster's Evaluation Dilemma" by Malte C. Tichy, the author addresses the issue of bias in forecast evaluation and its impact on forecast quality. The paper begins by posing a hypothetical scenario where a grocery item is sold 1,234 times on a given day and asks what an ideal forecast should have predicted for such a well-selling item, on average. The intuitive answer to this question would be "1,234", as it aligns with the empirical average of forecasted expectation values. However, the author highlights that grouping data by outcome value can introduce bias in forecast evaluation. When evaluations are based on such segregation by outcome, they can deceive forecasters and incentivize them to produce overly exaggerated or extreme forecasts. This phenomenon is referred to as the "Forecaster's Dilemma," which was named by Lerch et al. in 2017. The presence of anticipatory adjustments in forecasts jeopardizes their calibration and ultimately worsens their quality. To address this dilemma and check for bias across velocities, the author proposes a solution where forecast evaluators group pairs of forecasts and outcomes based on predicted values. By evaluating empirical mean outcomes per prediction bucket, it becomes possible to assess bias more accurately. The paper provides a simple mathematical treatment for the number of items sold in a supermarket to illustrate the forecaster's dilemma and its circumvention. Through this analysis, readers gain insights into how biases can arise in forecast evaluation and how they can be mitigated to improve overall forecast quality. Overall, this paper emphasizes the importance of considering biases induced by grouping data by outcome when evaluating forecasts. By adopting alternative evaluation procedures that account for bias across velocities, it is possible to enhance forecast calibration and ensure more accurate predictions.