How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis
How to prepare an accurate paper press kit? Illustrated guide
Peter Hull · arXiv: 2610.39995v1 · Summary page
Why cover this story this week?
A feared statistical problem meets a large empirical check
Many policy studies compare changes in places that adopt a policy at different times. A widely used estimator, two-way fixed effects, can assign negative weights to some comparisons, potentially distorting the answer. An AI-assisted replication study examines how much this matters across 1,346 specifications from 108 economics papers. Its estimates are strongly correlated with those from newer difference-in-differences methods, although observed sign reversals occur in 10.8-14.5% of comparisons.
Those disagreements partly reflect estimation noise. Hierarchical models accounting for that noise imply a 91-98% correlation between the underlying quantities the methods target, and a 0.6-2.1% probability of a statistically significant conventional estimate pointing in the opposite direction from the modern target. The preprint does not make the mathematical concern disappear. Instead, it suggests that its practical importance may be smaller than feared—and demonstrates how AI-assisted replication can help test methodological warnings against published research.
Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.
Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.
- Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
- Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
- Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
- For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.