How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis

Peter Hull

AI summary · arXiv: 2610.39995v1 · econ.EM

How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis

Peter Hull

Original paper Submitted to arXiv on 29 Oct. 2026 35 pages, 8 figures, 4 tables License: CC BY 4.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Across 1,346 replicated specifications, TWFE and modern difference-in-differences estimates largely agree, with TWFE offering greater precision and hierarchical models suggesting infrequent statistically significant wrong-sign estimates.

Question

How often do negative weights in two-way fixed effects estimation materially change empirical conclusions in staggered-adoption designs? How does agreement with modern difference-in-differences estimators compare with their relative precision?

Method

AI-assisted screening of 16,094 records from ten economics journals produced 1,346 replicated specifications from 108 papers. After leverage screening, the analysis compared TWFE with DiD-averaging in 1,069 comparisons and imputation in 1,116 comparisons, and measured negative weights in static specifications. Precision-dependent Gaussian and nonparametric hierarchical models separated estimand variation from sampling noise, using estimator covariances obtained through 999 bootstrap replications.

Findings

Result Observed sign reversals occurred in 14.5% of TWFE–DiD-averaging comparisons and 10.8% of TWFE–imputation comparisons.
See the passage in the paper
Around 14.5% of TWFE estimates change sign with DiD averaging while 3.5% do so with a significant difference. These shares are lower with imputation, at 10.8% and 2.2%.
Result Modern estimators were less precise on average, with standard errors 47% larger for DiD-averaging and 14% larger for imputation than for TWFE.
See the passage in the paper
These suggest increased noise, assuming similar estimands, and indeed DiD-averaging and imputation standard errors are on average 47% and 14% larger than corresponding TWFE standard errors.
Result In static specifications, negative weights affected a median of approximately 8% of treated rows but accounted for only 1.4% of absolute effect weight.
See the passage in the paper
Among static TWFE specifications, the median fraction of treated observation rows with negative ex post weights is approximately 8%, while the median share of absolute effect weight carried by negative weights is only 1.4% (Appendix Figure A3 shows full distributions).
Result The baseline hierarchical model estimated probabilities of statistically significant TWFE estimates opposite in sign to the modern estimand of 2.1% for DiD-averaging and 0.6% for imputation.
See the passage in the paper
These rates are 2.1% for DiD averaging and 0.6% for imputation, reflecting the high correlation in estimands.
Authors’ interpretation The author interprets the evidence as suggesting that negative weights are unlikely to dramatically alter conclusions across the replicated specifications.
See the passage in the paper
Findings suggest that negative weights, while potentially a serious problem for TWFE, are unlikely to dramatically affect empirical conclusions across the large number of replicated specifications.

Limitations

The analysis accounts for estimation noise but does not isolate estimand differences caused by parallel-trends violations.
See the passage in the paper
Estimates may also flip signs because of estimation error or estimand differences due to, e.g., failures of the key parallel trends assumption. The analysis here accounts for the former but not the latter.
The scope excludes time-varying controls, continuous treatments, and several event-study specification concerns.
See the passage in the paper
Other potentially important issues raised in this literature—such as with time-varying controls, continuous treatments, extrapolation or underidentification in event study models, and choices of binning or other treatment specifications—are outside the scope of the present analysis.
The paper does not establish that the estimated reversal rates are low enough to dismiss concerns about TWFE.
See the passage in the paper
Importantly, this paper does not take a stance on whether the estimated sign reversal rates or estimand correlations are “low enough” to dismiss concerns over TWFE estimation.

Why it matters

The paper provides systematic empirical evidence for weighing TWFE’s precision against modern estimators’ robustness, without arguing against application-specific checks. It also demonstrates how AI-assisted replication can support large-scale empirical assessments of econometric theory.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: Two-way fixed effects; Staggered-adoption difference-in-differences; Negative weights; Treatment-effect heterogeneity; Replication

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

Peter Hull’s paper examines how much the “negative weights” problem in two-way fixed effects (TWFE) estimation affects empirical conclusions in staggered-adoption difference-in-differences studies. When units adopt treatment at different times and treatment effects vary, TWFE can assign negative weights to some effects, potentially yielding a negative estimate even when all effects are positive. The paper assesses how often this theoretical concern matters in published applications by comparing TWFE with two modern approaches designed to avoid problematic weighting: DiD-averaging, following Callaway and Sant’Anna, and imputation, following Borusyak, Jaravel, and Spiess. Using AI-assisted screening and replication with human oversight, the author searches 16,094 bibliographic records from ten general-interest economics journals, covering journal-specific periods between 2000 and October 2026. The resulting sample includes 1,346 eligible, replicable specifications from 108 papers. The analysis focuses on relatively simple binary-treatment designs, excluding specifications with time-varying controls, unit-specific trends, and other complications. After excluding comparisons with excessive cluster leverage, the sample contains 1,069 TWFE–DiD-averaging comparisons from 82 papers and 1,116 TWFE–imputation comparisons from 89 papers. Event studies can contribute separate averages of pretreatment and post-treatment coefficients, and outcomes are standardized for comparability. TWFE estimates are strongly correlated with the modern estimates: the correlations are 0.88 for DiD-averaging and 0.93 for imputation. Switching estimators reverses the estimated sign in 14.5% and 10.8% of comparisons, respectively. However, only 3.5% and 2.2% of comparisons combine a sign reversal with a statistically significant difference between estimates. Excluding pretreatment comparisons lowers the sign-reversal rates to 12.8% and 8.3%. The modern methods are also less precise: standard errors average 47% larger than TWFE’s for DiD-averaging and 14% larger for imputation. Among static TWFE specifications, the median share of treated observation rows receiving negative weights is approximately 8%, but these weights account for only 1.4% of total absolute effect weight. The paper also examines which characteristics predict sign reversals. Later publication and reporting both TWFE and modern estimates are associated with fewer reversals, particularly for DiD-averaging. A ten-percentage-point increase in the always-treated share predicts a statistically significant 6.7-percentage-point increase in DiD-averaging reversals. A corresponding increase in the never-treated share predicts a statistically significant 2.1-percentage-point decrease in imputation reversals. Journal standing, citation counts, and specification prominence show no statistically significant associations. To distinguish sampling noise from underlying differences, the author estimates precision-dependent hierarchical models of the joint distribution of estimands. The baseline Gaussian model estimates underlying correlations of 0.913 for DiD-averaging and 0.976 for imputation, with underlying sign-reversal probabilities of 11.5% and 5.6%, respectively. The estimated probabilities of obtaining a statistically significant TWFE estimate with a sign opposite to the modern estimand are 2.1% and 0.6%. A nonparametric alternative produces similar underlying reversal rates, although the probabilities of statistically significant wrong-sign estimates rise to 2.4% and 1.6%. The findings suggest that negative weighting is less disruptive in this replication sample than its theoretical possibilities might imply, while modern estimators involve a precision tradeoff. They do not establish that TWFE is generally safe or that modern estimates identify the true causal effect. Differences between estimators may also reflect violations of parallel trends or no anticipation, differences in support, and aggregation choices. The models address sampling noise, not these identification problems. Moreover, the restricted sample of replicable journal studies limits generalizability, and the hierarchical results depend on modeling assumptions. The author therefore recommends application-specific comparisons rather than dismissing concerns about TWFE.

How Much Do Negative Weights Matter in Difference-in-Differences?

A statistical method can have a serious theoretical weakness without routinely overturning real-world findings. But how often does that happen? In “How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis,” Peter Hull investigates that question for one of empirical economics’ most widely used tools: two-way fixed effects, or TWFE.

The concern is that TWFE can produce misleading results when different units adopt a treatment at different times. Hull’s large-scale replication study finds that newer estimators usually produce results similar to TWFE, although meaningful disagreements remain. The lesson is neither to abandon TWFE automatically nor to assume it is safe.

The Problem: An Average That Can Reverse the Sign

Difference-in-differences studies compare changes in outcomes across groups to estimate a treatment’s effect. In staggered-adoption settings, units begin treatment at different times—for example, when jurisdictions introduce a policy in different years.

When treatment effects vary across groups or over time, TWFE can combine those effects using negative weights. That creates a counterintuitive possibility: an estimated effect can be negative even if every underlying treatment effect is positive.

Modern alternatives are designed to avoid this problematic weighting. Hull compares TWFE with two such approaches: DiD-averaging and imputation. His central question is practical: how much does changing the estimator change the conclusions researchers might draw?

A Large Replication Effort, with Important Boundaries

Using AI-assisted screening and replication with human oversight, Hull searches 16,094 bibliographic records. The search covers ten general-interest economics outlets over outlet-specific periods between 2000 and October 2026. It yields 1,346 eligible, replicable specifications from 108 papers.

The analysis deliberately focuses on relatively simple binary-treatment designs. Specifications with time-varying controls, unit-specific trends, and other complications are excluded. Outcomes are standardized to make comparisons more meaningful across studies, and event studies can contribute separate averages of pretreatment and post-treatment coefficients.

After excluding comparisons with excessive cluster leverage, the analysis includes:

1,069 TWFE–DiD-averaging comparisons from 82 papers.
1,116 TWFE–imputation comparisons from 89 papers.

These restrictions make systematic replication more feasible, but they also limit how broadly the findings can be generalized.

Most Results Align, but Some Change Direction

The headline finding is substantial agreement. TWFE estimates correlate 0.88 with DiD-averaging estimates and 0.93 with imputation estimates. Those are strong relationships, though they do not mean the methods are interchangeable.

Switching from TWFE reverses the estimated sign in 14.5% of DiD-averaging comparisons and 10.8% of imputation comparisons. Excluding pretreatment comparisons reduces those rates to 12.8% and 8.3%, respectively.

A sign reversal alone, however, can exaggerate the practical disagreement. Two small, noisy estimates may fall on opposite sides of zero without being statistically distinguishable. Only 3.5% of DiD-averaging comparisons and 2.2% of imputation comparisons combine a sign reversal with a statistically significant difference between estimates.

The Precision Tradeoff

The modern approaches also tend to be less precise. Their standard errors average 47% larger than TWFE’s for DiD-averaging and 14% larger for imputation. Avoiding problematic weighting can therefore come with greater statistical uncertainty.

Hull also examines negative weights directly in static TWFE specifications. The median share of treated observation rows receiving negative weights is approximately 8%. Yet those weights account for only 1.4% of total absolute effect weight. Their presence, in other words, does not necessarily imply that they dominate the estimate.

When Are Reversals More Likely?

Sample composition offers some clues. A ten-percentage-point increase in the always-treated share predicts a statistically significant 6.7-percentage-point increase in DiD-averaging reversals. A corresponding increase in the never-treated share predicts a statistically significant 2.1-percentage-point decrease in imputation reversals.

Later publication and reporting both TWFE and modern estimates are associated with fewer reversals, particularly for DiD-averaging. Citation counts and specification prominence show no statistically significant associations. These patterns are predictive relationships, not evidence that changing one characteristic would necessarily cause fewer disagreements.

Looking Beneath Sampling Noise

To separate sampling noise from underlying differences, Hull estimates precision-dependent hierarchical models. The baseline Gaussian model places underlying correlations at 0.913 for DiD-averaging and 0.976 for imputation, with underlying sign-reversal probabilities of 11.5% and 5.6%.

The estimated probabilities of a statistically significant TWFE estimate pointing opposite to the modern estimand are smaller: 2.1% and 0.6%. A nonparametric alternative gives similar underlying reversal rates, but raises those significant opposite-sign probabilities to 2.4% and 1.6%.

The Takeaway: Compare, Don’t Assume

Negative weighting appears less disruptive in this replication sample than its most alarming theoretical possibilities suggest. Nevertheless, these findings do not establish that TWFE is generally safe—or that modern estimates reveal the true causal effect.

Differences can also reflect violations of parallel trends or no anticipation, differences in support, and aggregation choices. The models address sampling noise, not those identification problems, and their conclusions depend on modeling assumptions. Hull’s practical recommendation is therefore application-specific comparison: examine how estimates change, assess uncertainty, and investigate the assumptions behind each method rather than treating any estimator as an automatic guarantee of credibility.

Created on 02 Nov. 2026

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Weekly letter Presented in "Economics and finance this week" of 2 November 2026. Economics and finance this week: When the measuring tool changes the answer

Journal club Premium+ Slides to present it Propose for the journal club

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.

How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis

The essentials

Question

Method

Findings

Limitations

Why it matters

Read further

Definitions

How Much Do Negative Weights Matter in Difference-in-Differences?

The Problem: An Average That Can Reverse the Sign

A Large Replication Effort, with Important Boundaries

Most Results Align, but Some Change Direction

The Precision Tradeoff

When Are Reversals More Likely?

Looking Beneath Sampling Noise

The Takeaway: Compare, Don’t Assume

Ask questions about this paper to our AI assistant

Tools for this paper

Advanced tools Premium

Papers around this one

Closest papers

Prior work it builds on

Go further

Cite this paper (BibTeX)