AI summary · arXiv: 2611.11003v1 · econ.EM
Do Modern Difference-in-Differences Estimators Change Published Findings in Business Research?
The essentials
In reanalyzed business-journal studies, modern difference-in-differences estimators often disagree with two-way fixed effects, especially in unbalanced panels and designs without never-treated units.
Question
Do the negative-weight concerns associated with two-way fixed-effects (TWFE) difference-in-differences estimators materially change findings in published business research? The paper compares published staggered-adoption estimates with heterogeneity-robust alternatives.
Method
The study uses language-model agents to screen articles and replication packages, then re-estimates 355 specifications from 40 papers in Management Science and Marketing Science. It compares TWFE with the Callaway–Sant’Anna DiD-averaging estimator and the Borusyak et al. imputation estimator; the main leverage-screened comparisons comprise 283 and 306 pairs from 28 and 29 papers, respectively.
Findings
-
Result
TWFE and modern estimates have correlations of 0.61 for DiD-averaging and 0.70 for imputation in the screened comparisons.
See the passage in the paper
The correlation between TWFE and modern estimates is 0.61 for DiD-averaging and 0.70 for imputation (Figure 1 plots the estimates).
-
Result
Estimates reverse sign in 25.1% of DiD-averaging comparisons and 18.0% of imputation comparisons; significant reversals occur in 10.2% and 7.2%, respectively.
See the passage in the paper
Estimates change sign in 25.1 percent of DiD-averaging comparisons and 18.0 percent of imputation comparisons. Reversals accompanied by a statistically significant difference between the two estimates are rarer, at 10.2 and 7.2 percent.
-
Result
Disagreement is greater in unbalanced panels: estimator correlations fall to 0.47 for DiD-averaging and 0.55 for imputation.
See the passage in the paper
In balanced panels and repeated cross-sections, the correlations between TWFE and modern estimates are 0.92 for DiD-averaging and 0.99 for imputation, close to the economics benchmarks. In unbalanced panels they are 0.47 and 0.55.
-
Result
Modern estimators have larger standard errors on average than TWFE, by factors of 1.61 for DiD-averaging and 1.30 for imputation.
See the passage in the paper
Standard errors of the DiD-averaging estimator are on average 1.61 times the TWFE standard error, and those of the imputation estimator 1.30 times, compared with 1.47 and 1.14 in Hull (2026).
-
Authors’ interpretation
The authors conclude that disagreement is more frequent than in the economics benchmark and is concentrated in unbalanced panels, designs without never-treated units, and a few papers.
See the passage in the paper
First, where staggered designs in business-school journals can be re-estimated on original data, TWFE and heterogeneity-robust estimators disagree more often than in economics.
Limitations
- The main comparisons cover only 28 and 29 papers from two journals, so individual papers can materially affect summary statistics.
See the passage in the paper
With 28 and 29 papers from two journals behind the main comparisons, single papers move the summary statistics, as Appendix C shows.
- The candidate search was not exhaustive, and screening depended on language-model agents working from article text and replication packages.
See the passage in the paper
The screening relied on language model agents working from article text and packages. The check of 343 articles outside the candidate list added 2 eligible designs, so the candidate search is not exhaustive.
- The reanalysis is limited to designs with usable original data; many eligible designs had synthetic or disguised data, missing analysis data, or no replication package.
See the passage in the paper
Of the 92 papers with an eligible design, 18 come with synthetic, disguised, or pseudo data, posted under the journals’ alternative disclosure plans for proprietary data.
Why it matters
The paper provides a business-research benchmark for how often TWFE and heterogeneity-robust estimators diverge, finding more disagreement than the cited economics study. The authors recommend reporting a heterogeneity-robust estimate alongside TWFE, particularly for unbalanced panels and designs without never-treated units.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 18 papers around this one, the name that comes back most: Pedro H. C. Sant'Anna (3 papers)
2 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
How Different are Different Differences-in-Differences? A Large-Scale Meta-AnalysisPeter Hull · 2026 · Econometrics · cited 1 time · AI summary available
How Different are Different Differences-in-Differences? compares TWFE and modern DiD estimates across economics studies, while this paper conducts a parallel reanalysis in business journals.
In short
Across 1,346 replicated specifications, TWFE and modern difference-in-differences estimates largely agree, with TWFE offering greater precision and hierarchical models suggesting infrequent statistically significant wrong-sign estimates.
-
When Can We Use Two-Way Fixed Effects (TWFE)? A Comparison of TWFE and Novel Dynamic Difference-in-Differences EstimatorsTobias Rüttenauer, O. Aksoy · 2026 · Sociological methodology · Not summarized yet
When Can We Use Two-Way Fixed Effects reviews TWFE limitations and compares five dynamic DiD estimators through simulations.
Abstract
The conventional two-way fixed effects (TWFE) estimator has come under increasing scrutiny in settings with staggered treatment adoption. Recent research shows that TWFE can be biased when treatment effects vary across time or groups, prompting the development of several alternative dynamic difference-in-differences (DiD) estimators. Yet for applied researchers, it often remains unclear when TWFE is biased, what the newer estimators solve, and what limitations they all have. The authors provide an accessible overview of this literature and compare conventional TWFE with five dynamic DiD estimators using Monte Carlo simulations. The simulations consider realistic scenarios with time- and grou…
-
How Much Should We Trust Staggered Difference-In-Differences Estimates?A. Baker, D. Larcker, Charles C. Y. Wang · 2021 · Social Science Research Network · cited 2,259 times · Not summarized yet
How Much Should We Trust Staggered Difference-In-Differences Estimates? examines bias from heterogeneous effects and applies newer methods to published corporate finance and accounting results.
Abstract
Difference-in-differences analysis with staggered treatment timing is frequently used to assess the impact of policy changes on corporate outcomes in academic research. However, recent advances in econometric theory show that such designs are likely to be biased in the presence of treatment effect heterogeneity. Given the pronounced use of staggered treatment designs in applied corporate finance and accounting research, this finding potentially impacts a large swath of prior findings in these fields. We survey the nascent literature and document how and when such bias arises from treatment effect heterogeneity. We apply recently proposed methods to a set of prior published results, and find …
-
Difference-in-Differences with compositional changesPedro H. C. Sant'Anna, Qi Xu · 2023 · Journal of Econometrics · cited 7 times · Not summarized yet
Difference-in-Differences with compositional changes develops estimators for repeated cross-sections with changing composition, rather than comparing published staggered-adoption estimates.
Abstract
This paper studies Difference-in-Differences (DiD) setups with repeated cross-sectional data and potential compositional changes across time periods. We begin our analysis by deriving the efficient influence function and the semiparametric efficiency bound for the average treatment effect on the treated (ATT). We introduce nonparametric estimators that attain the semiparametric efficiency bound under mild rate conditions on the estimators of the nuisance functions, exhibiting a type of rate doubly robust (DR) property. Additionally, we document a trade-off related to compositional changes: We derive the asymptotic bias of DR DiD estimators that erroneously exclude compositional changes and t…
-
Two-stage differences in differencesJ. Gardner · 2022 · cited 237 times · Not summarized yet
Two-stage differences in differences proposes an alternative estimator for staggered adoption with heterogeneous effects, a problem also addressed by the modern estimators compared here.
Abstract
A recent literature has shown that when adoption of a treatment is staggered and average treatment effects vary across groups and over time, difference-in-differences regression does not identify an easily interpretable measure of the typical effect of the treatment. In this paper, I extend this literature in two ways. First, I provide some simple underlying intuition for why difference-in-differences regression does not identify a group × period average treatment effect. Second, I propose an alternative two-stage estimation framework, motivated by this intuition. In this framework, group and period effects are identified in a first stage from the sample of untreated observations, and average treatmen…
-
Difference-in-Differences Designs: A Practitioner's GuideA. Baker, Brantly Callaway, Scott Cunningham et al. · 2025 · cited 50 times · Not summarized yet
Difference-in-Differences Designs: A Practitioner's Guide organizes DiD designs and estimators, including multiple periods and staggered treatments.
Abstract
Difference-in-differences (DiD) is arguably the most popular quasi-experimental research design. Its canonical form, with two groups and two periods, is well-understood. However, empirical practices can be ad hoc when researchers go beyond that simple case. This article provides an organizing framework for discussing different types of DiD designs and their associated DiD estimators. It discusses covariates, weights, handling multiple periods, and staggered treatments. The organizational framework, however, applies to other extensions of DiD methods as well.
-
What’s trending in difference-in-differences? A synthesis of the recent econometrics literatureJ. Roth, Pedro H. C. Sant'Anna, A. Bilinski et al. · 2022 · Journal of Econometrics · cited 1,835 times · Not summarized yet
What’s trending in difference-in-differences synthesizes recent DiD methods, including approaches to multiple periods and variation in treatment timing.
Abstract
This paper synthesizes recent advances in the econometrics of difference-in-differences (DiD) and provides concrete recommendations for practitioners. We begin by articulating a simple set of ``canonical'' assumptions under which the econometrics of DiD are well-understood. We then argue that recent advances in DiD methods can be broadly classified as relaxing some components of the canonical DiD setup, with a focus on $(i)$ multiple periods and variation in treatment timing, $(ii)$ potential violations of parallel trends, or $(iii)$ alternative frameworks for inference. Our discussion highlights the different ways that the DiD literature has advanced beyond the canonical model, and helps to…
-
How Much Should We Trust Differences-in-Differences Estimates?Marianne Bertrand, E. Duflo, S. Mullainathan · 2004 · National Bureau of Economic Research · cited 11,440 times · Not summarized yet
How Much Should We Trust Differences-in-Differences Estimates? studies serial-correlation bias in DiD standard errors, not negative weights from staggered adoption.
Abstract
Most Difference-in-Difference (DD) papers rely on many years of data and focus on serially correlated outcomes. Yet almost all these papers ignore the bias in the estimated standard errors that serial correlation introduce4s. This is especially troubling because the independent variable of interest in DD estimation (e.g., the passage of law) is itself very serially correlated, which will exacerbate the bias in standard errors. To illustrate the severity of this issue, we randomly generate placebo laws in state-level data on female wages from the Current Population Survey. For each law, we use OLS to compute the DD estimate of its 'effect' as well as the standard error for this estimate. The …
Prior work it builds on
-
Negative Weights are No Concern in Design-Based SpecificationsK. Borusyak, P. Hull · 2024 · Social Science Research Network · cited 12 times · Not summarized yet
Negative Weights are No Concern in Design-Based Specifications analyzes negative weights in partially linear and design-based specifications, rather than re-estimating published staggered DiD results.
Abstract
Recent work shows that popular partially-linear regression specifications can put negative weights on some treatment effects, potentially producing incorrectly-signed estimands. We show this is not an issue in design-based specifications, in which low-dimensional controls span the conditional expectation of the treatment. Specifically, the estimands of such specifications are convex averages of causal effects with ex-ante weights that average the potentially negative ex-post weights across possible treatment realizations. This result extends to design-based instrumental variable estimands under a first-stage monotonicity condition and applies to formula treatments and instruments such as shi…
-
Causal Panel Analysis under Parallel Trends: Lessons from a Large Reanalysis StudyAlbert Chiu, Xingchen Lan, Ziyi Liu et al. · 2023 · American Political Science Review · cited 50 times · Not summarized yet
Causal Panel Analysis under Parallel Trends reanalyzes published TWFE studies with robust estimators to assess the reliability of existing causal findings.
Abstract
Two-way fixed effects (TWFE) models are widely used in political science to establish causality, but recent methodological discussions highlight their limitations under heterogeneous treatment effects (HTE) and violations of the parallel trends (PT) assumption. This growing literature has introduced numerous new estimators and procedures, causing confusion among researchers about the reliability of existing results and best practices. To address these concerns, we replicated and reanalyzed 49 studies from leading journals that employ TWFE models for causal inference using observational panel data with binary treatments. Using six HTE-robust estimators, diagnostic tests, and sensitivity analy…
-
Estimating dynamic treatment effects in event studies with heterogeneous treatment effectsSarah C. Abraham, Li-Yang Sun · 2018 · Journal of Econometrics · cited 4,821 times · Not summarized yet
Estimating dynamic treatment effects in event studies with heterogeneous treatment effects shows how TWFE estimates can combine heterogeneous effects in misleading ways.
Abstract
Event studies are frequently used to estimate average treatment effects on the treated (ATT). In estimating the ATT, researchers commonly use fixed effects models that implicitly assume constant treatment effects across cohorts. We show that this is not an innocuous assumption. In fixed effect models where the sole regressor is treatment status, the OLS coefficient is a non-convex average of the heterogeneous cohort-specific ATTs. When regressors containing lags and leads of treatment are added, the OLS coefficient corresponding to a given lead or lag picks up spurious terms consisting of treatment effects from other periods. Therefore, estimates from these commonly used models are not causa…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 16 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.