Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments

Vasilis Syrgkanis; Victor Lei; Miruna Oprescu; Maggie Hei; Keith Battocchi; Greg Lewis

AI summary · arXiv: 1905.10176v3 · econ.EM

Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments

Vasilis Syrgkanis, Victor Lei, Miruna Oprescu, Maggie Hei, Keith Battocchi, Greg Lewis

Original paper Submitted to arXiv on 24 May. 2019 License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

The paper proposes a statistical learning approach to estimate heterogeneous treatment effects using arbitrary machine learning methods in the presence of unobserved confounders with the aid of a valid instrument.
The authors focus on A/B tests with an intent-to-treat structure where they are interested in the effect of the downstream action.
The proposed method reduces the problem to minimizing an appropriate loss function that depends on a set of auxiliary models corresponding to separate prediction tasks, enabling the use of recent algorithmic advances such as neural nets and forests.
The authors show that their estimated effect model is robust to estimation errors in auxiliary models by satisfying a Neyman orthogonality criterion.
This approach can be used to estimate projections of the true effect model on simpler hypothesis spaces, and parameter estimates are asymptotically normal enabling construction of confidence sets when these spaces are parametric.
To validate their method, they applied it to estimate the effect of membership on downstream webpage engagement on TripAdvisor using an intent-to-treat A/B test among 4 million users where some users received an easier membership sign-up process. They also validated their method on synthetic data and public datasets for effects of schooling on income.
Overall, this paper provides a novel statistical learning approach for estimating heterogeneous treatment effects that can be applied in various settings beyond A/B tests with intent-to-treat structures.

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Heterogeneous Treatment Effects Machine Learning Instrument A/B Test Neyman Orthogonality

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

The paper titled "Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments" by Vasilis Syrgkanis, Victor Lei, Miruna Oprescu, Maggie Hei, Keith Battocchi and Greg Lewis proposes a statistical learning approach to estimate heterogeneous treatment effects using arbitrary machine learning methods in the presence of unobserved confounders with the aid of a valid instrument. The authors focus on A/B tests with an intent-to-treat structure where the experimenter randomizes over which user will receive a recommendation to take an action and they are interested in the effect of the downstream action. The proposed method reduces the problem to minimizing an appropriate loss function that depends on a set of auxiliary models corresponding to separate prediction tasks. This reduction enables the use of all recent algorithmic advances such as neural nets and forests. The authors show that their estimated effect model is robust to estimation errors in auxiliary models by satisfying a Neyman orthogonality criterion. Furthermore, this approach can be used to estimate projections of the true effect model on simpler hypothesis spaces. When these spaces are parametric, parameter estimates are asymptotically normal enabling construction of confidence sets. To validate their method, they applied it to estimate the effect of membership on downstream webpage engagement on TripAdvisor using an intent-to-treat A/B test among 4 million users where some users received an easier membership sign-up process. They also validated their method on synthetic data and public datasets for effects of schooling on income. Overall, this paper provides a novel statistical learning approach for estimating heterogeneous treatment effects that can be applied in various settings beyond A/B tests with intent-to-treat structures.

Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments

In this paper, Vasilis Syrgkanis, Victor Lei, Miruna Oprescu, Maggie Hei, Keith Battocchi and Greg Lewis propose a statistical learning approach to estimate heterogeneous treatment effects using arbitrary machine learning methods in the presence of unobserved confounders with the aid of a valid instrument. This approach is applicable to A/B tests with an intent-to-treat structure where experimenters randomize over which user will receive a recommendation to take an action and they are interested in the effect of the downstream action.

Background

The authors focus on estimating heterogeneous treatment effects (HTE) which measure how different individuals respond differently to treatments. To accurately estimate HTEs requires controlling for any unobserved confounding factors that may influence both treatment assignment and outcomes. However, it is often difficult or impossible to directly observe these confounders due to privacy concerns or lack of data availability. The authors propose a method that uses instruments - variables that are correlated with treatments but not outcomes - as proxies for these unobserved confounders in order to accurately estimate HTEs even when direct observations are unavailable.

Methodology

The proposed method reduces the problem of estimating HTEs into minimizing an appropriate loss function that depends on a set of auxiliary models corresponding to separate prediction tasks. This reduction enables the use of all recent algorithmic advances such as neural nets and forests for estimation purposes. The authors show that their estimated effect model is robust to estimation errors in auxiliary models by satisfying a Neyman orthogonality criterion. Furthermore, this approach can be used to estimate projections of the true effect model on simpler hypothesis spaces; when these spaces are parametric, parameter estimates are asymptotically normal enabling construction of confidence sets around them.

Validation

To validate their method, they applied it to estimate the effect of membership on downstream webpage engagement on TripAdvisor using an intent-to-treat A/B test among 4 million users where some users received an easier membership sign-up process than others did not receive any recommendation at all . They also validated their method on synthetic data and public datasets for effects of schooling on income from various countries including Brazil and India . Overall , this paper provides a novel statistical learning approach for estimating heterogeneous treatment effects that can be applied in various settings beyond A/B tests with intent-to-treat structures .

Created on 19 May. 2023

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Journal club Premium+ Slides to present it Propose for the journal club

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

Close
Econometric Inference with Machine-Learned Proxies: Partial Identification via Data Combination
Lixiong Li · 2026 · Econometrics · AI summary available

Abstract
Empirical researchers increasingly use upstream machine-learning (ML) methods to construct proxies for latent target variables from complex, unstructured data. A naive plug-in use of such proxies in downstream econometric models, however, can lead to biased estimation and invalid inference. This paper develops a framework for partial identification and inference in general moment models with ML-generated proxies. Our approach does not require restrictive assumptions on the upstream ML procedure, such as consistency or known convergence rates, nor does it require a complete validation sample containing all variables used in the downstream analysis. Instead, we assume access to two datasets: a…
Close
A multi-cell experimental design to recover policy relevant treatment effects, with an application to online advertising
Caio Waisman, Brett R. Gordon · 2023 · Econometrics · AI summary available

Abstract
Experiments are an important tool to measure the impacts of interventions. However, in experimental settings with one-sided noncompliance, extant empirical approaches may not produce the estimates a decision-maker needs to solve their problem. For example, these experimental designs are common in digital advertising settings, but they are uninformative of decisions regarding the intensive margin -- how much should be spent or how many consumers should be reached with a campaign. We propose a solution that combines a novel multi-cell experimental design with modern estimation techniques that enables decision-makers to recover enough information to solve problems with an intensive margin. Our …
Close
Robust Synthetic Control
Muhammad Jehangir Amjad, Devavrat Shah, Dennis Shen · 2017 · Econometrics · AI summary available

Abstract
We present a robust generalization of the synthetic control method for comparative case studies. Like the classical method, we present an algorithm to estimate the unobservable counterfactual of a treatment unit. A distinguishing feature of our algorithm is that of de-noising the data matrix via singular value thresholding, which renders our approach robust in multiple facets: it automatically identifies a good subset of donors, overcomes the challenges of missing data, and continues to work well in settings where covariate information may not be provided. To begin, we establish the condition under which the fundamental assumption in synthetic control-like approaches holds, i.e. when the lin…
Related
Bounds on Distributional Treatment Effect Parameters using Panel Data with an Application on Job Displacement
Brantly Callaway · 2020 · Econometrics · AI summary available

Abstract
This paper develops new techniques to bound distributional treatment effect parameters that depend on the joint distribution of potential outcomes -- an object not identified by standard identifying assumptions such as selection on observables or even when treatment is randomly assigned. I show that panel data and an additional assumption on the dependence between untreated potential outcomes for the treated group over time (i) provide more identifying power for distributional treatment effect parameters than existing bounds and (ii) provide a more plausible set of conditions than existing methods that obtain point identification. I apply these bounds to study heterogeneity in the effect of …
Related
Adaptive Pricing in Insurance: Generalized Linear Models and Gaussian Process Regression Approaches
Yuqing Zhang, Neil Walton · 2019 · Econometrics · AI summary available

Abstract
We study the application of dynamic pricing to insurance. We view this as an online revenue management problem where the insurance company looks to set prices to optimize the long-run revenue from selling a new insurance product. We develop two pricing models: an adaptive Generalized Linear Model (GLM) and an adaptive Gaussian Process (GP) regression model. Both balance between exploration, where we choose prices in order to learn the distribution of demands & claims for the insurance product, and exploitation, where we myopically choose the best price from the information gathered so far. The performance of the pricing policies is measured in terms of regret: the expected revenue loss cause…
Related
Nonparametric Tests of Tail Behavior in Stochastic Frontier Models
William, C. Horrace, Yulong Wang · 2020 · Econometrics · AI summary available

Abstract
This article studies tail behavior for the error components in the stochastic frontier model, where one component has bounded support on one side, and the other has unbounded support on both sides. Under weak assumptions on the error components, we derive nonparametric tests that the unbounded component distribution has thin tails and that the component tails are equivalent. The tests are useful diagnostic tools for stochastic frontier analysis. A simulation study and an application to a stochastic cost frontier for 6,100 US banks from 1998 to 2005 are provided. The new tests reject the normal or Laplace distributional assumptions, which are commonly imposed in the existing literature.
Related
Robust decision-making under risk and ambiguity
Maximilian Blesch, Philipp Eisenhauer · 2021 · Econometrics · AI summary available

Abstract
Economists often estimate economic models on data and use the point estimates as a stand-in for the truth when studying the model's implications for optimal decision-making. This practice ignores model ambiguity, exposes the decision problem to misspecification, and ultimately leads to post-decision disappointment. Using statistical decision theory, we develop a framework to explore, evaluate, and optimize robust decision rules that explicitly account for estimation uncertainty. We show how to operationalize our analysis by studying robust decisions in a stochastic dynamic investment model in which a decision-maker directly accounts for uncertainty in the model's transition dynamics.
Related
Deep Learning Macroeconomics
Rafael R. S. Guimaraes · 2022 · Econometrics · AI summary available

Abstract
Limited datasets and complex nonlinear relationships are among the challenges that may emerge when applying econometrics to macroeconomic problems. This research proposes deep learning as an approach to transfer learning in the former case and to map relationships between variables in the latter case. Although macroeconomists already apply transfer learning when assuming a given a priori distribution in a Bayesian context, estimating a structural VAR with signal restriction and calibrating parameters based on results observed in other models, to name a few examples, advance in a more systematic transfer learning strategy in applied macroeconomics is the innovation we are introducing. We expl…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

For now only among the papers summarized on this site: search again to add the prior work, the follow-ups and Semantic Scholar’s recommendations.

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.