# Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

**Authors:** Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, Nathaniel Stevens  
**arXiv:** [2212.11366](https://arxiv.org/abs/2212.11366)  
**Date:** 2022-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.11366v1/

## Key points

- Impact of internet-based services and products on online businesses' decision-making processes
- Growth of online controlled experiments (OCEs) by major organizations like Airbnb, Amazon, Google, Facebook (Meta), etc.
- Challenges in implementing OCEs at scale across various domains
- Importance of innovative statistical methodologies to derive meaningful insights from online experimentation data
- Review of current statistical methodologies used in online experimentation
- Discussion on practical aspects, cultural implications, and existing statistics literature related to conducting online experiments
- Illustrative examples of OCE applications from industry giants like Airbnb and Amazon
- Call for collaboration between academic statisticians and the online industry to drive innovation in statistical methodology for OCEs

## Summary

In their paper titled "Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology," authors Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel Stevens delve into the impact of internet-based services and products on online businesses' decision-making processes. The late 1990s marked a turning point as organizations like Airbnb, Alibaba, Amazon, Baidu, Booking.com, Google, LinkedIn, Lyft, Facebook (Meta), Microsoft, Netflix, Twitter, Uber and Yandex began investing heavily in online controlled experiments (OCEs) to evaluate the effects of innovation on their customer base and overall business strategies. Over the past two decades,the implementation of OCEs at scale has presented numerous challenges that span various domains. To address these challenges effectively and derive meaningful insights from the data collected through online experimentation practices,new statistical methodologies have become essential. The authors highlight the need for innovative statistical approaches to tackle these challenges and provide a comprehensive review of the current methodologies used in online experimentation. The paper not only discusses the practical aspects and cultural implications of conducting online experiments,but also delves into the existing statistics literature related to this field.By placing current methodologies within their relevant statistical lineages and offering illustrative examples of OCE applications from industry giants mentioned earlier such as Airbnb or Amazon -the authors aim to raise awareness among academic statisticians about emerging research opportunities in collaboration with the online industry. Overall,this paper serves as a valuable resource for researchers looking to explore new avenues in statistical methodology within the realm of online controlled experiments.It underscores the importance of bridging academia with industry practices to drive innovation and enhance decision-making processes in today's digital landscape.