What is it about?
The objective of this study is to propose a mathematical framework to evaluate the impact caused by continuous long-term intervention (treatment e.g., product updates) in A/B testing (also called random controlled trials or online controlled experiments) by combining short-term outcomes and historical data.
Featured Image
Photo by path digital on Unsplash
Why is it important?
Nowadays, A/B testing has become the standard testing process before product version updating and has brought millions of additional revenues for companies. To reduce unnecessary costs and to accelerate product iterations, each A/B test lasts relatively short (e.g., 2 weeks). However, it is the long-term performance of treatment that reliably reflects management interests and the overall development strategy. Companies, therefore, urgently desire an effective approach that accurately assesses the performance of long-term intervention (e.g., the long-term effectiveness of product updates) in A/B testing with low costs, such as using only short-term experiments and historical data.
Perspectives
The motivation for this study arises from our consulting experience with companies. During our interactions, we consistently encountered a question from management at different levels: How can they trust the short-term experimental results when implementing long-term product interventions? They were also curious whether the short-term performance would remain stable over time. In response to these concerns, we embarked on developing a method to enhance the trustworthiness of AB testing. Our aim was to create a framework that could extrapolate the future long-term treatment effects immediately after conducting a short-term experiment. We hope that our method will prove valuable to a wider audience, including individuals and companies seeking to improve their understanding and decision-making in AB testing.
Shan Huang
University of Hong Kong
Read the Original
This page is a summary of: Estimating Effects of Long-Term Treatments, July 2023, ACM (Association for Computing Machinery),
DOI: 10.1145/3580507.3597701.
You can read the full text:
Contributors
The following have contributed to this page
