What is it about?
Online platforms run many experiments to see how changes affect users, but oftentimes these changes don’t show obvious results right away. Rather than wait for long-term effects to materialize, companies often look at short-term proxies, which can also be more sensitive. For example, user engagement is a sensitive, short-term proxy for long-term user retention. However, extrapolating from these proxies can be tricky. Our research develops better methods to combine results from many small experiments to make more accurate predictions about long-term effects, which ultimately helps companies make smarter decisions faster.
Featured Image
Photo by Venti Views on Unsplash
Why is it important?
Online platforms have undergone tremendous growth in recent decades, much of which has been fueled by A/B testing. Our methods help these companies learn better insights from historical A/B tests and leverage these to accelerate decision-making in new tests.
Perspectives
It is very easy to fall into some common traps when analyzing historical A/B tests (also known as meta-analysis). Our paper provides a clear explanation for why these traps can lead to mistaken inferences and helps researchers overcome them. We also describe how we put these methods into practice at Netflix.
Winston Chou
Read the Original
This page is a summary of: Learning the Covariance of Treatment Effects Across Many Weak Experiments, August 2024, ACM (Association for Computing Machinery),
DOI: 10.1145/3637528.3672034.
You can read the full text:
Contributors
The following have contributed to this page
