A new approach for Brand Lift Testing to quantify the effectiveness of online advertising

“

Unlike most inference strategies for Online Experiments that focus on unbiased tests, we propose a slightly biased test that is superior from a practical perspective when prior knowledge about the alternative hypothesis is available. By sacrificing power in less relevant regions in the parameter space, we achieve higher power in the region where the test owners really care about. It would be interesting to embed this view into a formal decision framework and incorporate such prior knowledge in a more coherent manner. Our current discussion only addresses brand lift surveys with binary responses. An immediate next step is to generalize the test to handle more flexible survey setups.

”