Models of search engine users are not robust, and our evaluation practices do not reflect that.

What is it about?

It is a common practice to build models of how users behave when confronted to a search engine result page. This usually involves defining assumptions on the causal factors that influence user behavior. Once the assumptions are laid out, the parameters of the model are fitted on real user data, in the hope that we can recover an accurate estimations of the underlying causal factors, such as relevance or examination probability. In this paper, we show that the common evaluation practice for the relevance estimation does not detect a lack of robustness of these models, i.e., that they fail to predict how users would behave if confronted to a different search engine algorithm.

Featured Image