Estimating and Penalizing Induced Preference Shifts in Recommender Systems

What is it about?

User preferences over the content they want to watch (or read, or purchase) are non-stationary. Further, the actions that a recommender system (RS) takes -- the content it exposes users to -- plays a role in \emph{changing} these preferences. Therefore, when an RS designer chooses which system or policy to deploy, they are implicitly \emph{choosing how to shift} or influence user preferences. Even more, if the RS is trained via long-horizon optimization (e.g. reinforcement learning), it will have incentives to manipulate user preferences -- shift them so they are more easy to satisfy, and thus conducive to higher reward. While some work has argued for making systems myopic to avoid this issue, the reality is that such systems will still influence preferences, sometimes in an undesired way. In this work, we argue that we need to enable system designers to 1) estimate the shifts an RS would induce, 2) evaluate, before deployment, whether the shifts are undesirable, and even 3) actively optimize to avoid such shifts. These steps involve two challenging ingredients: (1) requires the ability to anticipate how hypothetical policies would influence user preferences if deployed; instead, (2) requires metrics to assess whether such influences are manipulative or otherwise unwanted. We study how to do (1) from historical user interaction data by building a user predictive model that implicitly contains their preference dynamics; to address (2), we introduce the notion of a “safe policy”, which defines a trust region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable), while still generating engagement.

Featured Image

Photo by dole777 on Unsplash

Photo by dole777 on Unsplash