Bots and Surveys: Assessing the Impact of Bad Actors on Preference Data

What is it about?

The study analyzed data from an online survey to evaluate the impact of bad actors on the quality of data collected. The survey included a discrete-choice experiment to understand preferences for multiple myeloma therapies. The results showed a significant proportion of respondents provided inconsistent answers, which could indicate the presence of bad actors. The study also found that a wide-ranging selection of conditions in the survey screener was more consistent with choice patterns expected from bad actors. The researchers concluded that the impact of bad actors on preference estimates can be significant, and appropriate measures should be taken to address this issue in online preference surveys.

Featured Image

Why is it important?

This research is crucial because it reveals the potential impact of bad actors on the quality of preference data collected through online surveys, particularly in the context of medical research. Recognizing and addressing the presence of bad actors can help ensure the reliability and validity of survey results and maintain trust in the data. Key Takeaways: 1. The online administration of surveys can be vulnerable to bad actors (human and non-human bots) trying to influence study results or profit from the survey. 2. A latent-class model was used to identify respondents with perverse preferences or high model variance, and their impact on survey quality was assessed. 3. A significant proportion of respondents provided answers with a high degree of variability, consistent with responses from bad actors. 4. A wide-ranging selection of conditions in the survey screener was more consistent with choice patterns expected from bad actors. 5. Incorrect answers to comprehension questions peaked around 5 out of 10 questions in terms of their relationship with problematic choice patterns. 6. Completion speed had no significant effect on the quality of preference information. 7. The study highlights the need for a robust discussion on handling bad actors in online preference surveys and emphasizes the importance of addressing potential issues in survey design and data analysis.