When AI Makes Mistakes, Explanations Help Most When People Can Still Check the Answer

What is it about?

Telling people where an AI tends to go wrong helps them catch its mistakes—but only for tasks of moderate difficulty. When a task is very easy, people spot errors anyway; when it's too hard to judge, they can't verify the AI at all and increasingly just defer to it, which raises doubts about whether humans can truly "oversee" AI on tasks beyond their own abilities.

Featured Image

Photo by Growtika on Unsplash

Photo by Growtika on Unsplash

Why is it important?

Explainable AI is often proposed as a way to help people identify when AI systems make mistakes, but it has remained unclear when explanations are actually effective. Across three experiments, we show that explanations about an AI system's weaknesses are most helpful when errors are difficult but still possible for people to verify. In contrast, explanations provide little benefit when errors are either obvious or practically impossible to detect. These findings suggest that the value of explainable AI depends on the difficulty of the decision task and the user's ability to independently verify AI recommendations. This has important implications for the design of AI decision support systems in domains such as healthcare, where improving human oversight of AI errors is critical.