How to help LLMs identify confusing questions

What is it about?

When you ask a search engine or chatbot a question like "Who won the first season of Hell's Kitchen?", it usually just picks one answer and moves on, even though the question could mean the US, UK, or Australian version of the show. We built a system, RAAD, that catches these hidden double-meanings before they cause a wrong answer. It works by looking up several possible answers to a question and checking whether they actually agree with each other; if they contradict, the question must have been ambiguous in the first place. This lets the system flag confusing questions automatically, using only free, publicly available tools, and it works well even on types of questions it was never specifically trained on.

Featured Image

Photo by Imkara Visual on Unsplash

Photo by Imkara Visual on Unsplash

Why is it important?

Most prior approaches try to detect ambiguity from the question's wording alone. RAAD instead checks whether the answers a question leads to actually agree with each other, treating disagreement between retrieved answers as the signal that a question was ambiguous in the first place. This reframing turns out to generalize: a version trained only on one dataset still beats existing baselines on four other, unrelated benchmarks it never saw during training, something wording-based approaches have struggled to do. That matters right now because AI assistants are answering open-ended questions at massive scale, usually by silently picking one interpretation and answering it with full confidence, leaving the user with no way to know a different, equally valid reading existed. RAAD shows this failure mode can be caught using only free, off-the-shelf tools, with no expensive retraining or proprietary models required, pointing to a cheap, practical way to make everyday QA systems ask for clarification instead of guessing.

Perspectives