What is it about?
The Power of Noise Effect showcased that adding random documents in RAG scenario can help LLMs to extract information from retrieved context. This paper checks the experimental setup verifying the generalizability of the finding.
Featured Image
Why is it important?
The Power of Noise started a wave of research building on top of this finding. Our works suggests that the original effect is closely tied to the experimental conditions under which it was observed. When we relax this constraint and evaluate on newer models, the apparent benefit of irrelevant documents disappears.
Perspectives
The story of how this was discovered is what makes it interesting to me. The effect itself looked strange enough that I wanted to redo it, and once I started reading the actual generations I noticed a lot of them were off; trimmed outputs, small hallucinations. Once I connected those oddities to specific choices in the setup and started tweaking, they disappeared, and the results ended up telling a different story. The takeaway for me is just how much it pays to look at the real outputs and stay critical about your setup, since it's easy to introduce biases that end up shaping your results.
Michał Mazuryk
University of Amsterdam
Read the Original
This page is a summary of: The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise, July 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3805712.3808556.
You can read the full text:
Contributors
The following have contributed to this page
