Grant reviewers often disagree. It is rarely about who the reviewer is

What is it about?

When two experts score the same grant application, their scores often differ a lot. We wanted to know why. Using almost 135,000 reviews from four Norwegian research funders, we tested whether disagreement depends on the reviewers (such as their gender, age, experience and expertise), on the application, or on how the review process is organised. We fixed our questions, methods and analysis code before analysing the data, and had them peer reviewed in advance (a Registered Report). Most factors made only a small difference. Only about 3 percent of the disagreement came from stable habits of individual reviewers. About a quarter was tied to the specific application, and more than half was left unexplained, which points to ordinary variability in human judgement.

Featured Image

Photo by Gabrielle Henderson on Unsplash

Photo by Gabrielle Henderson on Unsplash

Why is it important?

Research funders distribute large sums through peer review, and a common response to disagreement is to fix the reviewers: train them, select them differently or replace them. Our results suggest this alone is unlikely to make reviews much more reliable, because so little of the disagreement is tied to the reviewers themselves. Gains are more likely to come from clearer criteria, more structured review formats and applications that leave less room for different readings, although even these should be expected to bring gradual rather than dramatic improvements. To our knowledge, this is the first study of the question to combine data from several funders with questions, methods and code fixed in advance.

Perspectives