
Low interrater reliability: The correlation between the judgments of two peers is
low. In 16 studies, Starbuck (2005, 2015) found a mean correlation between
reviewers of only 0.18. In clinical neuroscience the correlation between reviewers
“was little greater than would be expected by chance alone” (Rothwell and Martyn
2000, p. 1964). Luck of the referee drawn plays a big role (Bornmann and Daniel
2009).