Assassins and zealots: Variations in peer review: Special report

Radiology reviewers are required to assign numerical grades of 1 to 9 (1 = accept, 9 = reject) in the rating of manuscripts. The mean ratings for the 660 referees who were assigned 10 or more reviews over a 41/2-year period were analyzed. The mean score was 4.8 ± 0.8, and 87.4% of reviewers (the mainstream) had ratings of mean ± 1.5 standard deviations. Categories of reviewers with greater deviation from the mean were identified: zealots and pushovers, whose ratings of manuscripts were more favorable, versus assassins and demoters, who supplied less favorable ratings. To exclude the possibility that the referees who were classified as more critical had actually been sent less meritorious papers, the scores and rejection rates of 859 papers co-reviewed by assassins, demoters, and mainstream referees were compared. Significant differences were confirmed. Deviant referees were widely distributed in the pool of reviewers, including 13 members of the Editorial Board and representatives in each of 19 subspecialty areas. Failure to recognize and control for reviewer variation may be unfair to authors. An Editor has the capacity to reduce unfairness by monitoring reviewer variation and by modulating the review process accordingly.