NIH has posted a video on the Enhancing Peer Review Web site that details recent changes to the peer review system from the reviewer’s perspective. Running about 13 minutes, “What Reviewers Need to Know Now”, offers valuable information for current and prospective peer reviewers. You’ll find other resources in the PowerPoint presentation and related links that accompany the video.
Many aspects of the new peer review system are already being used in study section meetings. If you have participated in any of these meetings, I am very interested in your comments and reactions.
I’d also like to mention that NIH’s Center for Scientific Review, NIGMS and other institutes and centers have a critical need right now for reviewers to help evaluate the tremendous number of applications submitted in response to Recovery Act funding opportunities. If a Scientific Review Officer asks you to participate on a review panel, I hope that you’ll agree to serve if at all possible.
I recently participated in a study section review and came away with 2 suggestions for improvement:
1) Reviewers should be allowed finer distinctions in the scoring, e.g. allow half points rather than just whole numbers. In the old scoring system from 1 to 5 in tenths, one could make 41 distinct scores. Even if the entire range wasn’t used, 1 to 3 gives 21 scores. Nine is not enough to parse the differences between grants. It is frustrating to have to score two grants the same when you feel that one is definitely superior to the other!!!
I feel strongly about this point!!
2) I would not use bullet points and just let reviewers comment in the review as in the olde days. Applicants should be allowed to get the nuances of the comments, not just the bottom line.
I just returned from the MGC study section panel. I thought the use of bullet points focusing on strengths and weaknesses really helped the discussions of the grants so that there was much less specific description of the nitpicky details of each grant. I do have one suggestion. When all three reviewers agreed on the impact score, there was no range. So if, for example, everyone gave the grant a 3, the range was 3, and all of the panel members were expected to vote 3. If one wanted to vote one point up or down, one had to declare that and justify it. Some people will not want to do that because of peer pressure. My suggestion is that in cases where there is no range because all of the scores are the same, that people be allowed to vote one point up or down from that score without having to announce it.
The new “rule” that allows a particular grant to be submitted and reviewed only 2 times, rather than 3, is not good for science! Many excellent grants that should get funded in my study section fall below the pay line sometimes even after 3 submissions! That science should be funded–and therefore should continue to compete! Every Study Section is different, both in terms of reviewers and in terms of the competing pool of applicants (I have been on some study sections where virtually no grant should be funded and others where ~50% should be!). Get real, please. That new rule is plain stupid and there is no good defense for it.
One of the most frustrating part of the review process is when comments are made about points that are actually fully covered in the proposal, but were apparently missed by the reviewers. Having to address this via revision is inefficient since often new reviewers will re-read a revised proposal several months later, thus creating long and undesired delays. With the new possibilities for reviewing proposals electronically, I think that it would be very good if the PI had a chance to read the comments of the reviewers before the study section meeting (like 48 hours), and be allowed a brief response/clarification/rebuttal (like 1/2 page or 250 words max…) that would be accessible to the review panel. This would help re-focus an issue with the accurate facts of the proposal.