NIH has posted additional scoring guidance for reviewers to consider when determining overall impact scores for grant applications. Here are answers to some key questions about this new guidance.
Why provide more guidelines?
Scientific review officers and program directors noticed that reviewers have tended to arrive at overall impact scores by comparing the number of weaknesses and strengths in an application, rather than balancing the importance of the weaknesses and strengths. In addition, there was significant compression of overall impact scores around the perceived funding range. Both practices made it more difficult to gauge reviewer assessments.
What’s changed?
The additional guidance chart simplifies and clarifies the way in which reviewers should evaluate the overall impact of an application. In particular, it encourages reviewers to focus on the importance of the research problem and the likelihood that the project will succeed. The chart emphasizes a balanced assessment of the review criteria and the use of the entire scoring range. The general Scoring System and Procedure also includes a new, simplified scoring guidance chart for assigning individual criterion scores.
When will this change occur?
Some Center for Scientific Review (CSR) study sections used the new guidelines in application reviews for May 2013 advisory council meetings. So that these applications would not be disadvantaged by the deviation from the impact score distribution of previous review cycles used for percentiling, CSR recalculated the percentile base. The vast majority of study sections and special emphasis panels are using the new scoring guidance with the reviews for fall 2013 advisory council meetings.
Note: New scoring guidelines are also available for fellowships, career awards and institutional training grants.
The new scoring guidelines are consistent with NIH’s continuing desire to believe that it is funding the best projects. Unfortunately, the reality is that most applications are of fundable quality. When combined with the fact that nearly all proposals are read in detail by only two or three reviewers, whether or not a grant gets a fundable score is more a crap shoot than a rational decision based on careful comparisons of all proposals. In addition, the emphasis on likelihood of success is a killer for any innovative project; it almost guarantees that only projects that have almost been completed will receive fundable scores.