At the recent meeting of the National Advisory General Medical Sciences Council, our Council members had their first opportunity to examine summary statements using the new peer review scoring system.
Many aspects of the new scoring system are unfamiliar, including the use of overall impact scores as integers from 10 (best) to 90 (worst). A summary of the new scoring system is well described in a scoring system and procedure document, and an earlier version of this was shared widely with reviewers.
As background, I compiled some data for approximately 300 NIGMS R01 applications reviewed under the new system.
This plot shows the distribution of overall impact scores along with the corresponding percentiles. Note the relative spread of percentile scores at a given impact score. This spread is due to the fact that percentiles are determined independently for each study section that considered 25 or more R01 applications. Otherwise, percentiles are determined across the overall pool of R01 applications reviewed by the Center for Scientific Review.
For comparison, here is a plot of a similar number of NIGMS R01 applications reviewed using the old scoring system.
Note the similar spread of percentiles at a given score due to study section-specific percentiling.
I would like to mention another major change as a result of the NIH Enhancing Peer Review effort. You must use restructured application forms and instructions, including a 12-page length limit for R01s, for applications due on or after January 25, 2010. For details, see the recent NIH Guide notice. We plan to post updates about these changes as key dates approach.
Do you have a figure like Figure 1 at http://www.nigms.nih.gov/Research/pages/trends.aspx, which I believe is easier to interpret. By the way, what’s the expected success rate for the R01 applications in FY2010?
We will post figures similar to the ones that you cite (fraction of grants funded versus percentile score) once the final data for fiscal year 2009 are available, probably sometime in October.
Since the appropriation bills funding NIH for fiscal year 2010 have not yet passed, we do not yet have any estimates for success rates.