Impact Score Paragraph in Summary Statements, Plain Language in Public Sections of Grant Applications

Extramural NexusThe August issue of NIH’s Extramural Nexus includes two announcements that might interest you.

Impact Score Paragraph in Summary Statements

Starting with September grant application reviews, reviewers will include a summary paragraph to explain what factors they considered in assigning the overall impact score. This should help investigators better understand the reasons for the score.

Plain Language in Public Sections of Grant Applications

The director’s column talks about the importance of communicating research value in your grant application.

Your grant title, abstract and statement of public health relevance are very important. Once a grant is funded, these items are available to the public through NIH’s RePORTER database. Many people are interested in learning about research supported with taxpayer dollars, so I encourage you to be clear and accurate in writing these parts of your application. Reviewers are being told to expect plain language in these sections.

The Nexus column includes links to these helpful resources:

Scoring Analysis: 1-Year Comparison

I recently posted several analyses (on July 15, July 19 and July 21) of the relationships between the overall impact scores on R01 applications determined by study sections and the criterion scores assigned by individual reviewers. These analyses were based on a sample of NIGMS applications reviewed during the October 2009 Council round. This was the first batch of applications for which criterion scores were used.

NIGMS applications for the October 2010 Council round have now been reviewed. Here I present my initial analyses of this data set, which consists of 654 R01 applications that were discussed, scored and percentiled.

The first analysis, shown below, relates to the correlation coefficients between the overall impact score and the averaged individual criterion scores.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Overall, the trend in correlation coefficients is similar to that observed for the sample from 1 year ago, although the correlation coefficients for the current sample are slightly higher for four out of the five criterion scores.

Here are results from a principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

There is remarkable agreement between the results of the principal component analysis for the October 2010 data set and those for the October 2009 data set. The first principal component accounts for 72% of the variance, with the largest contribution coming from approach, followed by innovation, significance, investigator and finally environment. This agreement between the data sets extends through all five principal components, although there is somewhat more variation for principal components 2 and 3 than for the others.

Another important factor in making funding decisions is the percentile assigned to a given application. The percentile is a ranking that shows the relative position of each application’s score among all scores assigned by a study section at its last three meetings. Percentiles provide a way to compare applications reviewed by different study sections that may have different scoring behaviors. They also correct for “grade inflation” or “score creep” in the event that study sections assign better scores over time.

Here is a plot of percentiles and overall impact scores:

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

This plot reveals that a substantial range of overall impact scores can be assigned to a given percentile score. This phenomenon is not new; a comparable level of variation among study sections was seen in the previous scoring system, as well.

The correlation coefficient between the percentile and overall impact score in this data set is 0.93. The correlation coefficients between the percentile and the averaged individual criterion scores are given below:

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

As one would anticipate, these correlation coefficients are somewhat lower than those for the overall impact score since the percentile takes other factors into account.

The results of a principal component analysis applied to the percentile data show:

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

The results of this analysis are very similar to those for the overall impact scores, with the first principal component accounting for 72% of the variance and similar weights for the individual averaged criterion scores.

Our posting of these scoring analyses has led the NIH Office of Extramural Activities and individual institutes to launch their own analyses. I will share their results as they become available.

Even More on Criterion Scores: Full Regression and Principal Component Analyses

After reading yesterday’s post, a Feedback Loop reader asked for a full regression analysis of the overall impact score based on all five criterion scores. With the caveat that one should be cautious in over-interpreting such analyses, here it is:

Pearson correlation coefficients of overall impact score and five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. The various parameters are substantially correlated.

Pearson correlation coefficients of overall impact score and five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As one might expect, the various parameters are substantially correlated.

A principal component analysis reveals that a single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Here are more results of the full principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. A single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Principal component analysis of overall impact score based on the five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

The second component accounts for an additional 9% of the variance and has a substantial contribution from approach, with significant contributions of the opposite sign for investigator and environment. The third component accounts for an additional 8% of the variance and appears to be primarily related to innovation. The fourth component accounts for an additional 7% of the variance and is primarily related to significance. The final component accounts for the remaining 5% of the variance and has contributions from investigator and environment of the opposite sign.

More on Criterion Scores

In an earlier post, I presented an analysis of the relationship between the average significance criterion scores provided independently by individual reviewers and the overall impact scores determined at the end of the study section discussion for a sample of 360 NIGMS R01 grant applications reviewed during the October 2009 Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey Exit icon and Medical Writing, Editing & Grantsmanship Exit icon, I want to provide some additional aspects of this analysis.

As I noted in the recent post, the criterion score most strongly correlated (0.74) with the overall impact score is approach. Here is a plot showing this correlation:

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Similarly, here is a plot comparing the average innovation criterion score and the overall impact score:

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Note that the overall impact score is NOT derived by combining the individual criterion scores. This policy is based on several considerations, including:

  • The effect of the individual criterion scores on the overall impact score is expected to depend on the nature of the project. For example, an application directed toward developing a community resource may not be highly innovative; indeed, a high level of innovation may be undesirable in this context. Nonetheless, such a project may receive a high overall impact score if the approach and significance are strong.
  • The overall impact score is refined over the course of a study section discussion, whereas the individual criterion scores are not.

That being said, it is still possible to derive the average behavior of the study sections involved in reviewing these applications from their scores. The correlation coefficient for the linear combination of individual criterion scores with weighting factors optimized (approximately related to the correlation coefficients between the individual criterion scores and the overall impact factor) is 0.78.

The availability of individual criterion scores provides useful data for analyzing study section behavior. In addition, these criterion scores are important parameters that can assist program staff in making funding recommendations.

Model Organisms and the Significance of Significance

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Exit icon sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

Electronic Application Correction Period Temporarily Extended

To accommodate the transition to new application forms and instructions, NIH has temporarily extended the electronic application error correction window to five business days for applications due between January 25 and May 7, 2010. This allows additional time for applicants who may have inadvertently used the wrong forms to correct their applications. Please remember that applications using the wrong forms or that exceed the new page limits will not be reviewed.

For more on the application changes, see my October 13 post.

Major Application Changes Come in January

Two major recommendations of the NIH Enhancing Peer Review Initiative were to shorten grant applications and restructure their content. These changes will affect applications due on or after January 25, 2010.

Here’s a brief overview of the changes and their implementation. Be sure to follow the links for other details and important information.

New Application Structure and Length

These changes affect ALL applications (new, renewal, resubmission and revision). Exceptions will be considered only for AIDS applications from members of review committees. Specifics vary with the type of application (research, training, resource, etc.). For more, see:

Implementation

  • When submitting an application due on or after January 25, you must download the new application forms. You may sign up to be notified when new application packages become available, which will be in December.
  • Applications submitted early must follow the instructions for the actual due date (e.g., applications submitted on January 24 for the February 5 R01 due date must use the new forms).
  • You can begin working on your applications now and paste the text into the appropriate form when it’s available.
  • NIH will not accept any applications using any part of the old forms, including biosketches.
  • All existing Funding Opportunity Announcements (both electronic and paper) will be revised to incorporate these changes and will be reissued by December 2009.
  • Parent announcements will be reissued and have new Funding Opportunity Announcement numbers.

If you have specific questions, please contact the NIH Grants Information Help Desk at grantsinfo@nih.gov.

The New Scoring System

At the recent meeting of the National Advisory General Medical Sciences Council, our Council members had their first opportunity to examine summary statements using the new peer review scoring system.

Many aspects of the new scoring system are unfamiliar, including the use of overall impact scores as integers from 10 (best) to 90 (worst). A summary of the new scoring system is well described in a scoring system and procedure document, and an earlier version of this was shared widely with reviewers.

As background, I compiled some data for approximately 300 NIGMS R01 applications reviewed under the new system.

This plot shows the distribution of overall impact scores along with the corresponding percentiles.

This plot shows the distribution of overall impact scores along with the corresponding percentiles. Note the relative spread of percentile scores at a given impact score. This spread is due to the fact that percentiles are determined independently for each study section that considered 25 or more R01 applications. Otherwise, percentiles are determined across the overall pool of R01 applications reviewed by the Center for Scientific Review.

For comparison, here is a plot of a similar number of NIGMS R01 applications reviewed using the old scoring system.

A plot of a similar number of NIGMS R01 applications reviewed using the old scoring system.

Note the similar spread of percentiles at a given score due to study section-specific percentiling.

I would like to mention another major change as a result of the NIH Enhancing Peer Review effort. You must use restructured application forms and instructions, including a 12-page length limit for R01s, for applications due on or after January 25, 2010. For details, see the recent NIH Guide notice. We plan to post updates about these changes as key dates approach.

Changes to Peer Review—the Reviewer’s Perspective

NIH has posted a video on the Enhancing Peer Review Web site that details recent changes to the peer review system from the reviewer’s perspective. Running about 13 minutes, “What Reviewers Need to Know Now”, (Transcript [PDF, 79KB]) offers valuable information for current and prospective peer reviewers. You’ll find other resources in the PowerPoint presentation and related links that accompany the video.

Dr. Alan Willard discusses What Reviewers Need to Know Now - March 19, 2009

Many aspects of the new peer review system are already being used in study section meetings. If you have participated in any of these meetings, I am very interested in your comments and reactions.

I’d also like to mention that NIH’s Center for Scientific Review, NIGMS and other institutes and centers have a critical need right now for reviewers to help evaluate the tremendous number of applications submitted in response to Recovery Act funding opportunities. If a Scientific Review Officer asks you to participate on a review panel, I hope that you’ll agree to serve if at all possible.