Category: Peer Review

Posted on

Scoring Analysis with Funding and Investigator Status

by
17 comments

My previous post generated interest in seeing the results coded to identify new investigators and early stage investigators. Recall that new investigators are defined as individuals who have not previously competed successfully as program director/principal investigator for a substantial NIH independent research award. Early stage investigators are defined as new investigators who are within 10 years of completing the terminal research degree or medical residency (or the equivalent).

Below is a plot for 655 NIGMS R01 applications reviewed during the January 2010 Council round.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Solid symbols show applications for which awards have been made and open symbols show applications for which awards have not been made. Red circles indicate early stage investigators, blue squares indicate new investigators who are not early stage investigators and black diamonds indicate established investigators.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Solid symbols show applications for which awards have been made and open symbols show applications for which awards have not been made. Red circles indicate early stage investigators, blue squares indicate new investigators who are not early stage investigators and black diamonds indicate established investigators.

This plot reveals that many of the awards made for applications with less favorable percentile scores go to early stage and new investigators. This is consistent with recent NIH policies.

The plot also partially reveals the distribution of applications from different classes of applicants. This distribution is more readily seen in the plot below.

A plot of the cumulative fraction of applications for four classes of applications with a pool of 655 NIGMS R01 applications reviewed during the January 2010 Council round. The classes are applications from early stage investigators (red squares), applications from new investigators (blue circles), new (Type 1) applications from established investigators (black diamonds) and competing renewal (Type 2) applications from established investigators (black triangles). N indicates the number in each class of applications within the pool.

A plot of the cumulative fraction of applications for four classes of applications with a pool of 655 NIGMS R01 applications reviewed during the January 2010 Council round. The classes are applications from early stage investigators (red squares), applications from new investigators (blue circles), new (Type 1) applications from established investigators (black diamonds) and competing renewal (Type 2) applications from established investigators (black triangles). N indicates the number in each class of applications within the pool.

This plot shows that competing renewal (Type 2) applications from established investigators represent the largest class in the pool and receive more favorable percentile scores than do applications from other classes of investigators. The plot also shows that applications from early stage investigators have a score distribution that is quite similar to that for established investigators submitting new applications. The curve for new investigators who are not early stage investigators is similar as well, although the new investigator curve is shifted somewhat toward less favorable percentile scores.

Posted on

Scoring Analysis with Funding Status

by
15 comments

In response to a previous post, a reader requested a plot showing impact score versus percentile for applications for which funding decisions have been made. Below is a plot for 655 NIGMS R01 applications reviewed during the January 2010 Council round.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Green circles show applications for which awards have been made. Black squares show applications for which awards have not been made.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Green circles show applications for which awards have been made. Black squares show applications for which awards have not been made.

This plot confirms that the percentile representing the halfway point of the funding curve is slightly above the 20th percentile, as expected from previously posted data.

Notice that there is a small number of applications with percentile scores better than the 20th percentile for which awards have not been made. Most of these correspond to new (Type 1, not competing renewal) applications that are subject to the NIGMS Council’s funding decision guidelines for well-funded laboratories.

Posted on

Impact Score Paragraph in Summary Statements, Plain Language in Public Sections of Grant Applications

by and
0 comments

Extramural NexusThe August issue of NIH’s Extramural Nexus includes two announcements that might interest you.

Impact Score Paragraph in Summary Statements

Starting with September grant application reviews, reviewers will include a summary paragraph to explain what factors they considered in assigning the overall impact score. This should help investigators better understand the reasons for the score.

Plain Language in Public Sections of Grant Applications

The director’s column talks about the importance of communicating research value in your grant application.

Your grant title, abstract and statement of public health relevance are very important. Once a grant is funded, these items are available to the public through NIH’s RePORTER database. Many people are interested in learning about research supported with taxpayer dollars, so I encourage you to be clear and accurate in writing these parts of your application. Reviewers are being told to expect plain language in these sections.

The Nexus column includes links to these helpful resources:

Posted on

Scoring Analysis: 1-Year Comparison

by
12 comments

I recently posted several analyses (on July 15, July 19 and July 21) of the relationships between the overall impact scores on R01 applications determined by study sections and the criterion scores assigned by individual reviewers. These analyses were based on a sample of NIGMS applications reviewed during the October 2009 Council round. This was the first batch of applications for which criterion scores were used.

NIGMS applications for the October 2010 Council round have now been reviewed. Here I present my initial analyses of this data set, which consists of 654 R01 applications that were discussed, scored and percentiled.

The first analysis, shown below, relates to the correlation coefficients between the overall impact score and the averaged individual criterion scores.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Overall, the trend in correlation coefficients is similar to that observed for the sample from 1 year ago, although the correlation coefficients for the current sample are slightly higher for four out of the five criterion scores.

Here are results from a principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

There is remarkable agreement between the results of the principal component analysis for the October 2010 data set and those for the October 2009 data set. The first principal component accounts for 72% of the variance, with the largest contribution coming from approach, followed by innovation, significance, investigator and finally environment. This agreement between the data sets extends through all five principal components, although there is somewhat more variation for principal components 2 and 3 than for the others.

Another important factor in making funding decisions is the percentile assigned to a given application. The percentile is a ranking that shows the relative position of each application’s score among all scores assigned by a study section at its last three meetings. Percentiles provide a way to compare applications reviewed by different study sections that may have different scoring behaviors. They also correct for “grade inflation” or “score creep” in the event that study sections assign better scores over time.

Here is a plot of percentiles and overall impact scores:

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

This plot reveals that a substantial range of overall impact scores can be assigned to a given percentile score. This phenomenon is not new; a comparable level of variation among study sections was seen in the previous scoring system, as well.

The correlation coefficient between the percentile and overall impact score in this data set is 0.93. The correlation coefficients between the percentile and the averaged individual criterion scores are given below:

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

As one would anticipate, these correlation coefficients are somewhat lower than those for the overall impact score since the percentile takes other factors into account.

The results of a principal component analysis applied to the percentile data show:

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

The results of this analysis are very similar to those for the overall impact scores, with the first principal component accounting for 72% of the variance and similar weights for the individual averaged criterion scores.

Our posting of these scoring analyses has led the NIH Office of Extramural Activities and individual institutes to launch their own analyses. I will share their results as they become available.

Posted on

Even More on Criterion Scores: Full Regression and Principal Component Analyses

by
4 comments

After reading yesterday’s post, a Feedback Loop reader asked for a full regression analysis of the overall impact score based on all five criterion scores. With the caveat that one should be cautious in over-interpreting such analyses, here it is:

Pearson correlation coefficients of overall impact score and five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. The various parameters are substantially correlated.

Pearson correlation coefficients of overall impact score and five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As one might expect, the various parameters are substantially correlated.

A principal component analysis reveals that a single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Here are more results of the full principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. A single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Principal component analysis of overall impact score based on the five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

The second component accounts for an additional 9% of the variance and has a substantial contribution from approach, with significant contributions of the opposite sign for investigator and environment. The third component accounts for an additional 8% of the variance and appears to be primarily related to innovation. The fourth component accounts for an additional 7% of the variance and is primarily related to significance. The final component accounts for the remaining 5% of the variance and has contributions from investigator and environment of the opposite sign.

Posted on

More on Criterion Scores

by
6 comments

In an earlier post, I presented an analysis of the relationship between the average significance criterion scores provided independently by individual reviewers and the overall impact scores determined at the end of the study section discussion for a sample of 360 NIGMS R01 grant applications reviewed during the October 2009 Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey and Medical Writing, Editing & Grantsmanship Link to external web site, I want to provide some additional aspects of this analysis.

As I noted in the recent post, the criterion score most strongly correlated (0.74) with the overall impact score is approach. Here is a plot showing this correlation:

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Similarly, here is a plot comparing the average innovation criterion score and the overall impact score:

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Note that the overall impact score is NOT derived by combining the individual criterion scores. This policy is based on several considerations, including:

  • The effect of the individual criterion scores on the overall impact score is expected to depend on the nature of the project. For example, an application directed toward developing a community resource may not be highly innovative; indeed, a high level of innovation may be undesirable in this context. Nonetheless, such a project may receive a high overall impact score if the approach and significance are strong.
  • The overall impact score is refined over the course of a study section discussion, whereas the individual criterion scores are not.

That being said, it is still possible to derive the average behavior of the study sections involved in reviewing these applications from their scores. The correlation coefficient for the linear combination of individual criterion scores with weighting factors optimized (approximately related to the correlation coefficients between the individual criterion scores and the overall impact factor) is 0.78.

The availability of individual criterion scores provides useful data for analyzing study section behavior. In addition, these criterion scores are important parameters that can assist program staff in making funding recommendations.

Posted on

Model Organisms and the Significance of Significance

by
13 comments

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Link to external web site sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

Posted on

Electronic Application Correction Period Temporarily Extended

by
0 comments

To accommodate the transition to new application forms and instructions, NIH has temporarily extended the electronic application error correction window to five business days for applications due between January 25 and May 7, 2010. This allows additional time for applicants who may have inadvertently used the wrong forms to correct their applications. Please remember that applications using the wrong forms or that exceed the new page limits will not be reviewed.

For more on the application changes, see my October 13 post.

Posted on

Major Application Changes Come in January

by and
1 comment

Two major recommendations of the NIH Enhancing Peer Review Initiative were to shorten grant applications and restructure their content. These changes will affect applications due on or after January 25, 2010.

Here’s a brief overview of the changes and their implementation. Be sure to follow the links for other details and important information.

New Application Structure and Length

These changes affect ALL applications (new, renewal, resubmission and revision). Exceptions will be considered only for AIDS applications from members of review committees. Specifics vary with the type of application (research, training, resource, etc.). For more, see:

  • NIH Guide notice
  • Details of application changes
  • Table of page limits
  • Links to more information for applicants and reviewers

Implementation

  • When submitting an application due on or after January 25, you must download the new application forms. You may sign up to be notified when new application packages become available, which will be in December.
  • Applications submitted early must follow the instructions for the actual due date (e.g., applications submitted on January 24 for the February 5 R01 due date must use the new forms).
  • You can begin working on your applications now and paste the text into the appropriate form when it’s available.
  • NIH will not accept any applications using any part of the old forms, including biosketches.
  • All existing Funding Opportunity Announcements (both electronic and paper) will be revised to incorporate these changes and will be reissued by December 2009.
  • Parent announcements will be reissued and have new Funding Opportunity Announcement numbers.

If you have specific questions, please contact the NIH Grants Information Help Desk at grantsinfo@nih.gov.

Posted on

The New Scoring System

by
2 comments

At the recent meeting of the National Advisory General Medical Sciences Council, our Council members had their first opportunity to examine summary statements using the new peer review scoring system.

Many aspects of the new scoring system are unfamiliar, including the use of overall impact scores as integers from 10 (best) to 90 (worst). A summary of the new scoring system is well described in a scoring system and procedure document, and an earlier version of this was shared widely with reviewers.

As background, I compiled some data for approximately 300 NIGMS R01 applications reviewed under the new system.

This plot shows the distribution of overall impact scores along with the corresponding percentiles.

This plot shows the distribution of overall impact scores along with the corresponding percentiles. Note the relative spread of percentile scores at a given impact score. This spread is due to the fact that percentiles are determined independently for each study section that considered 25 or more R01 applications. Otherwise, percentiles are determined across the overall pool of R01 applications reviewed by the Center for Scientific Review.

For comparison, here is a plot of a similar number of NIGMS R01 applications reviewed using the old scoring system.

A plot of a similar number of NIGMS R01 applications reviewed using the old scoring system.

Note the similar spread of percentiles at a given score due to study section-specific percentiling.

I would like to mention another major change as a result of the NIH Enhancing Peer Review effort. You must use restructured application forms and instructions, including a 12-page length limit for R01s, for applications due on or after January 25, 2010. For details, see the recent NIH Guide notice. We plan to post updates about these changes as key dates approach.