Impact Score Paragraph in Summary Statements, Plain Language in Public Sections of Grant Applications

0 comments

Extramural NexusThe August issue of NIH’s Extramural Nexus includes two announcements that might interest you.

Impact Score Paragraph in Summary Statements

Starting with September grant application reviews, reviewers will include a summary paragraph to explain what factors they considered in assigning the overall impact score. This should help investigators better understand the reasons for the score.

Plain Language in Public Sections of Grant Applications

The director’s column talks about the importance of communicating research value in your grant application.

Your grant title, abstract and statement of public health relevance are very important. Once a grant is funded, these items are available to the public through NIH’s RePORTER database. Many people are interested in learning about research supported with taxpayer dollars, so I encourage you to be clear and accurate in writing these parts of your application. Reviewers are being told to expect plain language in these sections.

The Nexus column includes links to these helpful resources:

Scoring Analysis: 1-Year Comparison

12 comments

I recently posted several analyses (on July 15, July 19 and July 21) of the relationships between the overall impact scores on R01 applications determined by study sections and the criterion scores assigned by individual reviewers. These analyses were based on a sample of NIGMS applications reviewed during the October 2009 Council round. This was the first batch of applications for which criterion scores were used.

NIGMS applications for the October 2010 Council round have now been reviewed. Here I present my initial analyses of this data set, which consists of 654 R01 applications that were discussed, scored and percentiled.

The first analysis, shown below, relates to the correlation coefficients between the overall impact score and the averaged individual criterion scores.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Correlation coefficients between the overall impact score and averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Overall, the trend in correlation coefficients is similar to that observed for the sample from 1 year ago, although the correlation coefficients for the current sample are slightly higher for four out of the five criterion scores.

Here are results from a principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

Principal component analysis of overall impact score based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round. The corresponding scores for a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round are shown in parentheses.

There is remarkable agreement between the results of the principal component analysis for the October 2010 data set and those for the October 2009 data set. The first principal component accounts for 72% of the variance, with the largest contribution coming from approach, followed by innovation, significance, investigator and finally environment. This agreement between the data sets extends through all five principal components, although there is somewhat more variation for principal components 2 and 3 than for the others.

Another important factor in making funding decisions is the percentile assigned to a given application. The percentile is a ranking that shows the relative position of each application’s score among all scores assigned by a study section at its last three meetings. Percentiles provide a way to compare applications reviewed by different study sections that may have different scoring behaviors. They also correct for “grade inflation” or “score creep” in the event that study sections assign better scores over time.

Here is a plot of percentiles and overall impact scores:

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

A plot of the overall impact score versus the percentile for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

This plot reveals that a substantial range of overall impact scores can be assigned to a given percentile score. This phenomenon is not new; a comparable level of variation among study sections was seen in the previous scoring system, as well.

The correlation coefficient between the percentile and overall impact score in this data set is 0.93. The correlation coefficients between the percentile and the averaged individual criterion scores are given below:

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Correlation coefficients between the percentile and the averaged individual criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

As one would anticipate, these correlation coefficients are somewhat lower than those for the overall impact score since the percentile takes other factors into account.

The results of a principal component analysis applied to the percentile data show:

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

Principal component analysis of percentile data based on the five criterion scores for 654 NIGMS R01 applications reviewed during the October 2010 Council round.

The results of this analysis are very similar to those for the overall impact scores, with the first principal component accounting for 72% of the variance and similar weights for the individual averaged criterion scores.

Our posting of these scoring analyses has led the NIH Office of Extramural Activities and individual institutes to launch their own analyses. I will share their results as they become available.

The Value of Attending SACNAS and ABRCMS

1 comment

SACNAS National ConferenceThis fall, NIGMS will be sponsoring the Society for Advancement of Chicanos and Native Americans in Science (SACNAS) national conference in Anaheim, CA, September 30-October 3, and the Annual Biomedical Research Conference for Minority Students (ABRCMS) in Charlotte, NC, November 10-13.

ABRCMS Seal

These meetings present two wonderful opportunities for you to meet and recruit science-oriented undergraduate students from groups that are underrepresented in the biomedical sciences. In addition to listening to research talks, you can volunteer to mentor students during the meeting or judge their posters. For those of you who are MORE program directors, you’ll also be able to support your students.

I encourage you to attend one or both of these meetings, as they bring in some of the best and brightest undergraduate students in the nation.

You can visit NIGMS staff at SACNAS exhibit booth 817 or ABRCMS exhibit booth 700.

For additional information or to register as a meeting participant, mentor and/or judge, visit the SACNAS Link to external website and ABRCMS Link to external website Websites.

Even More on Criterion Scores: Full Regression and Principal Component Analyses

4 comments

After reading yesterday’s post, a Feedback Loop reader asked for a full regression analysis of the overall impact score based on all five criterion scores. With the caveat that one should be cautious in over-interpreting such analyses, here it is:

Pearson correlation coefficients of overall impact score and five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. The various parameters are substantially correlated.

Pearson correlation coefficients of overall impact score and five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As one might expect, the various parameters are substantially correlated.

A principal component analysis reveals that a single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Here are more results of the full principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. A single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Principal component analysis of overall impact score based on the five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

The second component accounts for an additional 9% of the variance and has a substantial contribution from approach, with significant contributions of the opposite sign for investigator and environment. The third component accounts for an additional 8% of the variance and appears to be primarily related to innovation. The fourth component accounts for an additional 7% of the variance and is primarily related to significance. The final component accounts for the remaining 5% of the variance and has contributions from investigator and environment of the opposite sign.

More on Criterion Scores

6 comments

In an earlier post, I presented an analysis of the relationship between the average significance criterion scores provided independently by individual reviewers and the overall impact scores determined at the end of the study section discussion for a sample of 360 NIGMS R01 grant applications reviewed during the October 2009 Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey and Medical Writing, Editing & Grantsmanship Link to external web site, I want to provide some additional aspects of this analysis.

As I noted in the recent post, the criterion score most strongly correlated (0.74) with the overall impact score is approach. Here is a plot showing this correlation:

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Similarly, here is a plot comparing the average innovation criterion score and the overall impact score:

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Note that the overall impact score is NOT derived by combining the individual criterion scores. This policy is based on several considerations, including:

  • The effect of the individual criterion scores on the overall impact score is expected to depend on the nature of the project. For example, an application directed toward developing a community resource may not be highly innovative; indeed, a high level of innovation may be undesirable in this context. Nonetheless, such a project may receive a high overall impact score if the approach and significance are strong.
  • The overall impact score is refined over the course of a study section discussion, whereas the individual criterion scores are not.

That being said, it is still possible to derive the average behavior of the study sections involved in reviewing these applications from their scores. The correlation coefficient for the linear combination of individual criterion scores with weighting factors optimized (approximately related to the correlation coefficients between the individual criterion scores and the overall impact factor) is 0.78.

The availability of individual criterion scores provides useful data for analyzing study section behavior. In addition, these criterion scores are important parameters that can assist program staff in making funding recommendations.

Model Organisms and the Significance of Significance

13 comments

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Link to external web site sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

Electronic Awards Administration

1 comment

As with the transition to electronic grant applications, NIH is now transitioning to electronic award administration. Here are some important upcoming changes.

Progress Reports

Beginning August 1, 2010, you must use the eRA Commons eSNAP (Electronic Streamlined Non-competing Award Process) feature to submit all eligible progress reports. Also, SNAP progress reports will be due 45 days prior to the next budget start date (instead of 60 days), and IRB and IACUC approval dates will not be required as part of the progress report.

You can refer to the Notice of Award to determine whether these new requirements apply to your grant. For detailed instructions, see the eSNAP User Guide.

Appointments and Terminations

Beginning January 1, 2011, you must use the xTrain feature in eRA Commons to electronically submit appointment forms and termination notices for research training, fellowship, education and career development awards. After this date, paper documents will not be accepted.

Earlier Submission Deadlines for 2011 Pioneer and New Innovator Awards

0 comments

NIH Director’s Pioneer AwardNIH has announced the 2011 competitions for the NIH Director’s Pioneer Awards and the NIH Director’s New Innovator Awards. These awards support exceptionally creative scientists who propose highly innovative—and often unconventional—approaches to major challenges in biomedical or behavioral research. Both programs are part of the NIH Common Fund and are managed by NIGMS.

The Pioneer Award provides $2.5 million in direct costs over 5 years and is open to scientists at U.S. institutions at any career level. The deadline for applying is September 13, 2010.

NIH Director’s New Innovator Award

The New Innovator Award provides $1.5 million in direct costs over 5 years and is designed for early stage investigators at U.S. institutions who have not yet obtained an NIH R01 or similar grant. Applications are due by September 20, 2010.

For more information about the programs and links to the requests for applications, see the Pioneer Award Web site and the New Innovator Award Web site.

A highlight of these programs is the annual symposium. This year’s symposium will take place near the NIH campus in Bethesda on September 30 and October 1, and it will include research talks by the second graduating class of Pioneer Award recipients. If you’re in the area, consider attending the symposium. It’s free, doesn’t require advance registration and also offers the opportunity to view poster presentations by many of the Pioneers and New Innovators. If you can’t attend, the symposium will be videocast live and archived at http://videocast.nih.gov.

60th Lindau Nobel Laureate Meeting Promises to Be a ‘Dynamite’ Event

1 comment

This year, I’m the lucky NIGMS program director attending the 60th Annual Lindau Nobel Laureate Meeting in Lindau, Germany. Like John Schwab, who traveled with last year’s group, my main job is to help the 16 super-energetic, really smart graduate students we sponsored interact with each other, hundreds of their peers from all over the world and, of course, dozens of Nobel laureates.

60th Lindau Nobel Laureate Meeting

The Lindau meeting, which started in 1951, is designed to “educate, inspire and connect” generations of scientists by bringing together Nobel laureates with young researchers. Unlike last year’s meeting, which focused on chemistry, this year’s is more interdisciplinary. It showcases all three of the Nobel Prize natural science fields: chemistry, physics and physiology or medicine.

As you might imagine, the competition for student slots is fierce, with more than 40,000 applying for 650 slots this year. The U.S. contingent consists of 75 students supported by the Oak Ridge Associated Universities (ORAU), the National Science Foundation, the Department of Energy, Mars (the company, not the planet!) and NIGMS/NIH.

It’s quite appropriate that NIGMS is involved in this program again this year. Our portfolio of funded research is extremely diverse, including a lot in chemistry and physics. After all, an interdisciplinary approach can help us better address problems relevant to human health. And around NIH, NIGMS is also widely called the “Nobel Prize Institute”—we’ve funded the Nobel Prize-winning of research of 73 laureates, and some of them are here in Lindau.

So far, the trip is going pretty well. We’re surrounded by World Cup fever! We haven’t had much of a chance to meet the laureates yet, since the opening ceremony was just on Sunday. I know that much of the close laureate-student interactions will take place through small group discussion sessions set up for this purpose.

I truly feel fortunate to be here—for the interactions, the interdisciplinary science and what I anticipate will be really inspiring lectures by the laureates. But even if you can’t be here, you can still watch the scientific proceedings online and follow the meeting using a variety of social media.

Post submitted from Lindau, Germany, on Monday, Jun 28, 2010 6:25 PM CEST

New Funding Opportunity for Biologists to Collaborate on Structure Studies

0 comments

PSI Logo

We have just issued a new “partnership” program announcement for researchers interested in a biological problem of significant scope to collaborate with structural biology researchers.

The announcement is a part of the PSI:Biology initiative, which will apply high-throughput structure determination to a broad range of biological problems. Successful applicants will partner with researchers within the PSI:Biology network.

Applicants should propose projects for which the determination of many protein structures will be important. The projects should also involve proteins amenable to high-throughput structure determination or targets that motivate new technology development for determining more difficult structures.