The Value of Attending SACNAS and ABRCMS

SACNAS National ConferenceThis fall, NIGMS will be sponsoring the Society for Advancement of Chicanos and Native Americans in Science (SACNAS) national conference in Anaheim, CA, September 30-October 3, and the Annual Biomedical Research Conference for Minority Students (ABRCMS) in Charlotte, NC, November 10-13.

ABRCMS Seal

These meetings present two wonderful opportunities for you to meet and recruit science-oriented undergraduate students from groups that are underrepresented in the biomedical sciences. In addition to listening to research talks, you can volunteer to mentor students during the meeting or judge their posters. For those of you who are MORE program directors, you’ll also be able to support your students.

I encourage you to attend one or both of these meetings, as they bring in some of the best and brightest undergraduate students in the nation.

You can visit NIGMS staff at SACNAS exhibit booth 817 or ABRCMS exhibit booth 700.

For additional information or to register as a meeting participant, mentor and/or judge, visit the SACNAS Exit icon and ABRCMS Exit icon Web sites.

Even More on Criterion Scores: Full Regression and Principal Component Analyses

After reading yesterday’s post, a Feedback Loop reader asked for a full regression analysis of the overall impact score based on all five criterion scores. With the caveat that one should be cautious in over-interpreting such analyses, here it is:

Pearson correlation coefficients of overall impact score and five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. The various parameters are substantially correlated.

Pearson correlation coefficients of overall impact score and five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As one might expect, the various parameters are substantially correlated.

A principal component analysis reveals that a single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Here are more results of the full principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. A single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Principal component analysis of overall impact score based on the five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

The second component accounts for an additional 9% of the variance and has a substantial contribution from approach, with significant contributions of the opposite sign for investigator and environment. The third component accounts for an additional 8% of the variance and appears to be primarily related to innovation. The fourth component accounts for an additional 7% of the variance and is primarily related to significance. The final component accounts for the remaining 5% of the variance and has contributions from investigator and environment of the opposite sign.

More on Criterion Scores

In an earlier post, I presented an analysis of the relationship between the average significance criterion scores provided independently by individual reviewers and the overall impact scores determined at the end of the study section discussion for a sample of 360 NIGMS R01 grant applications reviewed during the October 2009 Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey Exit icon and Medical Writing, Editing & Grantsmanship Exit icon, I want to provide some additional aspects of this analysis.

As I noted in the recent post, the criterion score most strongly correlated (0.74) with the overall impact score is approach. Here is a plot showing this correlation:

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Similarly, here is a plot comparing the average innovation criterion score and the overall impact score:

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Note that the overall impact score is NOT derived by combining the individual criterion scores. This policy is based on several considerations, including:

  • The effect of the individual criterion scores on the overall impact score is expected to depend on the nature of the project. For example, an application directed toward developing a community resource may not be highly innovative; indeed, a high level of innovation may be undesirable in this context. Nonetheless, such a project may receive a high overall impact score if the approach and significance are strong.
  • The overall impact score is refined over the course of a study section discussion, whereas the individual criterion scores are not.

That being said, it is still possible to derive the average behavior of the study sections involved in reviewing these applications from their scores. The correlation coefficient for the linear combination of individual criterion scores with weighting factors optimized (approximately related to the correlation coefficients between the individual criterion scores and the overall impact factor) is 0.78.

The availability of individual criterion scores provides useful data for analyzing study section behavior. In addition, these criterion scores are important parameters that can assist program staff in making funding recommendations.

Model Organisms and the Significance of Significance

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Exit icon sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

Electronic Awards Administration

As with the transition to electronic grant applications, NIH is now transitioning to electronic award administration. Here are some important upcoming changes.

Progress Reports

Beginning August 1, 2010, you must use the eRA Commons eSNAP (Electronic Streamlined Non-competing Award Process) feature to submit all eligible progress reports. Also, SNAP progress reports will be due 45 days prior to the next budget start date (instead of 60 days), and IRB and IACUC approval dates will not be required as part of the progress report.

You can refer to the Notice of Award to determine whether these new requirements apply to your grant. For detailed instructions, see the eSNAP User Guide.

Appointments and Terminations

Beginning January 1, 2011, you must use the xTrain feature in eRA Commons to electronically submit appointment forms and termination notices for research training, fellowship, education and career development awards. After this date, paper documents will not be accepted.

Earlier Submission Deadlines for 2011 Pioneer and New Innovator Awards

NIH Director’s Pioneer AwardNIH has announced the 2011 competitions for the NIH Director’s Pioneer Awards and the NIH Director’s New Innovator Awards. These awards support exceptionally creative scientists who propose highly innovative—and often unconventional—approaches to major challenges in biomedical or behavioral research. Both programs are part of the NIH Common Fund and are managed by NIGMS.

The Pioneer Award provides $2.5 million in direct costs over 5 years and is open to scientists at U.S. institutions at any career level. The deadline for applying is September 13, 2010.

NIH Director’s New Innovator Award

The New Innovator Award provides $1.5 million in direct costs over 5 years and is designed for early stage investigators at U.S. institutions who have not yet obtained an NIH R01 or similar grant. Applications are due by September 20, 2010.

For more information about the programs and links to the requests for applications, see the Pioneer Award Web site and the New Innovator Award Web site.

A highlight of these programs is the annual symposium. This year’s symposium will take place near the NIH campus in Bethesda on September 30 and October 1, and it will include research talks by the second graduating class of Pioneer Award recipients. If you’re in the area, consider attending the symposium. It’s free, doesn’t require advance registration and also offers the opportunity to view poster presentations by many of the Pioneers and New Innovators. If you can’t attend, the symposium will be videocast live and archived at http://videocast.nih.gov.