NIH-Wide Correlations Between Overall Impact Scores and Criterion Scores

In a recent post, I presented correlations between the overall impact scores and the five individual criterion scores for sample sets of NIGMS applications. I also noted that the NIH Office of Extramural Research (OER) was performing similar analyses for applications across NIH.

OER’s Division of Information Services has now analyzed 32,608 applications (including research project grant, research center and SBIR/STTR applications) that were discussed and received overall impact scores during the October, January and May Council rounds in Fiscal Year 2010. Here are the results by institute and center:

Correlation coefficients between  the overall impact score and the five criterion scores for 32,608 NIH applications  from the Fiscal Year 2010 October, January and May Council rounds.

Correlation coefficients between the overall impact score and the five criterion scores for 32,608 NIH applications from the Fiscal Year 2010 October, January and May Council rounds. High-res. image (112KB JPG)

This analysis reveals the same trends in correlation coefficients observed in smaller data sets of NIGMS R01 grant applications. Furthermore, no significant differences were observed in the correlation coefficients among the 24 NIH institutes and centers with funding authority.

Measuring the Scientific Output and Impact of NIGMS Grants

A frequent topic of discussion at our Advisory Council meetings—and across NIH—is how to measure scientific output in ways that effectively capture scientific impact. We have been working on such issues with staff of the Division of Information Services in the NIH Office of Extramural Research. As a result of their efforts, as well as those of several individual institutes, we now have tools that link publications to the grants that funded them.

Using these tools, we have compiled three types of data on the pool of investigators who held at least one NIGMS grant in Fiscal Year 2006. We determined each investigator’s total NIH R01 or P01 funding for that year. We also calculated the total number of publications linked to these grants from 2007 to mid-2010 and the average impact factor for the journals in which these papers appeared. We used impact factors in place of citations because the time dependence of citations makes them significantly more complicated to use.

I presented some of the results of our analysis of this data at last week’s Advisory Council meeting. Here are the distributions for the three parameters for the 2,938 investigators in the sample set:

Histograms  showing the distributions of total annual direct costs, number of publications linked  to those grants from 2007 to mid-2010 and average impact factor for the  publication journals for 2,938 investigators who held at least one NIGMS R01 or  P01 grant in Fiscal Year 2006.

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

For this population, the median annual total direct cost was $220,000, the median number of grant-linked publications was six and the median journal average impact factor was 5.5.

A plot of the median number of grant-linked publications and median journal average impact factors versus grant total annual direct costs is shown below.

A plot  of the median number of grant-linked publications from 2007 to mid-2010 (red  circles) and median average impact factor for journals in which these papers  were published (blue squares) for 2,938 investigators who held at least one  NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the  interquartile ranges for the number of grant-linked publications (longer red  bars) and journal average impact factors (shorter blue bars). The medians are  for bins, with the number of investigators in each bin shown below the bars.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

This plot reveals several important points. The ranges in the number of publications and average impact factors within each total annual direct cost bin are quite large. This partly reflects variations in investigator productivity as measured by these parameters, but it also reflects variations in publication patterns among fields and other factors.

Nonetheless, clear trends are evident in the averages for the binned groups, with both parameters increasing with total annual direct costs until they peak at around $700,000. These observations provide support for our previously developed policy on the support of research in well-funded laboratories. This policy helps us use Institute resources as optimally as possible in supporting the overall biomedical research enterprise.

This is a preliminary analysis, and the results should be viewed with some skepticism given the metrics used, the challenges of capturing publications associated with particular grants, the lack of inclusion of funding from non-NIH sources and other considerations. Even with these caveats, the analysis does provide some insight into the NIGMS grant portfolio and indicates some of the questions that can be addressed with the new tools that NIH is developing.

Collaborative Science Supplement Requests Due in January

Do you have an idea for a great collaboration that will advance your NIGMS-funded research project? If your current award has active funding through at least July 31, 2012, you may be eligible to jump-start your idea with an administrative supplement for collaborative science. The next submission deadline is January 15, 2011.

To be sure that your project is appropriate for this program, please review the funding opportunity announcement. You should also discuss the project idea with your NIGMS program director before preparing an application. For general questions about the program, contact me at hayness@nigms.nih.gov or Marion Zatz at zatzm@nigms.nih.gov.

New Funding Opportunity for Stem Cell Researchers

NIGMS has just issued a call for Program Projects for Collaborative Research on the Basic Biology of Pluripotency and Reprogramming (P01), with an emphasis on human induced pluripotent stem (iPS) cells. We are particularly interested in studies that propose comprehensive analyses of the basic biology of pluripotency, the molecular events and mechanisms of reprogramming, and the epigenetics and epigenomics of the pluripotent and reprogrammed states.

These applications have special requirements, so please read the announcement carefully. Letters of intent are due on November 1, and applications are due on December 1.

If your research involves stem cells but isn’t appropriate for this announcement, you may submit an investigator-initiated R01 application that addresses the basic biology of stem cells and/or uses these cells as model systems to study fundamental life processes.

You may contact me at hayness@nigms.nih.gov or Marion Zatz at zatzm@nigms.nih.gov with questions about this new opportunity or about NIGMS support for stem cell research.

Scoring Analysis with Funding and Investigator Status

My previous post generated interest in seeing the results coded to identify new investigators and early stage investigators. Recall that new investigators are defined as individuals who have not previously competed successfully as program director/principal investigator for a substantial NIH independent research award. Early stage investigators are defined as new investigators who are within 10 years of completing the terminal research degree or medical residency (or the equivalent).

Below is a plot for 655 NIGMS R01 applications reviewed during the January 2010 Council round.

A plot of the overall impact score versus the  percentile for 655 NIGMS R01 applications reviewed during the January 2010  Council round. Solid symbols show applications for which awards have been made  and open symbols show applications for which awards have not been made. Red  circles indicate early stage investigators, blue squares indicate new  investigators who are not early stage investigators and black diamonds indicate  established investigators.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Solid symbols show applications for which awards have been made and open symbols show applications for which awards have not been made. Red circles indicate early stage investigators, blue squares indicate new investigators who are not early stage investigators and black diamonds indicate established investigators.

This plot reveals that many of the awards made for applications with less favorable percentile scores go to early stage and new investigators. This is consistent with recent NIH policies.

The plot also partially reveals the distribution of applications from different classes of applicants. This distribution is more readily seen in the plot below.

A plot of the cumulative  fraction of applications for four classes of applications with a pool of 655 NIGMS R01 applications reviewed during the January 2010 Council  round. The classes are applications from early stage investigators (red  squares), applications from new investigators (blue circles), new (Type 1) applications  from established investigators (black diamonds) and competing renewal (Type 2)  applications from established investigators (black triangles). N indicates the number in  each class of applications within the pool.

A plot of the cumulative fraction of applications for four classes of applications with a pool of 655 NIGMS R01 applications reviewed during the January 2010 Council round. The classes are applications from early stage investigators (red squares), applications from new investigators (blue circles), new (Type 1) applications from established investigators (black diamonds) and competing renewal (Type 2) applications from established investigators (black triangles). N indicates the number in each class of applications within the pool.

This plot shows that competing renewal (Type 2) applications from established investigators represent the largest class in the pool and receive more favorable percentile scores than do applications from other classes of investigators. The plot also shows that applications from early stage investigators have a score distribution that is quite similar to that for established investigators submitting new applications. The curve for new investigators who are not early stage investigators is similar as well, although the new investigator curve is shifted somewhat toward less favorable percentile scores.

Apply Now for Microbial Community Dynamics Grants

NIGMS has just re-announced the Dynamics of Host-Associated Microbial Communities (R01) funding opportunity. Microbes make up the vast majority of our bodies’ cells, and this program supports projects that aim to dissect these complex communities and their roles within a host.

We are particularly interested in applications that propose:

  • Genetic, physiological and ecological research on mixed microbial communities, their internal dynamics and how they relate to those of the host; and
  • Studies on other experimental models that could make breakthrough contributions to understanding the formation and dynamics of host-microbe symbiotic systems.

We encourage interdisciplinary approaches, including bioinformatics/computational/modeling and/or experimental manipulations to investigate host-associated microbial community ecology.

You may apply for up to $250,000 (direct costs) per year (plus up to $100,000 for exceptional equipment in the first year). Most awards will be for 4 years. Letters of intent are due on December 14, 2010, and applications are due on January 14, 2011.

For more details about the program, see the funding opportunity announcement or contact me at 301-594-3900 or singhs@nigms.nih.gov.

Update: Status of hESC Applications and Grants

NIH has rescinded the earlier notice regarding the status of applications and grants involving human embryonic stem cells. The new notice states that the receipt, processing, review and awarding of NIH applications and proposals involving human embryonic stem cells will continue. It goes on to list the following actions:

  • The suspension of further NIH activity to implement, apply or act pursuant to the NIH Guidelines is hereby lifted.
  • The suspension of the issuance of all pending competing, and noncompeting continuation hESC awards and contracts approved for funding is hereby lifted.
  • The suspension of the peer review of all pending competing hESC applications and proposals is hereby lifted.
  • The NIH Human Embryonic Stem Cell Registry will resume accepting submissions of information about hESC lines for the purpose of establishing eligibility for funding under the NIH Guidelines. The NIH review of hESC lines for inclusion on the Registry under the NIH Guidelines will also resume.

Modeling Scientific Workforce Dynamics

In keeping with the Institute’s long-standing interest in training and its strong commitment to fostering a diverse scientific workforce, we have just re-announced our Modeling the Scientific Workforce (U01) program.

This program provides support for developing computational models of the scientific workforce in the United States. It takes a systems-based approach to understanding the underlying dynamics that produce successful scientists, examining strategies for increasing the diversity of the scientific workforce, identifying research questions and guiding data collection and analysis. The models will help inform our program development, management and evaluation.

We are particularly interested in models of the academic scientific workforce, but applicants should also consider industry and the government. We strongly encourage collaboration among scientists who are experts in simulation modeling, large-scale educational data sets, national policy and program development and other appropriate areas.

Letters of intent are due on October 4, 2010, and applications are due on November 4, 2010.

For additional information about the program, see the funding opportunity announcement or contact me at 301-594-3900 or singhs@nigms.nih.gov.

Status of hESC Applications and Grants

NIH has issued a notice describing the status of applications and grants that propose research using human embryonic stem cells (hESC). Among its points are:

  • Any further NIH activity to implement, apply or act pursuant to the NIH Guidelines is hereby suspended until further notice.
  • Issuance of all pending competing, and noncompeting continuation hESC awards and contracts is suspended until further notice.
  • The peer review of all pending competing hESC applications and proposals is suspended until further notice.

Grants affected include all types of research and training. We expect more guidance soon and will let you know when it’s posted.

Scoring Analysis with Funding Status

In response to a previous post, a reader requested a plot showing impact score versus percentile for applications for which funding decisions have been made. Below is a plot for 655 NIGMS R01 applications reviewed during the January 2010 Council round.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Green circles show applications for which awards have been made. Black squares show applications for which awards have not been made.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Green circles show applications for which awards have been made. Black squares show applications for which awards have not been made.

This plot confirms that the percentile representing the halfway point of the funding curve is slightly above the 20th percentile, as expected from previously posted data.

Notice that there is a small number of applications with percentile scores better than the 20th percentile for which awards have not been made. Most of these correspond to new (Type 1, not competing renewal) applications that are subject to the NIGMS Council’s funding decision guidelines for well-funded laboratories.