Category: Funding Trends

Measuring the Scientific Output and Impact of NIGMS Grants

29 comments

A frequent topic of discussion at our Advisory Council meetings—and across NIH—is how to measure scientific output in ways that effectively capture scientific impact. We have been working on such issues with staff of the Division of Information Services in the NIH Office of Extramural Research. As a result of their efforts, as well as those of several individual institutes, we now have tools that link publications to the grants that funded them.

Using these tools, we have compiled three types of data on the pool of investigators who held at least one NIGMS grant in Fiscal Year 2006. We determined each investigator’s total NIH R01 or P01 funding for that year. We also calculated the total number of publications linked to these grants from 2007 to mid-2010 and the average impact factor for the journals in which these papers appeared. We used impact factors in place of citations because the time dependence of citations makes them significantly more complicated to use.

I presented some of the results of our analysis of this data at last week’s Advisory Council meeting. Here are the distributions for the three parameters for the 2,938 investigators in the sample set:

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

For this population, the median annual total direct cost was $220,000, the median number of grant-linked publications was six and the median journal average impact factor was 5.5.

A plot of the median number of grant-linked publications and median journal average impact factors versus grant total annual direct costs is shown below.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

This plot reveals several important points. The ranges in the number of publications and average impact factors within each total annual direct cost bin are quite large. This partly reflects variations in investigator productivity as measured by these parameters, but it also reflects variations in publication patterns among fields and other factors.

Nonetheless, clear trends are evident in the averages for the binned groups, with both parameters increasing with total annual direct costs until they peak at around $700,000. These observations provide support for our previously developed policy on the support of research in well-funded laboratories. This policy helps us use Institute resources as optimally as possible in supporting the overall biomedical research enterprise.

This is a preliminary analysis, and the results should be viewed with some skepticism given the metrics used, the challenges of capturing publications associated with particular grants, the lack of inclusion of funding from non-NIH sources and other considerations. Even with these caveats, the analysis does provide some insight into the NIGMS grant portfolio and indicates some of the questions that can be addressed with the new tools that NIH is developing.

Fiscal Year 2009 R01 Funding Outcomes

0 comments

Fiscal Year 2009, which ended on September 30, was a time of unprecedented opportunities for NIH due to the American Recovery and Reinvestment Act. As I noted previously, NIGMS used Recovery Act resources to support a variety of funding mechanisms.We have now analyzed the overall results for R01 grants using both our regular appropriation and Recovery Act funds. These results are shown in Figures 1-3.

Figure 1. Competing R01 applications reviewed (open rectangles) and funded (solid bars) in Fiscal Year 2009. The thicker bars (blue) correspond to applications supported using regular appropriated funds, while the thinner bars (red) correspond to applications supported using Recovery Act funds (2-year awards).

Figure 1. Competing R01 applications reviewed (open rectangles) and funded (solid bars) in Fiscal Year 2009. The thicker bars (blue) correspond to applications supported using regular appropriated funds, while the thinner bars (red) correspond to applications supported using Recovery Act funds (2-year awards).
Figure 2. NIGMS competing R01 funding curves for Fiscal Years 2005-2009. For Fiscal Year 2009, two curves are shown. The thicker curve (black) corresponds to grants made with regular appropriated funds, while the thinner curve (red) includes grants made with both regular appropriated and Recovery Act (ARRA) funds.

Figure 2. NIGMS competing R01 funding curves for Fiscal Years 2005-2009. For Fiscal Year 2009, two curves are shown. The thicker curve (black) corresponds to grants made with regular appropriated funds, while the thinner curve (red) includes grants made with both regular appropriated and Recovery Act (ARRA) funds. The success rate for R01 applications paid with regular appropriated funds was 27%, and the midpoint of the funding curve was at approximately the 22nd percentile. This percentile is slightly lower than that for Fiscal Year 2008. The success rate for R01 applications paid with regular appropriated and Recovery Act funds in Fiscal Year 2009 was 32%, with a midpoint on the funding curve near the 30th percentile. The curve including Recovery Act-funded awards is fairly broad because NIGMS considered additional factors in making funding decisions for Recovery Act awards.

The total NIGMS expenditures (including both direct and indirect costs) for R01 grants are shown in Figure 3 for Fiscal Year 1995 through Fiscal Year 2009.

Figure 3.  Overall NIGMS expenditures on R01 grants (competing and noncompeting, including supplements) in Fiscal Years 1995-2009.  The dotted line shows the impact of awards (including supplements) made with Recovery Act funds.  Results are in actual dollars with no correction for inflation.

Figure 3. Overall NIGMS expenditures on R01 grants (competing and noncompeting, including supplements) in Fiscal Years 1995-2009. The dotted line shows the impact of awards (including supplements) made with Recovery Act funds. Results are in actual dollars with no correction for inflation.

We are analyzing additional data on NIGMS funding trends and will be posting these results on the NIGMS Funding Trends Web site.

Nearly Half of NIGMS Recovery Act Funds Now Awarded

12 comments

Three weeks ago, I posted a graph of the cumulative investment of NIGMS Recovery Act funds as a function of the award start date. Below is an updated version.

This graph shows that the rate at which we have made Recovery Act awards from July to September has accelerated.

This graph differs from my previous one in two ways. First, it is up to date as of yesterday. Second, it includes commitments for the second year of awarded 2-year R01 grants. As it shows, we are close to allocating half of our $505 million of Recovery Act funds. We are making more awards every day, with the Challenge grants, GO grants, and faculty start-up (P30) grants to be awarded soon.

The awards made to date can be broken down into seven major categories.

This graph shows the amount and total number of awards in each category. R01: 112. R01 equivalent supplements: 511. AREA grants (R15): 11.  AREA grants supplements: 22. MBRS SCORE (S06, SC1,2,3, R25) supplements: 77. MARC (T34) supplements. Research training grant (T32) supplements: 90.

This graph shows the amount and total number of awards (blue type above each bar) in each category. The gray-shaded area over the R01 category reflects second-year commitments.

To put these figures in context, NIGMS currently supports approximately 3,600 R01 grants; 50 AREA (R15) grants; 250 Minority Biomedical Research Support (MBRS) S06, SC1, SC2, SC3, and R25 grants; 50 Minority Access to Research Careers (MARC) T34 grants; and 300 institutional research training T32 grants.

We will continue to update you on our Recovery Act activities as we move from one fiscal year to the next at the end of this month.