Tag: NIGMS Advisory Council

Posted on

The Advisory Council’s Critical Roles

by
5 comments

Later this month, the National Advisory General Medical Sciences Council will hold the first of its three meetings in 2011. While many applicants, grantees and reviewers are familiar with the roles and processes of study sections, fewer know how an advisory council works. In this post, I’ll provide an overview of its many critical roles.

Council members are leaders in the biological and medical sciences, education, health care and public affairs. Their areas of expertise cover the broad range of scientific fields supported by NIGMS. The Council performs the second level of peer review for research and research training grant applications assigned to NIGMS. Council members also offer advice and recommendations on policy and program development, program implementation, evaluation and other matters of significance to the mission and goals of the Institute.

A portion of each Council meeting is open to the public.

For the peer review function, which occurs during the part of the meeting that is closed to the public, Council members read summary statements, providing a general check on the quality of the first level of peer review. They advise us if they find cases where the comments and scores do not appear to be in good alignment. Their evaluation complements the initial peer review done by study sections, as it focuses primarily on summary statements rather than on applications (although Council members may have access to the applications).

Members also provide advice regarding formal appeals, typically discussing 10-20 cases per meeting in which a procedural aspect may have significantly influenced the initial peer review process.

The Council also provides input on cases where staff are considering exceptions to the well-funded laboratory policy, and it approves the potential funding of grants to investigators at foreign institutions. Another area of Council input relates to Method to Extend Research in Time (MERIT) awards. Finally, Council members point out applications that they feel are particularly interesting based on their scientific expertise and knowledge of trends in particular fields. They explain their perspective to NIGMS staff, who incorporate this input in subsequent steps of the funding decision process. I’ll describe these steps in an upcoming post.

The policy and program advisory function includes discussing “concept clearances,” or ideas for new initiatives being considered within the Institute. These can take the form of proposed requests for applications (RFAs) or program announcements (PAs). Council members provide critical analysis and feedback about the appropriateness of proposed initiatives and factors to consider should they be implemented. Approved concept clearances are posted soon after each Council meeting on the NIGMS Web site and often on the Feedback Loop. NIGMS staff can then receive input from the scientific community as they refine the funding opportunity announcements.

This month’s meeting will include one concept clearance presentation, on macromolecular complexes.

Council members also give input and feedback on assessments and formal evaluations of specific NIGMS programs, such as the Protein Structure Initiative. When the need arises, Council members form working groups focused on specific issues. To ensure an appropriate range of expertise and perspectives, these groups can include non-Council members, as well. Finally, the Council receives periodic reports about ongoing initiatives in order to monitor how they are proceeding and offer advice about possible changes.

Posted on

Measuring the Scientific Output and Impact of NIGMS Grants

by
29 comments

A frequent topic of discussion at our Advisory Council meetings—and across NIH—is how to measure scientific output in ways that effectively capture scientific impact. We have been working on such issues with staff of the Division of Information Services in the NIH Office of Extramural Research. As a result of their efforts, as well as those of several individual institutes, we now have tools that link publications to the grants that funded them.

Using these tools, we have compiled three types of data on the pool of investigators who held at least one NIGMS grant in Fiscal Year 2006. We determined each investigator’s total NIH R01 or P01 funding for that year. We also calculated the total number of publications linked to these grants from 2007 to mid-2010 and the average impact factor for the journals in which these papers appeared. We used impact factors in place of citations because the time dependence of citations makes them significantly more complicated to use.

I presented some of the results of our analysis of this data at last week’s Advisory Council meeting. Here are the distributions for the three parameters for the 2,938 investigators in the sample set:

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

Histograms showing the distributions of total annual direct costs, number of publications linked to those grants from 2007 to mid-2010 and average impact factor for the publication journals for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006.

For this population, the median annual total direct cost was $220,000, the median number of grant-linked publications was six and the median journal average impact factor was 5.5.

A plot of the median number of grant-linked publications and median journal average impact factors versus grant total annual direct costs is shown below.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

A plot of the median number of grant-linked publications from 2007 to mid-2010 (red circles) and median average impact factor for journals in which these papers were published (blue squares) for 2,938 investigators who held at least one NIGMS R01 or P01 grant in Fiscal Year 2006. The shared bars show the interquartile ranges for the number of grant-linked publications (longer red bars) and journal average impact factors (shorter blue bars). The medians are for bins, with the number of investigators in each bin shown below the bars.

This plot reveals several important points. The ranges in the number of publications and average impact factors within each total annual direct cost bin are quite large. This partly reflects variations in investigator productivity as measured by these parameters, but it also reflects variations in publication patterns among fields and other factors.

Nonetheless, clear trends are evident in the averages for the binned groups, with both parameters increasing with total annual direct costs until they peak at around $700,000. These observations provide support for our previously developed policy on the support of research in well-funded laboratories. This policy helps us use Institute resources as optimally as possible in supporting the overall biomedical research enterprise.

This is a preliminary analysis, and the results should be viewed with some skepticism given the metrics used, the challenges of capturing publications associated with particular grants, the lack of inclusion of funding from non-NIH sources and other considerations. Even with these caveats, the analysis does provide some insight into the NIGMS grant portfolio and indicates some of the questions that can be addressed with the new tools that NIH is developing.