Category: Peer Review

As part of NIH’s efforts to address racial disparities  in grant funding, the Center for Scientific Review (CSR) has just launched two America COMPETES Act Challenges. We hope that the ideas we receive will help us maximize the fairness and vitality of the peer review process, and we encourage you to enter.

One challenge, New Methods to Detect Bias in Peer Review, solicits ideas for strategies to detect possible bias in the NIH peer review process. Submissions can include approaches, strategies, methodologies and/or measures that would be sensitive to detecting bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science and amount of research experience. We’ll award first place ($10,000) and second place ($5,000) prizes in two categories: best empirically based idea and most creative idea.

The other challenge, Strategies to Strengthen Fairness and Impartiality in Peer Review, seeks ideas for reviewer training methods aimed at enhancing fairness and impartiality in NIH peer review. The submission does not require the full development of training materials. However, ideas should be presented with enough detail to allow assessment of their ability to address fairness and impartiality in review with regard to gender, race/ethnicity, institutional affiliation, area of science and amount of research experience. We’ll give first place ($10,000) and second place ($5,000) prizes for the best overall idea.

The challenges close on June 30, and winners will be announced on September 2. Details on the rules and submission procedures are on the CSR Challenge Web site and at http://www.challenge.gov  .

This contest is just one of many initiatives CSR is working on to evaluate the sources of racial disparities in grant funding in collaboration with the ACD Diversity Working Group Subcommittee on Peer Review.

A recent article in NIH’s Peer Review Notes illustrated how Alejandro Sánchez Alvarado developed the planarian as a model organism for studying regeneration.

When he applied for his first NIH grant in 1997, Sánchez Alvarado was a new investigator at the Carnegie Institution of Washington who had little preliminary data supporting his proposal. The peer reviewers astutely recognized the promise of his grant application, and they scored it favorably.

NIGMS funded the grant and continues to support Sánchez Alvarado’s studies, now at the Stowers Institute for Medical Research. Since receiving that first grant, Sánchez Alvarado has helped turn planaria into tiny workhorses that have opened new avenues for understanding tissue regeneration and other developmental processes. He has also trained and mentored graduate students and postdocs and served the scientific community as a study section and NIGMS advisory council member.

Sánchez Alvarado’s work is just one example of how NIGMS-funded research on model organisms, including those that are currently considered unconventional, has significantly advanced scientific progress. Think of the protozoan Tetrahymena thermophila and the discovery of telomeres, or the nematode Caenorhabditis elegans and the discovery of RNA interference. The fungus Neurospora crassa has been an important player in understanding the genetics of circadian rhythms, and the bacterium Caulobacter crescentus has led to important insights on the regulation of cell division.

We have long supported research using these and many more model organisms, including mammalian species and other vertebrates, invertebrates, plants, yeast, and additional bacteria. We know that findings made in such systems can translate rapidly—and often unexpectedly—into leaps in conceptual understanding of fundamental life processes, the development of better research tools and approaches, and new strategies for diagnosing and treating diseases. We also know that we can never predict where the next breakthrough will come from—or in which organism. So we continue to encourage applications for projects that use model organisms, including uncommon ones, when these systems offer distinct scientific advantages.

One of my jobs as an NIGMS program director is helping investigators navigate the review process. This includes understanding where their applications will be reviewed and how they can make a recommendation about this assignment.

Applications can be grouped for review by research area or grant mechanism, or as a cohort submitted in response to a specific funding opportunity announcement (FOA). I would like to briefly walk through a few scenarios and share some advice along the way.

The “Review and Selection Process” (V.2) section of FOAs provides clues about where your application will be reviewed. An application can be reviewed at the Center for Scientific Review (CSR) or at an individual NIH institute or center (IC), depending upon the particular FOA. Information about who will review your application is posted in your eRA Commons account soon after it is determined, but you should contact your program director or CSR if you have questions or concerns.

The vast majority of applications received by NIH on topics relevant to NIGMS are in response to “parent” program announcements, such as PA-13-302, for unsolicited R01 applications. Section V.2 of PA-13-302 states, “Applications will be evaluated for scientific and technical merit by (an) appropriate Scientific Review Group(s) convened by CSR.” This means that the application will be reviewed by a regular NIH study section or a special emphasis panel (SEP) with expertise in the research area explored by the application. To identify a possible “review home” for your application, I suggest you peruse CSR’s list of study sections, find the ones that seem most suitable for your application and then use NIH RePORTER to search for funded applications that have been reviewed by those study sections. This will allow you to identify the group of scientists who have the appropriate research expertise to review your application.

Specific requests for applications (RFAs), such as RFA-GM-14-003 (Revisions for Macromolecular Interactions in Cells), are often reviewed together in the IC that issued the RFA. For example, section V.2 of GM-14-003 states, “Applications will be evaluated for scientific and technical merit by (an) appropriate Scientific Review Group(s) convened by the NIGMS.” This means that the NIGMS Office of Scientific Review will organize a review panel to review applications submitted in response to this RFA.

For applications that will be reviewed by groups convened by CSR, I encourage the investigators I speak with to write a brief cover letter for their applications that indicates which study section they think is appropriate and how they arrived at that conclusion. Sometimes, it is also helpful to indicate the type(s) of expertise you believe is needed to review your application, but you should not provide a list of reviewers, as that creates issues with potential conflicts of interest. The recommendations made in the cover letter are advisory, but the CSR Division of Receipt and Referral makes every effort to accommodate reasonable requests.

NIH has posted additional scoring guidance for reviewers to consider when determining overall impact scores for grant applications. Here are answers to some key questions about this new guidance.

Why provide more guidelines?

Scientific review officers and program directors noticed that reviewers have tended to arrive at overall impact scores by comparing the number of weaknesses and strengths in an application, rather than balancing the importance of the weaknesses and strengths. In addition, there was significant compression of overall impact scores around the perceived funding range. Both practices made it more difficult to gauge reviewer assessments.

What’s changed?

The additional guidance chart simplifies and clarifies the way in which reviewers should evaluate the overall impact of an application. In particular, it encourages reviewers to focus on the importance of the research problem and the likelihood that the project will succeed. The chart emphasizes a balanced assessment of the review criteria and the use of the entire scoring range. The general Scoring System and Procedure also includes a new, simplified scoring guidance chart for assigning individual criterion scores.

When will this change occur?

Some Center for Scientific Review (CSR) study sections used the new guidelines in application reviews for May 2013 advisory council meetings. So that these applications would not be disadvantaged by the deviation from the impact score distribution of previous review cycles used for percentiling, CSR recalculated the percentile base. The vast majority of study sections and special emphasis panels are using the new scoring guidance with the reviews for fall 2013 advisory council meetings.

Note: New scoring guidelines are also available for fellowships, career awards and institutional training grants.

The Center for Scientific Review (CSR) has recently adjusted the base that a number of permanent study sections and special emphasis panels (SEPs) use to calculate percentiles for certain application types. Applications that were reviewed in a SEP meeting for the May 2013 Council round will be assigned a percentile ranking calculated from the new base. In some cases, revised summary statements will be issued for those applications missing percentile scores or with an incorrect percentile. The percentile ranking will also appear in the eRA Commons account of the principal investigator(s).

We here at NIGMS are often asked to explain how percentile rankings factor into determining which awards to make. We do not rely solely on a percentile cutoff; we consider a number of additional factors, including an investigator’s career stage, a laboratory’s other research funding, NIGMS research portfolio balance, and other programmatic priorities. For more information, see http://www.nigms.nih.gov/Research/Application/Pages/SuccessRateFAQs.aspx.