Dr. Peggy Weidman

About Dr. Peggy Weidman

Peggy is a cell biologist who organizes the scientific review of grant applications for programs that seek to increase the diversity of the biomedical research workforce.

Additional Reviewer Scoring Guidance for Overall Impact Scores on Research Applications

NIH has posted additional scoring guidance for reviewers to consider when determining overall impact scores for grant applications. Here are answers to some key questions about this new guidance.

Why provide more guidelines?

Scientific review officers and program directors noticed that reviewers have tended to arrive at overall impact scores by comparing the number of weaknesses and strengths in an application, rather than balancing the importance of the weaknesses and strengths. In addition, there was significant compression of overall impact scores around the perceived funding range. Both practices made it more difficult to gauge reviewer assessments.

What’s changed?

The additional guidance chart simplifies and clarifies the way in which reviewers should evaluate the overall impact of an application. In particular, it encourages reviewers to focus on the importance of the research problem and the likelihood that the project will succeed. The chart emphasizes a balanced assessment of the review criteria and the use of the entire scoring range. The general Scoring System and Procedure also includes a new, simplified scoring guidance chart for assigning individual criterion scores.

When will this change occur?

Some Center for Scientific Review (CSR) study sections used the new guidelines in application reviews for May 2013 advisory council meetings. So that these applications would not be disadvantaged by the deviation from the impact score distribution of previous review cycles used for percentiling, CSR recalculated the percentile base. The vast majority of study sections and special emphasis panels are using the new scoring guidance with the reviews for fall 2013 advisory council meetings.

Note: New scoring guidelines are also available for fellowships, career awards and institutional training grants.

How NIH Makes Grant Application Assignments

Here are answers to some of the questions we’re frequently asked about grant application assignments.

Who receives applications and makes assignments?

All applications are received by the Division of Receipt and Referral (DRR) in the Center for Scientific Review (CSR). The DRR gives each application two assignments, one for review and one for funding consideration.

How are review assignments made?

Referral officers in CSR assign an application to an integrated review group (IRG), a branch of CSR that manages the review of applications in a general scientific area. The chief of the IRG or his/her representative, in consultation with scientific review officers (SROs) in the IRG, makes the final assignment to a specific scientific review group (SRG), which can be a standing study section (SS) or a special emphasis panel (SEP). An SS reviews applications in a specific topic area within the purview of the IRG. A SEP reviews applications on special topics or when conflicts of interest exist. Find an IRG, SS or SEP.

Why isn’t the study section listed on my application?

Since the application is first assigned to an IRG, the IRG abbreviation initially appears as the review assignment. This abbreviation is replaced by the SRG label once that assignment has been made. A similar process occurs with applications assigned to an NIH institute or center (IC) for review. At NIGMS, the initial assignment is to our Office of Scientific Review, followed by assignment to a specific review group and SRO.

How are assignments for funding consideration made?

The DRR assigns the application to an IC for funding consideration. In some cases, the DRR may make a secondary assignment to another IC. The assignment is based on referral guidelines developed by each IC. These describe the IC’s focus and mission areas, interests shared with other ICs, and the funding opportunity announcements that the IC supports. Other considerations may include requests made by investigators or by ICs for secondary assignments.

Why wasn’t my application assigned to the person I thought was my program director?

It’s likely that your application received a “temporary” assignment. When an IC receives an application, it assigns a temporary program director. At NIGMS, this person is Ann Hagan, our associate director for extramural activities. Once the application has been matched with the most appropriate NIGMS division, branch and program director, it will receive a final assignment.

Why was my application assigned to an IC or SS that I didn’t request?

If you have traditionally been funded by one IC and request a change to another IC, reassignment will occur only if the application is deemed a better fit with the new IC. Another factor is the IC’s program interests, as described in its referral guidelines, program announcements (PAs) and requests for applications (RFAs). Many of these announcements are only supported by one or a subset of ICs. If an application is submitted in response to a funding opportunity announcement that is not supported by the requested IC, then it can’t be assigned to that IC.

Several factors influence the likelihood that a request for review assignment to a particular SS will be honored. The most important factor is whether the proposed research is a good fit for the scientific focus of the requested SS. Like science itself, the scientific focus of an SS evolves over time. Therefore, the SS that reviewed your application 4 years ago may no longer be suitable, and the IRG may make a different assignment. The funding opportunity announcement can also play a role in the review assignment. Applications for many RFAs and PAs with special receipt, referral and/or review considerations are reviewed by SEPs organized by CSR or by IC-specific scientific review offices. If the application is responding to one of these funding opportunity announcements, it can’t be assigned to a standing SS for review.

What should I do if I don’t get a requested study section or IC assignment?

You should contact the SRO or program director who was assigned the application. If, after discussion, a reassignment is warranted, that person will facilitate the change. If you still have concerns, you should contact the DRR.

For more details on this topic, read the CSR’s The Assignment Process.

Why Overall Impact Scores Are Not the Average of Criterion Scores

One of the most common questions that applicants ask after a review is why the overall impact score is not the average of the individual review criterion scores. I’ll try to explain the reasons in this post.

What is the purpose of criterion scores?

Criterion scores assess the relative strengths and weaknesses of an application in each of five core areas. For most applications, the core areas are significance, investigator(s), innovation, approach and environment. The purpose of the scores is to give useful feedback to PIs, especially those whose applications were not discussed by the review group. Because only the assigned reviewers give criterion scores, they cannot be used to calculate a priority score, which requires the vote of all eligible reviewers on the committee.

How do the assigned reviewers determine their overall scores?

The impact score is intended to reflect an assessment of the “likelihood for the project to exert a sustained, powerful influence on the research
field(s) involved.” In determining their preliminary impact scores, assigned reviewers are expected to consider the relative importance of each scored review criterion, along with any additional review criteria (e.g., progress for a renewal), to the likely impact of the proposed research.

The reviewers are specifically instructed not to use the average of the criterion scores as the overall impact score because individual criterion scores may not be of equal importance to the overall impact of the research. For example, an application having more than one strong criterion score but a weak score for a criterion critical to the success of the research may be judged unlikely to have a major scientific impact. Conversely, an application with more than one weak criterion score but an exceptionally strong critical criterion score might be judged to have a significant scientific impact. Moreover, additional review criteria, although not individually scored, may have a substantial effect as they are factored into the overall impact score.

How is the final overall score calculated?

The final impact score is the average of the impact scores from all eligible reviewers multiplied by 10 and then rounded to the nearest whole number. Reviewers base their impact scores on the presentations of the assigned reviewers and the discussion involving all reviewers. The basis for the final score should be apparent from the resume and summary of discussion, which is prepared by the scientific review officer following the review.

Why might an impact score be inconsistent with the critiques?

Sometimes, issues brought up during the discussion will result in a reviewer giving a final score that is different from his/her preliminary score. If this occurs, reviewers are expected to revise their critiques and criterion scores to reflect such changes. Nevertheless, an applicant should refer to the resume and summary of discussion for any indication that the committee’s discussion might have changed the evaluation even though the criterion scores and reviewer’s narrative may not have been updated. Recognizing the importance of this section to the interpretation of the overall summary statement, NIH has developed a set of guidelines to assist review staff in writing the resume and summary of discussion, and implementation is under way.

If you have related questions, see the Enhancing Peer Review Frequently Asked Questions.

Editor’s Note: In the third section, we deleted “up” for clarity.