Author: Jeremy Berg

Headshot of Jeremy Berg.

As former NIGMS director, Jeremy oversaw the Institute’s programs to fund biomedical research and to train the next generation of scientists. He was a leader in many NIH-wide activities and also found time to study a variety of molecular recognition processes in his NIH lab.

Posts by Jeremy Berg

Even More on Criterion Scores: Full Regression and Principal Component Analyses

4 comments

After reading yesterday’s post, a Feedback Loop reader asked for a full regression analysis of the overall impact score based on all five criterion scores. With the caveat that one should be cautious in over-interpreting such analyses, here it is:

Pearson correlation coefficients of overall impact score and five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. The various parameters are substantially correlated.

Pearson correlation coefficients of overall impact score and five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As one might expect, the various parameters are substantially correlated.

A principal component analysis reveals that a single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Here are more results of the full principal component analysis:

Principal component analysis of overall impact score based on the five criterion scores (significance, approach, innovation, investigator and environment) in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. A single principal component accounts for 71% of the variance in the overall impact scores. This principal component includes substantial contributions from all five criterion scores, with weights of 0.57 for approach, 0.48 for innovation, 0.44 for significance, 0.36 for investigator and 0.35 for environment.

Principal component analysis of overall impact score based on the five criterion scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

The second component accounts for an additional 9% of the variance and has a substantial contribution from approach, with significant contributions of the opposite sign for investigator and environment. The third component accounts for an additional 8% of the variance and appears to be primarily related to innovation. The fourth component accounts for an additional 7% of the variance and is primarily related to significance. The final component accounts for the remaining 5% of the variance and has contributions from investigator and environment of the opposite sign.

More on Criterion Scores

6 comments

In an earlier post, I presented an analysis of the relationship between the average significance criterion scores provided independently by individual reviewers and the overall impact scores determined at the end of the study section discussion for a sample of 360 NIGMS R01 grant applications reviewed during the October 2009 Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey and Medical Writing, Editing & Grantsmanship Link to external web site, I want to provide some additional aspects of this analysis.

As I noted in the recent post, the criterion score most strongly correlated (0.74) with the overall impact score is approach. Here is a plot showing this correlation:

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. 

Plot of approach and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Similarly, here is a plot comparing the average innovation criterion score and the overall impact score:

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. 

Plot of innovation and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Note that the overall impact score is NOT derived by combining the individual criterion scores. This policy is based on several considerations, including:

  • The effect of the individual criterion scores on the overall impact score is expected to depend on the nature of the project. For example, an application directed toward developing a community resource may not be highly innovative; indeed, a high level of innovation may be undesirable in this context. Nonetheless, such a project may receive a high overall impact score if the approach and significance are strong.
  • The overall impact score is refined over the course of a study section discussion, whereas the individual criterion scores are not.

That being said, it is still possible to derive the average behavior of the study sections involved in reviewing these applications from their scores. The correlation coefficient for the linear combination of individual criterion scores with weighting factors optimized (approximately related to the correlation coefficients between the individual criterion scores and the overall impact factor) is 0.78.

The availability of individual criterion scores provides useful data for analyzing study section behavior. In addition, these criterion scores are important parameters that can assist program staff in making funding recommendations.

Model Organisms and the Significance of Significance

13 comments

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Link to external web site sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round. 

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

Presenting NIGMS to the NIH Director’s Advisory Committee

0 comments

Presentation to the Advisory Committee to the DirectorThe Advisory Committee to the Director, NIH (ACD) is a group knowledgeable in the fields of research pertinent to the NIH mission. It includes individuals from the academic and private sector research communities as well as representatives of the general public.

The ACD meets in person twice a year to provide advice on a range of NIH activities. At almost every meeting, the NIH Director invites an institute or center director to present information about his or her organization, including his or her vision of its key features.

I was delighted when Dr. Collins invited me to present at the most recent ACD meeting. In my talk, I highlighted the Institute’s focus on investigator-initiated research and aspects of our role in training and workforce development. I also described some advances in five areas across NIGMS, including structural biology and the Protein Structure Initiative, RNA biology, pharmacogenomics, the Institutional Research and Academic Career Development Award (IRACDA) postdoctoral program and the Models of Infectious Disease Agent Study (MIDAS) program.

A video of my talk and a question-and-answer session is now available. It begins at minute 138:45 with an introduction by Dr. Collins.

NIGMS Grantee and Council Member Wins Lemelson-MIT Prize

1 comment

Congratulations to Carolyn Bertozzi!

We were delighted to learn that this longtime NIGMS grantee, current NIGMS Advisory Council member and former Stetten lecturer has been awarded the Lemelson-MIT Prize Link to external web site. The prize honors technological invention and innovation.

Carolyn is a clear leader in chemical biology and glycobiology whose work extends from very basic studies of chemical reactivity through a variety of applications in biology. In addition, she is a committed teacher and mentor.

Please join me in congratulating her for this award in recognition of her many significant contributions.

Hearings on Fiscal Year 2011 Budget

1 comment

Congressional hearings on the Fiscal Year 2011 President’s budget request for NIH are now under way. The hearing before the House subcommittee that handles NIH appropriations began at 10 a.m. today, and the Senate hearing will take place on May 5. The ultimate outcome will be a bill that appropriates funds for NIH, including NIGMS.

My written testimony and NIH Director Francis Collins’ written testimony on next year’s budget are now available.

Blogging for One Year!

1 comment

Today marks the 1-year anniversary of our conversion of the NIGMS Feedback Loop from a thrice-yearly electronic newsletter into a blog. The year has passed quickly, but I think successfully.

Our inaugural post, Answering Your Recovery Act Questions, was the first of 16 entries related to the Recovery Act, and many of these led to extensive comments and replies. Throughout this extremely fast-paced period, the blog format served us all well: We provided you information in a timely fashion, and you helped us better understand what questions and concerns we needed to address.

The Feedback Loop has also covered many other topics. These are grouped into over 20 categories for easy browsing or searching. As of today, the site includes 86 posts by 28 NIGMS staff members.

During the last year, many people from the scientific community have told me how much they appreciate receiving information from us about NIGMS and NIH. The Feedback Loop is our major outlet for doing this, and we want make it as useful as possible to you and to us. Please don’t hesitate to comment on posts, e-mail me suggestions or other ideas or submit new post topics to the blog’s editor.

Online Site for Giving Input on Research Training and Career Development Closes Next Week

1 comment

Give us your input on the future of NIGMS-sponsored research trainingIn March, I announced the launch of a strategic planning process focused on training and career development. I encourage all stakeholders, including faculty members, postdocs, graduate students, university administrators and government and industry scientists to provide input on this important topic.

One easy way to make your voice heard is through our online form (no longer available). We list several questions that you can respond to anonymously. You don’t have to respond to every question, and you can also make other comments or recommendations.

If you have not yet shared your views, there’s still time—the Web site closes April 21.

Save the Date: Commemorating the Life and Accomplishments of Ruth Kirschstein

2 comments

Ruth Kirschstein, M.D.Since I first wrote about the death of former NIGMS director Dr. Ruth Kirschstein, plans have progressed for a daylong commemorative event on Monday, May 17, at NIH. The program will include several remembrances as well as scientific presentations and posters by recipients of the Ruth L. Kirschstein National Research Service Award.

I will update you when the final program is available.

New NIH Director’s Initiative on Scientific Workforce Diversity

0 comments

NIH recently announced the NIH Director’s Pathfinder Award to Promote Diversity in the Scientific Workforce, a new program under the American Recovery and Reinvestment Act that NIGMS will manage. The program is designed to foster new ways of thinking about initiatives related to scientific workforce diversity. Awards will empower exceptionally creative scientists to develop highly innovative, and possibly transformative, approaches to this complex challenge.

NIH expects to make approximately five awards, each up to $2 million in total costs over a 3-year period. Awardees must commit a substantial portion (generally 30% or more) of their research effort to the funded activities. Also note that letters of intent—which are encouraged but not required—are due by April 5 and that applications are due by May 4.

If you’d like more information, please e-mail Clif Poodry or call him
at 301-594-3900.