Model Organisms and the Significance of Significance

I recently had the opportunity to speak at the Model Organisms to Human Biology meeting Exit icon sponsored by the Genetics Society of America. I shared some of my perspectives on the powerful interplay between studies of model organisms and studies of humans (both individuals and populations) enabled through genetics. I illustrated why results over many decades have shown that studying fundamental mechanisms in a wide range of organisms can elucidate important processes relevant to human health and disease.

I also discussed aspects of the NIH peer review system, particularly with regard to proposed studies of model organisms.

One of the key changes in the new peer review system is the use of individual scores for five specific criteria. During my talk, I focused on the significance criterion:

Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

This definition is intended to cover the entire range of research supported by NIH, spanning basic studies of fundamental mechanisms through applied studies that have the potential for direct clinical impact.

Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

To examine how reviewers apply the significance criterion in determining overall impact scores, I analyzed 360 NIGMS R01 applications reviewed during the October 2009 Council round. A plot comparing the average significance scores with the overall impact scores for these applications is shown below.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

Plot of significance and overall impact scores in a sample of 360 NIGMS R01 applications reviewed during the October 2009 Council round.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

This analysis indicates that approach and significance are the most important factors, on average, in determining the overall impact score, at least for this sample of NIGMS R01 grant applications.

UPDATE: Jeremy Berg has posted similar analyses of the approach and innovation criteria.

13 comments on “Model Organisms and the Significance of Significance

  1. For years NIH has tried various changes in grant format, partly to try to reward high impact, high risk studies and to encourage innovative research. The general consensus is that study sections are too conservative in rewarding projects that are quite “doable” but not projects that emphasize innovation but are risky.

    Your correlation coefficient analysis is revealing with regard to the lack of high correlation of the new Innovation section of the grant. This tells me that no matter what format is adopted for the R01 grant, the deciding factor is the culture of the study section.

    The conclusion then is that more effort should be made to instill uniform practices and guidelines among study sections and less on changes to the grant format.

  2. Some applicants who use model organisms try to explain the significance of their project by making relatively tenuous links to specific clinical areas. As an alternative, they should consider highlighting the study’s importance to a basic field of biomedical or behavioral research and the reason for using a specific experimental system.

    As Director of one of the National Institutes of HEALTH, it is tragic that you have this completely wrong. As an alternative to applicants pitching the tenuous links of their “model organism” to specific clinical areas, applicants should work on organisms with less tenuous links to specific clinical areas.

    • Come on, really ? What’s the point of doing basic research on fundamental biology if you constantly have to make a link to some specific clinical application? Without basic research, there is nothing to apply. It’s not truly “research” if you already know what you’re going to find. If all scientists thought this way, we’d be decades behind where we are right now.

    • A major theme of my presentation (and, indeed, of the Model Organisms to Human Biology meeting) was how studies of model organisms can have a substantial impact relevant to human health. This approach is empowered by the tremendous unity of biology at the cellular and molecular levels. Consider, for example, fundamental studies of developmental pathways in the nematode C. elegans. These investigations led to the discovery of RNA interference, an unanticipated mechanism that has produced one of the most powerful tools in biological research and that also represents an entirely new approach to human therapy. Additional studies in this organism resulted in the discovery of microRNAs, non-protein-coding gene products that play significant roles in regulating gene expression in many organisms, including humans. Genomic studies have implicated microRNA variations in a range of human diseases. It is quite unlikely that these fundamental pathways of great relevance to human health would have been discovered through more disease-focused studies. (For more on some of the applications of model organisms, see our fact sheet).

      Accomplishing the mission of the National Institutes of Health requires a breadth of research strategies. Clearly, investing in studies closely tied to human beings with the goal of developing knowledge that can be directly linked to improvements in health is a key component. At the same time, given how much about biological systems remains to be discovered, continued investments directed toward elucidating fundamental processes in biology are also essential for future medical advances.

  3. Unfortunately, a director at the NIH clearly does not comprehend the use of model organisms and how an understanding of basic biology has tremendous impact on clinical areas. In the long run if the National Institutes of HEALTH continues the short sighted policies espoused by whimple, there will not be any new understanding or knowledge on which to confront human health issues.

  4. Vis-a-vis the new scoring system, it is interesting to note that the strongest correlation remains between “overall score” and “approach”. Wasn’t the point of the new system to increase the emphasis on “significance” and “innovation” and get away from “approach”?

  5. Jeremy, could you address why this dataset doesn’t seem to show the “peaks” in number of grants receiving 20 and 30 and 40 impact scores that have been seen in CSR datasets, and that occur when all assigned reviewers give the same impact score and no one votes outside the range?

    • The absence of these peaks may be due to the relatively small size of this data set or to the fact that these scores are derived from a number of different study sections with a range of practices with regard to voting out of range. We have seen such peaks in other data sets that we have examined.

  6. And looking at the graph some more, it does make perfect sense that the significance scores mostly look quantized in thirds of a unit, since most of the time there are three assigned reviewers.

  7. To add to the comment made by Deutschman, these data show that about one-half of the variability in significance can be accounted for by the rated approach while only about one-third of the variability in significance can be accounted for by the rated impact. Stated slightly differently, while the strongest correlation accounts for a little over one-half of the variability in significance, there’s another one-half of the the variability in the rated significance of the proposed research that is explained by other factors. It might be more informative if NIGMS and NIH as a whole uses a multi-factor model in determining the relative contributions of the five criteria in determining overall significance.

  8. I would reiterate comments made by Belmont and Deutschman, that this provides quantitative evidence of a continued emphasis on approach over innovation and/or significance. Higher-risk projects, or technically demanding projects proposed by those who have not yet published in such areas are not rewarded. This means NIH is not benefiting from the full range of good ideas and innovation available, and is not helping to develop that expertise in labs that don’t already have it.

  9. My impression is that most reviews are driven by the response to the approach. This fits with your report of a .74 correlation of approach and overall impact. I think you would find that it is a rare application that is viewed as being highly significant but has a weak approach. That is, reviewers use their approach score to drive their significance score. If this is true, significance should correlate better with approach than with any of the other elements.

  10. I think this information is very helpful in putting together one’s grant, even if it can be difficult to ascertain what a given reviewer will consider a worthy approach. I don’t know that any revising of the standards of “grading” will ever make funding more fair. Unfortunately, if your grant doesn’t grab the immediate interest of the 3 scientists who happen to review your grant, no amount of impeccable science will save you. I’m not sure that telling people how they should grade a grant, or what they should emphasize when assigning a score, will truly alter the gut reaction of the reviewer, which is really the main factor. That said, it seems that making your approach clear to the reviewer, and ensuring that your approach makes sense to a random group of your peers prior to submitting, should certainly help one get funded! So I for one will heed this correlation.

Post a Comment

Required fields are marked *

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.

*