Author: Dr. Jake Basson

Headshot of Jacob Basson.

Jake, who trained in biostatistics and cardiovascular genetics, is a statistical policy analyst in the NIGMS Division of Data Integration, Modeling, and Analytics. He uses a diverse suite of data science tools to study the Institute’s research portfolios, training programs, and funding policies.

Posts by Dr. Jake Basson

Application, Review, Funding, and Demographic Trends for Maximizing Investigators’ Research Awards (MIRA): FY 2016-2018

1 comment

NIGMS has made MIRA awards to Established Investigators (EI) and Early-Stage Investigators (ESI) for three full Fiscal Years (FY). In this Feedback Loop post, we provide an analysis of application, review, funding, and demographic trends for the MIRA program.

For the first two rounds of EI MIRAs, eligibility was limited to well-funded NIGMS investigators: PIs with two or more NIGMS R01-equivalent awards or one NIGMS R01-equivalent award for >$400,000 in direct costs. For the FY 2018 EI competition and beyond, eligibility was expanded to include any investigator with a single PD/PI NIGMS R01-equivalent that is up for renewal. For the FY 2016 ESI MIRA competition, ESIs and New Investigators (NI) at the assistant professor or equivalent level were eligible, whereas eligibility was restricted to ESIs in subsequent rounds. As always, a PI can apply for an extension of ESI status for various life and career events, including childbirth.

Continue reading “Application, Review, Funding, and Demographic Trends for Maximizing Investigators’ Research Awards (MIRA): FY 2016-2018”

Analysis of NIGMS Support of Research Organisms

1 comment

NIGMS is committed to supporting a wide-ranging portfolio of biomedically relevant fundamental research. As we discussed in a previous Feedback Loop post, we see this approach as the best way to increase our understanding of life. For many years, one important dimension of diversity in our scientific portfolio—the organisms scientists use to conduct their research—was limited by technical considerations. However, recent advances such as the decreasing cost of genome sequencing and the development of the CRISPR system for genetic modification now make it possible to use an expanded range of research organisms.

Continue reading “Analysis of NIGMS Support of Research Organisms”

More Information About New and Early Stage Investigator MIRA Outcomes

0 comments

There has been ongoing discussion—both here and in the general scientific community—related to the first MIRA awards to New and Early Stage Investigators (NI/ESI). One question that arose was why applications were administratively withdrawn. Both the NIH Center for Scientific Review and multiple NIGMS staff members, including the program director with a portfolio of grants closest to the applicant’s area of science, screened the applications. Of the withdrawn applications, a majority (~80%) were returned prior to review because they proposed research that fell outside of the NIGMS mission. Others were withdrawn because the applicant was not eligible for the FOA. After review, some applications were withdrawn because the PI accepted another award that was mutually exclusive with the MIRA. As recommended on the MIRA website and elsewhere, we encourage anyone who intends to apply for the Early Stage Investigator MIRA to discuss their plans with the appropriate NIGMS program director to determine whether the proposed research area is within the mission of the Institute and if the applicant is eligible to apply.

A major NIGMS goal is to support a broad portfolio that is diverse in research topics, approaches, institutions and investigators. This means we are looking carefully at the outcomes of awards, including gender and race/ethnicity data. We are also trying to take proactive steps to prevent bias during the review, for instance by covering the topic as part of reviewer orientations that take place several weeks before the MIRA study sections meet.

In our recent summary of MIRA applicant and awardee demographics, we looked to see how applications from underrepresented groups compared to those from well-represented groups (White and Asian). The p-value for a difference between the distributions of funded and unfunded applications from these groups was 0.63, meaning that there was no statistically significant difference between the two groups. We also compared the MIRA success rates to those of ESI applicants for NIGMS R01s in fiscal years (FY) 2011-2015 (Table 1).

Continue reading “More Information About New and Early Stage Investigator MIRA Outcomes”

Trending Young in New and Early Stage Investigator MIRA

4 comments

Dr. Jon Lorsch

The MIRA presentation at the September 2016 Advisory Council meeting begins at 17:13.

Following up on the previous post regarding the first MIRA awards to New and Early Stage Investigators, we issued awards to a total of 94 grantees. In addition to ensuring that we are funding the highest quality science across areas associated with NIGMS’ mission, a major goal is to support a broad and diverse portfolio of research topics and investigators. One step in this effort is to make sure that existing skews in the system are not exacerbated during the MIRA selection process. To assess this, we compared the gender, race/ethnicity and age of those MIRA applicants who received an award with those of the applicants who did not receive an award, as well as with New and Early Stage Investigators who received competitive R01 awards in Fiscal Year (FY) 2015.

We did not observe any significant differences in the gender or race/ethnicity distributions of the MIRA grantees as compared to the MIRA applicants who did not receive an award. Both groups were roughly 25% female and included ≤10% of underrepresented racial/ethnic groups. These proportions were also not significantly different from those of the new and early stage R01 grantees. Thus although the MIRA selection process did not yet enhance these aspects of the diversity of the awardee pool relative to the other groups of grantees, it also did not exacerbate the existing skewed distribution.

We did observe significant differences among the mean ages of the MIRA grantees, MIRA applicants who did not receive an award and the R01-funded grantees. The MIRA grantees are 1.5 years younger on average than those MIRA applicants who did not receive an award (37.2 vs. 38.7 years, p<0.05), and about 2 years younger than the FY 2015 R01-funded Early Stage Investigators (37.2 vs. 39.1 years, p<0.001). The R01-funded New Investigators in FY 2015, a pool which includes a few individuals older than 60 years, average an age of 45.6 years. This selection for funding investigators earlier is a promising feature of the first round of MIRA awards to New and Early Stage Investigators. As noted at the recent meeting of our Advisory Council, where Jon presented these data, 37 years is still relatively late for investigators to be getting their first major NIH grant. We will continue to monitor this issue with the goal of further decreasing that figure.

Revisiting the Dependence of Scientific Productivity and Impact on Funding Level

13 comments

A 2010 analysis by NIGMS and subsequent studies by others (Fortin and Currie, 2013; Gallo et al., 2014; Lauer et al., 2015; Doyle et al., 2015; Cook et al., 2015) have indicated that, on average, larger budgets and labs do not correspond to greater returns on our investment in fundamental science. We have discussed the topic here in A Shared Responsibility and in an iBiology talk Link to external website. In this updated analysis, we assessed measures of the recent productivity and scientific impact of NIGMS grantees as a function of their total NIH funding.

We identified the pool of principal investigators (PIs) who held at least one NIGMS P01 or R01-equivalent grant (R01, R23, R29, R37) in Fiscal Year 2010. We then determined each investigator’s total NIH funding from research project grants (RPGs) or center grants (P20, P30, P50, P60, PL1, U54) for Fiscal Years 2009 to 2011 and averaged it over this 3-year period. Because many center grants are not organized into discrete projects and cores, we associated the contact PI with the entire budget and all publications attributed to the grant. We applied the same methodology to P01s. Thus, all publications citing the support of the center or P01 grant were also attributed to the contact PI, preventing underrepresentation of their productivity relative to their funding levels. Figure 1 shows the distribution of PIs by funding level, with the number of PIs at each funding level shown above each bar.

Continue reading “Revisiting the Dependence of Scientific Productivity and Impact on Funding Level”

NIGMS Training Application and Funding Trends: Individual NRSA Postdoc and Pathway to Independence Awards

0 comments

Our Division of Training, Workforce Development, and Diversity (TWD) supports programs at multiple career stages to foster the development of a strong and diverse biomedical research workforce. This post is the first in a series focused on data from NIGMS training programs and is similar to the ones we have done for our research project grant portfolio. Below, we examine trends in NIGMS applications and awards for the Individual Postdoctoral National Research Service Award (NRSA) (F32) and Pathway to Independence Award (K99) programs. NIGMS also supports institutional postdoctoral awards that include the Institutional Research and Academic Career Development Awards (IRACDA) (K12) and NRSA Institutional Postdoctoral Training Grants (T32) focused in clinical areas, and data on these programs were shared previously.

NIGMS F32 Applications, Success/Award Rates, and Demographic Data – Past 5 Fiscal Years

The F32 fellowship program supports research training of highly promising postdoctoral candidates toward becoming productive, independent investigators in biomedical research fields relevant to the NIH mission. Within the last 5 fiscal years (FY 2015 to FY 2019), NIGMS received approximately 350 to 425 applications each year to the NRSA F32 program (Figure 1).

F32 Applications, Awards, and Success Rates

Figure 1. Number of NIGMS F32 Applications, Awards, and NIGMS and NIH Success Rates, FY 2015-2019. The blue bars (left axis) represent the number of F32 applications, while the orange-striped bars (left axis) represent the number of F32 awards. The solid gray line (right axis) represents the NIGMS success rate, defined as the percentage of reviewed grant applications that receive funding. For comparison to the NIGMS success rate, the NIH success rate is included as a yellow dashed line and indicates that NIGMS had higher success rates over the past 5 years than NIH overall. NIH-wide data are from the NIH Data Book.

Figure 1 shows the NIGMS F32 competing application submissions, awards, and award success rates. A comparison with NIH-wide data indicates that the NIGMS application numbers generally mirror the pattern of NIH-wide F32 applications and the NIGMS success rates are slightly higher than that of NIH overall. Note the decline in awards between FY 2017 and FY 2018 is a result of applications considered at our September 2017 advisory council meeting. These awards were funded ahead of schedule and therefore are reported in an earlier fiscal year.

Figure 2 illustrates yearly award rates by percentile score and indicates that applications at a percentile lower than 25 were most likely to be awarded. Funding decisions are based on multiple criteria and not exclusively on the application’s percentile, as indicated by the fact that some awards are made for applications into the 40th percentile. Additionally, some applicants with favorable review scores (below the 20th percentile) may have accepted other fellowship opportunities and declined the NIGMS F32 award.

F32 Award Rate

Figure 2. NIGMS F32 Award Rate, FY 2015-2019. Each line represents the fiscal year award rate as a function of percentile and shows that awards at percentiles lower than 25 were most likely to be awarded. Applicants who were offered an NIGMS F32 but declined it to accept another fellowship are counted as not having received an award.

Below, we provide the demographic characteristics of NRSA F32 applicants and awardees by race/ethnicity (Figure 3), gender (Figure 4), and age (Figure 5) over the same 5-year period. The numbers of applications and awards interactively displayed are rounded values and are not actual counts. The numbers were systematically rounded up or down to the nearest 10 (for small values) or 25 (for larger values) relative to the actual number. This representation of the data shows overarching trends while ensuring adequate masking of small sample sizes to protect the applicants’ privacy. In comparing award rates, we performed a Fisher’s exact test using funded and unfunded applications across all groups (excluding “Other” or “Not Reported”) to determine whether the differences in award rates across groups are unlikely due to random chance based on simulated alternative distributions. In some cases, making follow-up pairwise comparisons to identify potential drivers of overall differences observed.

Figure 3 shows that approximately 80% of applications and awards were among researchers who are white or Asian (well represented). The award/acceptance rates for each racial/ethnic group reveal differences across groups (p = 0.04) with the award/acceptance rate difference between white and Hispanic awardees driving this effect. A larger proportion of Hispanic applicants were among the well-scoring applicants who did not receive/accept an award than applicants from other racial and ethnic groups. Upon closer inspection, most of these well-scoring applicants declined awards in favor of fellowships from other funding organizations, which strongly contributed to the difference in apparent award rates for Hispanics. When combining race/ethnicity data into well-represented and underrepresented groups, we found award rates were higher for applicants from well-represented groups than those in underrepresented groups, at 29% and 22% respectively, primarily driven by the lower acceptance rate for Hispanic applicants, with marginal significance found on statistical testing (p = 0.06). Overall, there appears to be no statistical difference in the awarding rates, but there is a measurable difference in the acceptance rates for Hispanic applicants.

F32 Applications, Awards, and Award Rates by Race/Ethnicity and Representation

Figure 3. Percentages of NIGMS F32 Applications, Awards, and Award Rates by Race/Ethnicity and Representation, FY 2015-2019. The graph at the top left shows the relative percentage of F32 applications and awards by race/ethnic groups. The category “Other” includes individuals who are multiracial or who withheld their race/ethnicity. The graph at the top right shows the award rate by race/ethnic group. Note that the award rates for American Indian or Alaska Native and Pacific Islander groups are excluded because the numbers fell below the thresholds (n < 11) used at NIH to protect against disclosure in keeping with the Privacy Act of 1974. The graph at the bottom left shows the relative percentage of F32 applications and awards by well-represented and underrepresented groups. The well-represented group includes white and Asian, and the underrepresented group includes Hispanic, Black or African American, American Indian or Alaska Native, and Pacific Islander. The graph at the bottom right shows the award rate by representation status. Rounded application and award counts can be displayed by hovering over the graph.

Figure 4 shows that the gender distribution of applications (30% women, 60% men, 10% not reported) is similar to that of the NIGMS early-stage investigator Maximizing Investigators’ Research Awards (ESI MIRA) program (27-30% women, 70-73% men). This indicates that this gender gap is also reflected in the applicant pool of the early postdoctoral fellowship awards. The award rates were 30% for men and 28% for women (p = 0.47). This is in keeping with other studies that have shown that women and men have similar success rates in most award programs.

F32 Applications, Awards, and Award Rates by Gender

Figure 4. Percentage of NIGMS F32 Applications, Awards, and Award Rates by Gender, FY 2015-2019. The graph on the left side of the figure shows the relative percentage of F32 applications and awards by gender. The graph on the right side shows the award rate by gender (p = 0.47). The category “Not Reported” includes individuals who did not provide a gender identity. Rounded application and award counts can be displayed by hovering over the graph.

Lastly, Figure 5 shows that researchers under age 30 made up almost one-third of applicants and almost 40% of awardees, while researchers between ages 30 and 35 made up almost two-thirds of applicants and awardees, and researchers over 35 made up a small fraction of applicants and awardees. Award rates decreased as age increased, and a comparison of award rates across all three age groups detected statistically significant differences between groups (p < 0.001). As noted in our guidelines for postdoctoral fellowships, NIGMS considers length of time already spent in the sponsor’s laboratory as a factor in making award decisions. Postdoctoral applicants who have been in their sponsor’s laboratory for a long time without a strong justification for the need for this extended period of training are less competitive.

F32 Applications, Awards, and Award Rates by Age

Figure 5. Percentage of NIGMS F32 Applications, Awards, and Award Rates by Age, FY 2015-2019. The graph on the left shows the relative percentage of F32 applications and awards by age group. The graph on the right shows the award rate by age group. Rounded application and award counts can be displayed by hovering over the graph.

In addition to the demographic characteristics of NRSA F32 applicants and awardees, we also examined geographic representation, specifically by Institutional Development Award (IDeA) and non-IDeA states. States that are eligible for the IDeA program historically have had low levels of NIH funding, and the IDeA program supports centers and networks to strengthen institutional biomedical research capacity and infrastructure in these states. From FY 2015 to FY 2019, fewer than 2% of NRSA F32 applications were from individuals in IDeA states. The award rate was higher for individuals from non-IDeA than IDeA states, though the difference is not statistically significant (p = 0.07). While the F32 program is not part of IDeA, NIGMS is interested in promoting geographic diversity across its portfolio and encourages eligible applicants from IDeA states to apply to all of the Institute’s programs.

NIGMS K99/R00 Applications, Success/Award Rates, and Demographic Data – Past 5 Fiscal Years

The NIH Pathway to Independence Award (K99/R00) program facilitates timely transitions of outstanding postdoctoral researchers with research and/or clinical doctorate degrees from mentored, postdoctoral research positions to independent, tenure-track or equivalent faculty positions. The program provides support during this transition to help awardees launch competitive, independent research careers.

As is the case across NIH, the NIGMS K99/R00 program is smaller in overall numbers of applications and awards when compared to the NRSA F32 program (compare Figure 1 and Figure 6). The K99/R00 success rate increased from 16% in FY 2015 to 23% in FY 2018, before dropping to 18% in FY 2019. The total number of competing awards has been stable over the last 3 years at 20 awards, and the changes in success rate have been primarily driven by variation in number of applications.

K99 Applications, Awards, and Success Rate

Figure 6. Number of NIGMS K99 Applications, Awards, and NIGMS Success Rate, FY 2015-2019. The blue bar (left axis) represents the number of K99 applications. The orange-striped bar (left axis) represents the number of K99 awards. The gray line (right axis) represents the NIGMS success rate.

Unlike F32 applications, which are reviewed by the NIH Center for Scientific Review, K99 applications are reviewed by NIGMS special emphasis panels and receive only overall impact scores. Applications with overall impact scores lower than 20 were most likely to be awarded (Figure 7). Like the F32 program, funding decisions are based on multiple factors and not exclusively on overall impact score.

K99 Award Rate

Figure 7. NIGMS K99 Award Rate, FY 2015-2019. Each line represents the fiscal year award rate as a function of priority score and shows that applications with an overall impact score lower than 20 were most likely to be awarded.

Demographic trends of the K99 program (Figure 8) indicate that researchers from well-represented groups submitted approximately 80% of applications (approximately 50% from white and 30% from Asian). Men (approximately 60%) submitted a higher percentage than women (about 32%), and the largest share of applications, approximately 75%, came from researchers between ages 30 and 35.Applications from non-IDeA states far outnumbered applications from IDeA states (<1%). Comparisons of K99 award rates across demographic characteristics or geographic representation were not statistically significant, but the small number of applications from underrepresented groups and states makes it difficult to detect possible differences.

K99 Applications, Awards, and Award Rates by Demographic Categories

Figure 8. Percentage of NIGMS K99 Applications/Awards by Race/Ethnicity, Representation, Gender, and Age, FY 2015-2019. The graph at the top left shows the relative percentage of K99 applications by race/ethnic groups. The graph at the top right shows the relative percentage of applications from well-represented and underrepresented groups. The well-represented group includes white and Asian and the underrepresented group includes Hispanic, Black or African American, American Indian or Alaska Native, and Pacific Islander. Award count percentages are excluded for either of these demographic groupings, as there were fewer than 11 awardees in underrepresented groups. The graph at the bottom left shows the relative percentage of applications and awards by gender. The graph in the bottom right shows the relative percentage of applications and awards by age. Statistical tests comparing award rates did not reveal significant differences by these demographic characteristics. Rounded application and award counts can be displayed by hovering over the graph.

Reflections on the F32 and K99/R00 Data

The purpose of this and future data posts is to provide transparency for success rates and demographics of NIGMS’ funded training programs. The demographic data from this post underscore the need to 1.) increase diversity of the F32 and K99/R00 applicant pools, and 2.) take additional steps to ensure equity in the grant application and award decision-making process. We provide a few examples below of how NIGMS is addressing this.

NIH research award rate disparities have been shown to persist over time [PDF] due to a number of factors. One of the initiatives undertaken to address this is the Diversity Program Consortium (DPC), managed by NIGMS. One DPC component is the National Research Mentoring Network, which has developed evidence-informed mentoring, networking, and grant writing workshops for biomedical researchers from diverse backgrounds, such as those from underrepresented groups.

NIGMS supports many initiatives to enhance the diversity of the applicant pool. The Division for Research Capacity Building (DRCB) supports research, faculty development, research training, and research infrastructure improvements in states where levels of NIH research funding have historically been low. NIGMS’ TWD collaborates regularly with DRCB to reach out to individuals conducting research in IDeA states to encourage applications for TWD funding, including the F32 and K99/R00 programs.

NIGMS also supports research to understand and inform interventions that promote the research careers of individuals in biomedical sciences. The research contributes to the evidence base for effective, high-impact, scalable interventions that improve the success of individuals from diverse backgrounds pursuing independent biomedical research careers.

The data presented here confirm a trend showing that women’s overall representation in academic biomedical sciences declines between training and independent scientist career stages, including during the postdoctoral research phase, which is then reflected in their lower representation among NIH research grant applicants and awardees compared to men. In addition to our diversity enhancing programs and supporting research to understand and inform interventions that increase persistence in the biomedical research workforce, NIGMS contributes to the NIH Working Group on Women in Biomedical Careers, which strives to understand the barriers women face in career advancement in biomedical research and identify possible approaches that NIH can take to help overcome those barriers.

NIGMS has a long tradition of supporting diversity-enhancing programs from the community college level to the postdoctoral level; however, we recognize that both NIGMS and our institutional partners can do more to promote diversity in the biomedical workforce, particularly at critical junctures in researchers’ careers such as the postdoctoral and early career phase. NIGMS wants to work with institutions to achieve this goal and has already taken some steps to make additional progress. The mismatch between the increasing proportion of underrepresented biomedical PhD earners and comparable representation among biomedical faculty was one reason that NIGMS recently launched the Maximizing Opportunities for Scientific and Academic Independent Careers (MOSAIC) program. This program aims to facilitate the timely transition of promising researchers from diverse backgrounds from their mentored, postdoctoral research positions to independent, tenure-track or equivalent faculty positions at research-intensive institutions. To date, the MOSAIC program has already received more applications from Black and African American applicants than the total number received in the last 5 years of the NIGMS K99 program. By actively and regularly assessing our training portfolio, NIGMS will ensure that programs such as MOSAIC meet their objectives and make inroads to address this long-standing challenge in the biomedical workforce.