Application, Review, Funding, and Demographic Trends for Maximizing Investigators’ Research Awards (MIRA): FY 2016-2018

1 comment

NIGMS has made MIRA awards to Established Investigators (EI) and Early-Stage Investigators (ESI) for three full Fiscal Years (FY). In this Feedback Loop post, we provide an analysis of application, review, funding, and demographic trends for the MIRA program.

For the first two rounds of EI MIRAs, eligibility was limited to well-funded NIGMS investigators: PIs with two or more NIGMS R01-equivalent awards or one NIGMS R01-equivalent award for >$400,000 in direct costs. For the FY 2018 EI competition and beyond, eligibility was expanded to include any investigator with a single PD/PI NIGMS R01-equivalent that is up for renewal. For the FY 2016 ESI MIRA competition, ESIs and New Investigators (NI) at the assistant professor or equivalent level were eligible, whereas eligibility was restricted to ESIs in subsequent rounds. As always, a PI can apply for an extension of ESI status for various life and career events, including childbirth.

Table 1 shows the number of awards made and associated success rates by fiscal year for EIs and ESIs.

Table 1. Number of MIRAs Funded by Fiscal Year and Cohort (Success Rate)
FY Established Investigators Early-Stage Investigators
2016 136 (76.0%) 93* (29.1%)
2017 95 (68.3%) 100 (28.8%)
2018 115 (50.2%) 112 (36.8%)
  *FY 2016 ESI MIRA includes 27 awards to Non-Early-Stage New Investigators.

Because the EI MIRA applications generally proposed continuations of research programs, we compared them to R01 renewal applications during the same time period. As shown in Table 2, the EI MIRA success rates are much higher than the success rates for R01 renewal applications.

Table 2. Number of NIGMS R01 Renewals and EI MIRAs (Success Rate)
FY R01 Renewals Established Investigator MIRA
2016 390 (40.0%) 136 (76.0%)
2017 332 (40.1%) 95 (68.3%)
2018 259 (44.1%) 115 (50.2%)

Overall, ESI MIRAs had a significantly higher success rate over the last 3 years (31.4%) compared to the ESI R01s (25.7%). The ESI MIRA success rate was also substantially higher than the success rate for all new (non-renewal) NIGMS R01s during the same time period.

Table 3. Number of NIGMS New R01s, ESI R01s, and ESI MIRAs (Success Rate)
FY New R01s: ESI only All New R01s ESI MIRAs
2016 83 (23.1%) 443 (18.1%) 93* (29.1%)
2017 113 (30.2%) 538 (21.3%) 100 (28.8%)
2018 79 (23.6%) 508 (20.8%) 112 (36.8%)
  *FY 2016 ESI MIRA includes 27 awards to Non-Early-Stage New Investigators.

Over the last 3 fiscal years, 1,890 investigators were eligible for the EI MIRA program, and 516 (27%) of those eligible chose to apply. For well-funded investigators who had >$400,000 in NIGMS direct costs for their research prior to applying for MIRA, their MIRA budgets were on average 12% lower than their previous level of NIGMS research support. In exchange, MIRA provides a longer funding period (5 years instead of 4), increased funding stability and scientific flexibility, and reduced administrative burden. This trade-off made it possible for the Institute to fund a larger number of PIs and enabled us to increase the budgets for most ESI MIRAs relative to what the PIs would have received with an R01 (median annual direct costs of $250,000 vs. $210,000, respectively). In addition, we are working to generally give established PIs with only one modular R01 who convert to MIRAs budgets of at least $250,000 in order to mitigate some of the losses to inflation that have occurred over the past 15 years. In this manner, the MIRA program has contributed to two key strategic objectives of NIGMS [PDF 702KB]: Investing in and sustaining a broad and diverse portfolio of highly meritorious research, and promoting the ability of investigators to pursue new research directions, novel scientific insights, and innovative ideas. More information on MIRA budgets can be found in our recent Feedback Loop post on factors to consider when applying for an EI MIRA and in the FAQs for the ESI and EI MIRA programs.

In addition to scientific diversity, NIGMS monitors other factors contributing to the breadth of its portfolio, including demographic and geographic diversity. Over all three rounds of MIRA, men submitted 71% of ESI applications and 77% of EI applications (among applicants whose gender was reported). The skewed gender distributions in ESI and EI applicants is, unfortunately, consistent with NIH-wide trends and faculty demographics (e.g., Hechtman et al., 2018 Link to external web site; Valantine et al., 2016 Link to external web site). Across all six MIRA cohorts, no significant differences were observed between men and women in the fraction of applications reviewed (some applications are not reviewed if the proposal is outside of the NIGMS mission, the investigator is ineligible, or the application is incomplete), the mean impact scores, or the success rates (Table 4). The fraction of applications discussed during review was also not significantly different in five of the six cohorts, although a significantly greater percentage of applications from women than men were discussed in the FY 2018 EI cohort (87% vs. 69%, p = 0.01).

Table 4. Review Process by Gender, ESI and EI MIRA
  2016 2017 2018 2016 2017 2018
Percentage of Applications Received: Men 73% 70% 71% 80% 75% 77%
Percentage of Applications Received: Women 27% 30% 29% 20% 25% 23%
Percentage of Received Applications Reviewed: Men 79% 85% 89% 99% 93% 91%
Percentage of Received Applications Reviewed: Women 79% 90% 85% 94% 97% 88%
P-value p = 1.0 p = 0.15 p = 0.46 p = 0.18 p = 0.45 p = 0.62
Percentage of Reviewed Applications Discussed: Men 54% 54% 54% 100% 100% 69%
Percentage of Reviewed Applications Discussed: Women 46% 54% 57% 100% 100% 87%
P-value p = 0.31 p = 1.0 p = 0.60 p = 1.0 p = 1.0 p = 0.01
Mean Impact Scores: Men 40.0 39.2 33.6 27.1 27.4 36.9
Mean Impact Scores: Women 39.2 39.2 32.3 28.1 29.1 34.9
P-value p = 0.73 p = 0.97 p = 0.56 p = 0.72 p = 0.49 p = 0.45
Percentage of Reviewed Applications Awarded: Men 30% 29% 38% 77% 66% 47%
Percentage of Reviewed Applications Awarded: Women 25% 30% 38% 73% 74% 62%
P-value p = 0.48 p = 1.0 p = 1.0 p = 0.65 p = 0.40 p = 0.08

Analysis of the review process as a function of race/ethnicity groups, on the other hand, did reveal some significant differences in review rates, average scores, and success rates. We focus first on White and Asian investigators, who comprised over 85% of MIRA applicants. Applications from White investigators were reviewed significantly more often than those from Asian investigators in all three ESI cohorts, as well as the combined ESI cohort, but not in either the individual or combined EI cohorts (Table 5; statistically significant values are highlighted in red). Applications proposing research outside the NIGMS mission accounted for a strong majority of the non-reviewed ESI applications while most of the remaining applications in this category came from investigators who were not eligible at the time they applied or from investigators who received other funding after applying for MIRA, thus becoming ineligible before the MIRA application was reviewed. The main driver of the differing review rates between Whites and Asians was the higher rate at which applications from Asian investigators were determined to be outside of the NIGMS mission. Because of this observation, in FY 2017 NIGMS began using a blinded initial assessment of whether an application for MIRA was within the Institute’s mission. This assessment was based on the application title and abstract, excluding the investigator’s name and institution, and was made independently by two program directors. Any recommendations to decline an application were then reviewed by the relevant NIGMS division director and NIGMS leadership using the full application, if necessary. Because the difference in review rates between White and Asian applicants persisted after these new procedures were instituted, beginning with the FY 2018 ESI cohort, the likelihood of an application’s relevance to the NIGMS mission was assessed initially by machine learning tools using only the titles, abstracts, and specific aims of funded NIH R01s as a training set, followed by a blinded process similar to that used in 2017. The results of the machine learning tools largely corresponded with subsequent human assessment. Despite these steps to mitigate potential bias in determining if an application falls within the mission of the Institute, there continued to be a significant difference in review rates between Whites and Asians in the FY 2018 ESI MIRA cohort, suggesting that factors other than bias are responsible for the disparity. We will continue monitoring these outcomes and using a similar multi-tiered process as described above in evaluating applications for relevance to NIGMS’ mission.

Table 5. Review Rate, Whites and Asians
  Whites Asians Fisher p-value Individual Cohorts Fisher p-value Combined Cohorts
ESI 2016 84.1 73 0.02 ESI combined: 0.00004
ESI 2017 89.9 80.2 0.02
ESI 2018 92.5 82.6 0.02
EI 2016 97.9 100 1.0 EI combined: 0.66
EI 2017 93.6 88.9 0.41
EI 2018 90.6 89.8 0.79

Some differences in average scores between White and Asian investigators were also observed. For the FY 2016 ESI MIRA cohort, Asians had a mean priority score 5.5 points higher (worse) than Whites, which was a statistically-significant difference (Table 6), although this disparity was not observed in the FY 2017 or 2018 cohorts nor in the combined FY 2016-2018 group. When the non-ESI NIs are excluded from the FY 2016 cohort, the mean score for Whites increases by half a point (38.3) whereas the mean score for Asians is unchanged (43.3), a difference that is no longer statistically significant (p = 0.11). Thus, some of the score

Table 6. Mean Scores, Whites and Asians
  Whites Asians Wilcoxon p-value Individual Cohorts Wilcoxon p-value Combined Cohorts
ESI 2016* 37.8 43.3 0.028 0.23
ESI 2017 40.0 38.1 0.44
ESI 2018 32.2 33.1 0.79
EI 2016 26.2 31.9 0.09 0.01
EI 2017 26.4 32.4 0.07
EI 2018 35.6 37.6 0.44
  *FY 2016 ESI MIRA includes 27 awards to Non-Early-Stage New Investigators.

disparity between White and Asian applicants in the FY 2016 cohort could have been driven by more senior NIs who were eligible in that round, but not in subsequent rounds, and were more competitive on average than ESIs because of their more advanced career stage. In all three of the EI MIRA cohorts, the mean score for Asian applicants was higher (2-6 points) than for White applicants. Although these differences did not reach statistical significance in any individual cohort, there is a significant difference for the combined FY 2016-2018 group.

Success rates largely tracked these scores differences, with a significant difference in the ESI FY 2016 cohort but not for any other individual or combined ESI cohort (Table 7). By contrast, success rates were higher for Whites than Asians in each individual EI MIRA FY cohort and, although the difference in the combined cohort was only marginally statistically significant, if future cohorts follow the same trend it would likely become more significant. We are working to understand the sources for these observed disparities, which have also been noted in studies of NIH-wide R01 funding (Ginther et al., 2016 Link to external web site).

Table 7. Percentage of Reviewed Applications Awarded, Whites and Asians
  Whites Asians Fisher p-value Individual Cohorts Fisher p-value Combined Cohorts
ESI 2016* 36.1% 18.5% 0.003 0.06
ESI 2017 25.8% 25.8% 1.0
ESI 2018 37.2% 36.8% 1.0
EI 2016 78.3% 72.4% 0.48 0.08
EI 2017 71.8% 58.3% 0.22
EI 2018 53.4% 45.5% 0.40
  *FY 2016 ESI MIRA includes 27 awards to Non-Early-Stage New Investigators, who were eligible for ESI MIRA in FY 2016 but not in subsequent years.

The remaining MIRA applications were received from investigators whose race/ethnicity is Hispanic; Black or African-American; American Indian or Alaska Native; and from investigators with multiple, unknown, or withheld race/ethnicity. Because of the small number of investigators in these groups, it is difficult to make meaningful comparisons at this stage, and privacy considerations prohibit presenting the data for individual groups and cohorts. Therefore, we present the data for Hispanic, Black or African-American, and American Indian or Alaska Native investigators collectively (as “underrepresented”) in comparison to the combination of White and Asian investigators (“well-represented”).

Well-represented investigators were reviewed more frequently (86% vs. 71%) than underrepresented investigators in the ESI FY 2017 cohort (Table 8). No statistically significant differences were found in the other five individual cohorts nor in either combined cohort.

Table 8. Percentage of Applications Reviewed, Well-Represented vs Underrepresented
  Applications Received, Well-represented Applications Received, Under- represented Applications Reviewed, Well-represented Applications Reviewed, Under- represented Wilcoxon p-value Individual Cohorts Wilcoxon p-value Combined Cohorts
ESI FY 2016 340 26 272 (80%) 22 (85%) 0.8 0.07
ESI FY 2017 314 34 271 (86%) 24 (71%) 0.02
ESI FY 2018 278 28 248 (89%) 22 (79%) 0.12
EI FY 2016 170 < 11 167 (98%) < 11 (> 80%) 0.13 0.24
EI FY 2017 137 < 11 127 (93%) < 11 (> 90%) 1
EI FY 2018 229 16 207 (90%) 14 (88%) 0.66

For ESI applicants, comparison of mean scores showed no significant differences between well-represented and underrepresented investigators in any individual FY cohort nor in the combined cohort (Table 9). For EI applicants, well-represented investigators scored significantly better than underrepresented investigators in the FY 2016, FY 2017, and combined cohorts (Table 9).

Table 9. Mean Scores, Well-Represented vs Underrepresented
  Mean Impact Score, Well-Represented Mean Impact Score, Underrepresented Wilcoxon p-value Individual Cohorts Wilcoxon p-value Combined Cohorts
ESI FY 2016 39.5 36.3 0.44 0.63
ESI FY 2017 39.3 37.9 0.65
ESI FY 2018 32.5 33.5 0.77
EI FY 2016 27.2 43.2 0.02 0.004
EI FY 2017 27.5 21.3 0.34
EI FY 2018 36.0 50.3 0.008

Turning to success rate, no statistically significant differences were found in any individual or combined FY cohorts for ESIs (Table 10). Consistent with the scores shown in Table 9, the success rate for well-represented EIs was significantly higher than for underrepresented EIs for the FY 2018 cohort (Table 10) and for the combined EI cohort (65% vs. 32%)

Table 10. Percentage of Reviewed Applications Awarded, Well-Represented vs Underrepresented
  Well-Represented Underrepresented Fisher p-value Individual Cohorts Fisher p-value Combined Cohorts
ESI FY 2016 30% 23% 0.63 0.59
ESI FY 2017 26% 38% 0.23
ESI FY 2018 37% 41% 0.82
EI FY 2016 77% 40% 0.09 0.003
EI FY 2017 69% 67% 1
EI FY 2018 52% 21% 0.05

A comparison of the mean ages of the ESI MIRA cohorts and the ESI R01 cohorts from the corresponding years is shown in Table 11. All three ESI MIRA cohorts have a lower average age than their matched ESI R01 cohorts. This difference is statistically significant for the FY 2017 cohort and for the combined cohort, but not for the FY 2016 and 2018 cohorts. This age difference is reflected in the mean ages of applicants for ESI MIRA and NIGMS ESI R01 grants in the FY 2017, FY 2018, and combined cohorts (Table 11) suggesting that the MIRA program is encouraging people to apply sooner than they would for an R01. In addition to encouraging investigators to apply earlier, the MIRA program also appears to be selecting people for earlier awards, as shown by the lower average age of ESI MIRA awardees than ESI MIRA applicants in the combined cohort. The overall decrease in average age observed for the ESI MIRA program relative to ESI R01 awardees represents a step toward an NIH-wide goal of reducing the average age at which investigators receive their first major award.

Table 11. Mean Age (Years) by Cohort
  ESI MIRA Applicants ESI R01 Applicants Fisher p-value Applicants ESI MIRA Awardees ESI R01 Awardees Fisher p-value Awardees
FY 2016* 38.8 38.8 0.82 37.5 38.3 0.17
FY 2017 37.6 38.7 0.00003 36.7 38.3 0.0009
FY 2018 37.8 39.0 0.0008 37.7 38.7 0.25
Combined 38.1 38.8 0.00001 37.3 38.4 0.00008
  *FY 2016 ESI MIRA includes 27 awards to Non-Early-Stage New Investigators, who were eligible for ESI MIRA in FY 2016 but not in subsequent years.

Finally, we examined the geographic distribution of MIRAs. The Institutional Development Award (IDeA) program is intended to build research capacity in states that have had historically low NIH funding levels. Figure 1 shows the number of ESI and EI MIRAs (FY 2016-2018) in each state, with IDeA states highlighted in blue (note: the Institute has already made some EI MIRA awards in FY 2019, but these are not reflected in the map or the data above). Of the 23 states and one territory in the IDeA program, 16 have at least one MIRA. Investigators in IDeA states comprised a much larger fraction of applicants to the ESI MIRA program (11%) than to the EI MIRA program (4%). In both groups, the IDeA state investigators’ success rate was not significantly different from non-IDeA state investigators. A total of 14 EIs and 27 ESIs in IDeA states received MIRAs over FY 2016-2018. Thus, investigators in all but 8 states have MIRAs.

This is a map of the United States and Puerto Rico. IDeA states are shown in one color and all other regions in another. Each state and Puerto Rico has a pie chart whose overall size indicates the number of MIRA awards in that region. Each pie chart shows the fraction of MIRA awards in that region going to Early-Stage Investigators compared to Established Investigators. The distribution of MIRA awards is comparable to recent trends in the distribution of R01 awards, with the most number of awards going to California (103 awards), New York (78 awards), and Massachusetts (52 awards) with all other states receiving 37 or fewer awards.

Figure 1: ESI vs. EI MIRA Awards by State or U.S. Territory

As MIRAs become an increasingly large fraction of our research portfolio (from 7.8% of R01-equivalents in FY 2016 to 16.0% in FY 2018), we will continue to monitor these trends and other outcomes measures to help evaluate the effectiveness of the program.

One Reply to “Application, Review, Funding, and Demographic Trends for Maximizing Investigators’ Research Awards (MIRA): FY 2016-2018”

  1. What does the new investigator status not apply anymore? It puts several new assistant professors and new associate professors in a bind because they don’t qualify for the MIRA due to their years past graduation and they can not apply for the established PI because they haven’t been awarded an R01. I think something is amiss here and needs to be rectified.Either make everyone qualified for the R35 or eliminate it.

Submit a Comment

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.