United States

Interest plays a major role in STEM motivation. Our research is based in introductory biology classes for pre-health majors, where students start out with high levels of interest. The course is challenging, however, and we test interventions (based in expectancy-value theory [Eccles &amp; Wigfield, 2002]) to help underrepresented students perform better. Both competence beliefs and values are critically important, but we intervene on value, and test competence beliefs as moderators. Different outcomes reflect different motivational processes, and we use different measures to study students&#39; performance in gateway science classes and persistence in STEM. We know that these value-focused interventions can promote students&#39; performance and engagement in the targeted biology classes (Harackiewicz et al., 2016). We have now begun to analyze long-term follow-up data from biology students in order to shed light on intervention processes and inform the design of future interventions (Harackiewicz &amp; Priniski, 2018).

First, we conducted a two-year follow-up of a utility-value intervention (UVI) with 1039 students, examining subsequent course-taking and whether students had chosen biomedical majors. The original study (Harackiewicz et al., 2016) had found that the UVI improved course grades for all students, as well as for underrepresented students, by promoting engagement. In the follow-up (Hecht et al., 2019), students who were more confident that they could succeed in the course were more likely to persist in biomedical fields if they had received the UVI (a classic expectancy-value effect), and this effect was mediated by higher levels of personal relevance in students&#39; writing assignments (a linguistic measure). We also found that the UVI increased persistence through the original effects on course grades, for all students, and for underrepresented students in particular. In other words, engagement was more relevant for performance (with clear implications for persistence in the biomedical pipeline), but a focus on personal relevance (for confident students) was most relevant for academic choices over time. These data help shed light on the different motivational processes that underly the efficacy of the UVI for different groups of students.

Second, we conducted two large-scale interview studies (Rosenzweig et al., 2021, Rosenzweig et al., in press), to examine students&#39; long-term interests in biomedical fields throughout college and how that translated into different career trajectories. We identified 1193 students from the introductory biology courses who intended to pursue biomedical fields of study early in college, and then interviewed them at the time of college graduation about their future plans (a 2-3 year follow-up). If students had changed their career plans since beginning college, we examined their explanations for changing their plans, specifically looking at (1) whether students who left biomedical fields referenced competence beliefs or value and (2) whether they felt pushed out or became disenchanted with their original path, or attracted to another path (e.g., pulled away by interest in another field). As shown below (Figure 1), there were 196 students who left biomedical fields of study during college, 76 who remained in biomedical fields of study but did not intend to pursue biomedical career paths post-graduation, and 921 who remained in biomedical fields and planned to pursue biomedical career paths. However, among the 921 who remained in biomedical fields, 422 (46%) still changed their career plans. We therefore examined two kinds of change: leaving biomedical fields altogether (e.g., changing to pre-law), or changing within the pipeline (e.g., changing from pre-med to biology PhD). With respect to leaving the pipeline we found results consistent with previous research: underrepresented ethnic minority students (African American, Latinx, or Native American students in this sample) were more likely to leave the biomedical pipeline. A different pattern emerged for change within the pipeline: There were no differences as a function of race/ethnicity, but women were more likely to change plans than men.

We coded students interviews and found that among students who left biomedical fields of study altogether, 62.5% reported changing primarily due to disenchantment, referencing both low interest and low perceived competence in biomedical fields. However, 25.2% of students who left primarily changed plans because felt pulled towards alternative fields of study, with almost all of them referencing a growing interest in other fields. Among those who remained in biomedical fields but changed plans, 34.8% reported disenchantment with their original plan, whereas 49.5% reported attraction, with most students referencing changing plans due to value and interest. Results highlight the importance of value and interest in shaping students&#39; educational decision-making, underscoring the critical role of value-focused interventions in introductory biology courses. They also point to an important future direction for such interventions, focusing on the relative value of one career path or field of study compared to one another, as opposed to focusing only on value for one particular field.

Although it is not always easy to collect long-term follow up data from interventions, this data provides critical insights that can inform both theoretical understandings of motivation and more specific understandings of when and how to intervene to support students&#39; motivation in pursuing biomedical careers. The follow-up of the utility value intervention explores the different processes by which reflecting on value can shape diverse students&#39; learning outcomes and persistence in the field; the long-term interview data explores how students reflect on their own motivation in a way that can drive future intervention design.

13th Conference on Understanding Interventions 2021

Using Long-term Follow-up Data to Understand and Refine Utility-Value Interventions in the Biomedical Sciences

persistence

utility-value intervention

stem pipeline

The annual UI conference has grown from a meeting to a community of more than 6,000 researchers, program officers, evaluators, program directors, training faculty, and others within STEM, and now operates with a mission to mutually inform scholarship and practice, develop and enhance the community interested in educational interventions, and provide professional development for all STEM persons. UI held its first Virtual conference in December of 2020 in response to Covid-19 restrictions that forced the cancellation of the original March 2020 in person meeting. Born out of necessity, the virtual conference introduced a format that provided pre conference availability of symposia, Deeper Dives and Poster presentations. The 13th Annual Understanding Interventions Conference will continue with that design and we foresee future conferences adopting aspects of that format in reaching a virtual audience as well as those who attend the conferences in-person.

Attendees will experience plenaries, workshops, symposia, and poster presentations on hypothesis-based research on interventions and initiatives that broaden participation in STEM careers. The conference will continue to facilitate the dialogue among education, behavioral, and social science researchers, evaluators, and faculty in STEM fields who participate in intervention programs. Graduate students in these fields are strongly encouraged to attend. Workshops, sessions, and posters will communicate effective strategies from successful STEM intervention programs, present results from empirical research studies, and synthesize the translation of research findings into practice on STEM learning, undergraduate research, graduate education, and student retention.

The cost to attend the conference is $150. There are special rates for post-doctoral fellows and graduate students, as well as conference grants that can be provided for those with need. Registration for the conference is open, with additional information to come.

The annual UI conference has grown from a meeting to a community of more than 6,000 researchers, program officers, evaluators, program directors, training faculty, and others within STEM, and now operates with a mission to mutually inform scholarship and practice, develop and enhance the community interested in educational interventions, and provide professional development for all STEM persons. 

Interest plays a major role in STEM motivation. Our research is based in introductory biology classes for pre-health majors, where students start out with high levels of interest. The course is challenging, however, and we test interventions (based in expectancy-value theory [Eccles & Wigfield, 2002]) to help underrepresented students perform better. Both competence beliefs and values are critically important, but we intervene on value, and test competence beliefs as moderators. Different outcomes reflect different motivational processes, and we use different measures to study students' performance in gateway science classes and persistence in STEM. We know that these value-focused interventions can promote students' performance and engagement in the targeted biology classes (Harackiewicz et al., 2016). We have now begun to analyze long-term follow-up data from biology students in order to shed light on intervention processes and inform the design of future interventions (Harackiewicz & Priniski, 2018).

First, we conducted a two-year follow-up of a utility-value intervention (UVI) with 1039 students, examining subsequent course-taking and whether students had chosen biomedical majors. The original study (Harackiewicz et al., 2016) had found that the UVI improved course grades for all students, as well as for underrepresented students, by promoting engagement. In the follow-up (Hecht et al., 2019), students who were more confident that they could succeed in the course were more likely to persist in biomedical fields if they had received the UVI (a classic expectancy-value effect), and this effect was mediated by higher levels of personal relevance in students' writing assignments (a linguistic measure). We also found that the UVI increased persistence through the original effects on course grades, for all students, and for underrepresented students in particular. In other words, engagement was more relevant for performance (with clear implications for persistence in the biomedical pipeline), but a focus on personal relevance (for confident students) was most relevant for academic choices over time. These data help shed light on the different motivational processes that underly the efficacy of the UVI for different groups of students.

Second, we conducted two large-scale interview studies (Rosenzweig et al., 2021, Rosenzweig et al., in press), to examine students' long-term interests in biomedical fields throughout college and how that translated into different career trajectories. We identified 1193 students from the introductory biology courses who intended to pursue biomedical fields of study early in college, and then interviewed them at the time of college graduation about their future plans (a 2-3 year follow-up). If students had changed their career plans since beginning college, we examined their explanations for changing their plans, specifically looking at (1) whether students who left biomedical fields referenced competence beliefs or value and (2) whether they felt pushed out or became disenchanted with their original path, or attracted to another path (e.g., pulled away by interest in another field). As shown below (Figure 1), there were 196 students who left biomedical fields of study during college, 76 who remained in biomedical fields of study but did not intend to pursue biomedical career paths post-graduation, and 921 who remained in biomedical fields and planned to pursue biomedical career paths. However, among the 921 who remained in biomedical fields, 422 (46%) still changed their career plans. We therefore examined two kinds of change: leaving biomedical fields altogether (e.g., changing to pre-law), or changing within the pipeline (e.g., changing from pre-med to biology PhD). With respect to leaving the pipeline we found results consistent with previous research: underrepresented ethnic minority students (African American, Latinx, or Native American students in this sample) were more likely to leave the biomedical pipeline. A different pattern emerged for change within the pipeline: There were no differences as a function of race/ethnicity, but women were more likely to change plans than men.

We coded students interviews and found that among students who left biomedical fields of study altogether, 62.5% reported changing primarily due to disenchantment, referencing both low interest and low perceived competence in biomedical fields. However, 25.2% of students who left primarily changed plans because felt pulled towards alternative fields of study, with almost all of them referencing a growing interest in other fields. Among those who remained in biomedical fields but changed plans, 34.8% reported disenchantment with their original plan, whereas 49.5% reported attraction, with most students referencing changing plans due to value and interest. Results highlight the importance of value and interest in shaping students' educational decision-making, underscoring the critical role of value-focused interventions in introductory biology courses. They also point to an important future direction for such interventions, focusing on the relative value of one career path or field of study compared to one another, as opposed to focusing only on value for one particular field.

Although it is not always easy to collect long-term follow up data from interventions, this data provides critical insights that can inform both theoretical understandings of motivation and more specific understandings of when and how to intervene to support students' motivation in pursuing biomedical careers. The follow-up of the utility value intervention explores the different processes by which reflecting on value can shape diverse students' learning outcomes and persistence in the field; the long-term interview data explores how students reflect on their own motivation in a way that can drive future intervention design.

technical paper

Undergraduate research and international experiences are often described as high-impact educational practices beneficial for undergraduate student success and for supporting the development of science identity and intercultural competencies. While several studies have investigated the impact of undergraduate research on students from minoritized groups, fewer studies have focused on their engagement in global experiences, and fewer still have explored their engagement in international research experiences. Drawing on the theoretical frameworks of Science Identity, Social Cognitive Career Theory, and the Intercultural Competence Model, this present study explores the benefits of participating in an international research experience for minority undergraduate scientists. Using a qualitative case study methodology, we examined the evolution of students' science identity, research competencies, and intercultural competence after engaging in a three-month international research opportunity in France and Belgium. We found that after participating in international research, minority undergraduate scientists had: (1) Increased confidence in their science identity and abilities; (2) Gained and strengthened skills necessary to be a successful researcher, (3) Recognized the influence of international exposure on their growth personally and professionally, (4) Expressed how monumental this research opportunity is for all minority students to experience. Our findings suggest substantial benefits from an international research experience on the development of minority undergraduate scientists.

The Role of International Research Experiences in the Development of Minority Undergraduate Scientists

Hostos Community College (HCC) of The City University of New York (CUNY) recently created and was awarded the NSF S-STEM titled “Hostos Engineering Academic Talent” (HEAT) Scholarship Program. The HEAT Program trains a subset of student as part of the HCC Joint Dual Engineering Degree Program with The City College of New York’s Grove School of Engineering (CCNY’s GSoE) of CUNY. The HEAT Program offers low-income academically talented scholars financial support, and exposure to a variety of high-impact extracurricular activities that include: faculty and peer mentoring model; undergraduate research opportunities; STEM-related field trips; participation at professional STEM conferences; career orientation workshops; and other professional development opportunities. These activities are designed to increase efficacy, belonging and science identity, and are considered complementary to financial aid assistance. Financial aid by itself is insufficient for academic success for low-income academically talented college students.1,2 Here we report partial data as to the impact of the HEAT model on the first cohort of ten HEAT Scholars and five Associates who had benefitted from the program with the exception of direct financial support, as well as faculty and peer mentoring. To this end, we have administered a series of surveys3,4 targeting their perceived utility, program satisfaction, and assessment of non-cognitive and cognitive outcomes. Measures include academic self-efficacy for engineering courses, emotional engagement, identity as an engineer, expectancy for an engineering career, and cognitive engagement. These outcomes were assessed in HEAT Scholars, Associate HEAT Cohort and a control population of non-participating students taken from the HCC engineering general population. 

At the time the surveys were administered, six out of the ten scholars were conducting research. However, by the end of their first year of the program all were engaged in research. In addition, the scholars reported high levels of cognitive engagement, thinking about different ways to solve problems, and trying to connect current learning to previous knowledge. Most scholars participated in field trips to museums and engineering laboratories and took part in the 9th Hostos STEM Olympiad. As of today, eight out of the ten scholars have obtained their A.S. in Engineering from HCC; five have transferred to CCNY’s GSoE, and two to Cornell University. Due to the COVID-19 pandemic, one HEAT Scholar had to move out of New York City, but intends to continue pursuing a B.E. degree and thus, 90% of the scholars remain in engineering and 80% remain in the program. The United States is in need of an increased and diverse STEM workforce.5 HCC via its Joint Dual Engineering Degree Program with The City College of New York’s Grove School of Engineering of CUNY is contributing towards this endeavor.

Results of our study on non-cognitive impacts show that HEAT scholars have higher academic self-efficacy for engineering courses, feeling confident in the ability to learn the material, and to meet the challenge of performing well in engineering courses compared to the engineering population. HEAT Scholars also reported high levels of emotional engagement looking forward to engineering classes, enjoying learning new things about engineering, wanting to understand what is taught, and feeling good when in engineering classes. In addition, the Scholars reported high levels of cognitive engagement through the work in their engineering. Associate scholars scored significantly lower in all these measures comparable to that seen in the non-participating control. We consider this work exploratory as we look to confirm with a larger cohort size. This preliminary analysis of the HEAT program indicates its positive impact on the development of Scholars’ non-cognitive and cognitive outcomes due to their participation in the program’s activities. The scholars feel supported both morally and financially and appreciate that faculty and peer mentors are available at all times to encourage and advise.

Acknowledgments - We are grateful to the National Science Foundation (NSF S-STEM Award DUE-1833767), the HCC Office of Academic Affairs, our Hostos STEM students, our HEAT Program Assistant Luis Tejeda Ortiz for his significance contribution to HEAT, and the HCC and CCNY’s GSoE Financial Aid Offices, and the CUNY Research Foundation.

References
1. Smith, J. M., & Lucena, J. C. (2016). Invisible innovators: how low-income, first-generation students use their funds of knowledge to belong in engineering. Engineering Studies, 8, 1-26.
2. Patel, R. and Rudd, T. (2012). Can scholarships alone help students succeed? Lessons from two New York City Community Colleges. The Performance-Based Scholarship Demonstration. MDRC.
3. Campbell, T. A., & Campbell, D. E. (1997). Faculty/student mentor program: Effects on academic performance and retention. Research in Higher Education, 38(6), 727-742.
4. Paglis, L. L., Green, S. G. and Bauer, T. N. (2006). Does adviser mentoring add value? A longitudinal study of mentoring and doctoral student outcomes. Research in Higher Education, 47(4), 451-476.

Increasing HSI Community College Engineering Students Success: Non-Cognitive Outcome Relationships with Financial Support and Mentoring

This concurrent mixed methods study focused on the outcomes of the contribution of specific support structures to the development of targeted skills and the self-efficacy of under-represented students in Life Sciences at an urban research university with a strong commitment to diversity. According to the SCCT model (Lent et al, 1994), increases in self-efficacy are expected to have long-term impact on Scholars intent to persist in STEM careers.The Social Cognitive Career Theory (SCCT) model framed the design of the intervention and informed the study that asks: (1) Does: (a) scholarship support; (b) scholars' sense of self-efficacy in STEM programs; (c) outcome expectations for obtaining a STEM degree, and (d) experiences of support in college STEM contexts predict the STEM major student success? (2) What are the students' perceptions of the impact that their academic experiences, their sense of self-efficacy, and their outcome expectations for obtaining the STEM degrees have on their success in the program? 

Simple non-parametric statistical analysis was used to describe the characteristics of the members of various cohort groups. A survey with a retrospective pretest design collected end-of-year data from students in four domains: Science/STEM Identity, Lab Research Self-Efficacy, Scientific Communication Self-Efficacy, and Metacognitive Awareness (which had 8 subdomains). Results were analyzed using t-tests, and effect sizes were calculated. Their academic disposition and perceptions of their experiences were gathered using semi-structured individual interviews. 
Ninety-eight percent of the 60 students have been retained and/or graduated in a STEM discipline. Of the 25 students who have graduated with BS degrees, 19 have gone on to MD programs, PhD programs or still working in a STEM field. One student recently received an NSF Graduate Research Fellowship. Eight students continued in MS programs at the same institution and continue to receive financial and professional development support from S-STEM. On the quantitative survey, students reported growth to most or all items in each domain and subdomain, with the greatest growth reported in Scientific Communication Self-Efficacy, the students' perceptions of their ability to communicate effectively as scientists. The participants who completed the program participated in exit interviews over the past three years on the importance of the support the program provided. This support might have contributed to the growth in each domain and sub domain. The subthemes associated with "support" included personal relationships with mentors, boosts in confidence as STEM scholars, career guidance, preparation and practice for interviews, presentations, peer feedback, and financial support. 
The size of the cohort groups limited the kind of analysis that could be done and so the findings are not generalizable but instead represent a case of one institution that might be transferred or translated to others. The support provided the students moved beyond counselling and instead paid attention to practical taken for granted skills and meeting the financial needs of the students who could then focus more on their academic outcomes. The confidence gained from the personal and scholarly supports and mentorship that were an integral aspect of the intervention might have allowed for continued interest in a career in STEM field. The students were not only performing well academically but also gaining the knowledge about possibilities available to them. The increase in the kinds of supports offered should demonstrate to institutions serving underrepresented students, who have extreme financial needs, that providing them with not only funding but also hands-on mentoring that is not just individualistic, but collaborative, can go a long way in making a difference in ensuring growth in specific skills, increased self-efficacy, graduation and retention in STEM fields.

Advancing Careers of Upper Level Students in the Life Sciences: The Case of a Minority Serving Institution

Introduction. Effective instruction of Gauss’s Law is extremely important because it has many applications in electrostatics. Perhaps more importantly, Gauss’s Law is a major component of the curriculum for gateway courses in which students decide if physics is the right area of study for them. In general, novice students have difficulty understanding and properly applying Gauss’s Law, and those with weaker spatial reasoning skills may face additional challenges as they struggle to visualize the relevant electric field symmetries (Pepper et al., 2012). There is some evidence that, on average, women have weaker spatial reasoning skills than men. Likewise, access to certain types of early life educational experiences may also impact spatial reasoning skills later in life (Uttal et al., 2012). Therefore, supporting students learning about Gauss’s Law by tailoring instruction to support spatial thinking could have a positive impact on the retention of women and other underrepresented groups in STEM. The present study proposes a new pedagogical tool to support students that struggle to visualize electric fields. Namely, we created a coherent tutorial utilizing interactive three-dimensional electric field simulations to teach students about electric field strength and direction.

Methods. Starting from a previously published tutorial that approached the topic of electric field symmetry from a more traditional perspective (Li & Singh, 2017), we developed a simulation-augmented tutorial covering the same material. The traditional tutorial relied on students' ability to perform vector addition and to visualize the results to answer questions about vector fields. The simulation-augmented tutorial was designed to support spatial thinking using a “guess and check” approach. After attempting to visualize the three-dimensional electric fields arising from a variety of charge distributions, students were able to view and interact with mathematically accurate simulated representations of those vector fields to see for themselves whether their mental images matched the simulations. We assessed student learning using the pre- and post-test that accompanied the published tutorial. Prior to taking the tutorial, all participants in the present study completed a demographics questionnaire, as well as a spatial reasoning assessment, the Paper Folding Test (Ekstrom et al., 1976). We used this data to separate participants into two groups that were roughly the same in composition with respect to gender, prior physics experience, and spatial reasoning ability. One group took the original tutorial, while the other group took our simulation-augmented tutorial.
Results. We found that the simulation-augmented tutorial was more effective than the traditional tutorial. Students who took the tutorial with simulations displayed larger improvements from pre- to post-test than those who took the traditional tutorial, and the Cohen’s effect size was large (d = 0.98). A stepwise multivariate regression revealed that the strongest predictor of improvement from pre- to post-test was which version of the tutorial participants completed. We also found that there was a contribution from the interaction term between the tutorial version and the participants’ paper folding test score. However, the most surprising result was the contribution of the interaction term between spatial ability and gender. We decided to further investigate this interaction by plotting improvement as a function of paper folding test score for each gender-tutorial combination. The lines of best fit suggest that, for female students, having weaker spatial thinking skills negatively impacted their ability to learn from the traditional tutorial. This effect was not seen with the simulation-based tutorial, nor was there a negative impact for male students with weaker spatial skills, regardless of the tutorial version. This result suggests that simulation-augmented tutorials may be especially helpful for female students with weaker spatial skills.

Conclusions and Discussion. Our results indicate that using simulations to support students in their spatial thinking about electric fields can be extremely helpful to students as they learn to apply Gauss’s Law. Previous physics education research suggests that students struggle with Gauss’s Law problems because: electric fields are not intuitive, students have trouble understanding the three-dimensional nature of electric fields, and students have difficulty recognizing symmetries in electric fields (Adrian & Fuller, 1997; Pepper et al., 2012; Singh, 2006). The simulation-augmented tutorial directly addresses these issues, as the simulations: help develop intuition, show electric fields in three dimensions, and allow students to explore symmetries of the field. While interventions such as simulations have been implemented in other STEM domains that require strong mental manipulation, no research before has investigated implementing manipulable three-dimensional simulations in the instruction electric fields. The success of this spatially targeted tutorial provides a first glimpse into the power of using simulations to help students overcome challenges in learning electrostatics. In addition, the strong interaction between gender and spatial reasoning ability with respect to learning gains further suggests that providing spatially targeted instruction to students in introductory STEM courses may also help achieve higher levels of persistence for members of underrepresented groups. Based on these findings, we are developing an entire Gauss’s Law tutorial series that incorporates the use of simulations.

References. 
Adrian, B. W., & Fuller, R. G. (1997). A qualitative investigation of college students' conceptions of electric fields. AAPT, 16. 
Ekstrom, R. B., & Harman, H. H. (1976). Manual for kit of factor-referenced cognitive tests, 1976. Educational testing service.
Li, J., & Singh, C. (2017). Investigating and improving introductory physics students’ understanding of the electric field and superposition principle. European Journal of Physics, 38(5), 055702.
Pepper, R. E., Chasteen, S. V., Pollock, S. J., & Perkins, K. K. (2012). Observations on student difficulties with mathematics in upper-division electricity and magnetism. Physical Review Special Topics-Physics Education Research, 8(1), 010111.
Singh, C. (2006). Student understanding of symmetry and Gauss’s law of electricity. American journal of physics, 74(10), 923-936.
Uttal, D. H., & Cohen, C. A. (2012). Spatial thinking and STEM education: When, why, and how?. In Psychology of learning and motivation (Vol. 57, pp. 147-181). Academic Press.

Using Three-Dimensional Simulations to Support Spatial Thinking About Electric Fields

Students’ sense of belonging and academic self-efficacy are important factors in their college success. In this study, we examine the relationship between a first year intervention, NxtGEN, with STEM students and their reported sense of belonging and academic self-efficacy. Moreover, we explore a mechanism for explaining differences in sense of belonging and academic self-efficacy among NxtGEN participants and a comparison group. Specifically, we describe the characteristics of students’ academic and social support networks and ways these networks differ by participation in the intervention. This study relies on longitudinal survey data in which students listed people to whom they are connected in academic and social settings and provided details about the frequency and nexus of their connection to these people. Students also completed scales to measure sense of belonging and academic self-efficacy on this survey. We also surveyed a similar group of first year students who did not receive the NxtGEN intervention in order to better examine the effects of program participation. 

Using these data, we find that, on average, students who did not participate in NxtGEN had slightly larger academic and social networks in both the fall and winter terms of their first year. In addition, NxtGEN students in the spring term reported a higher sense of STEM self-efficacy but a lower sense of social belonging relative to their non-NxtGEN counterparts. Average NxtGEN student STEM self-efficacy scores remained stable across the fall and spring terms, while non-NxtGEN student STEM self-efficacy scores decreased over time. At the same time, scores for a sense of social belonging decreased over time for NxtGEN students while it increased for non-NxtGEN students.


Academic and Social Networks among First Year STEM Students

Introduction. The COVID-19 pandemic that began in Spring 2020 and the subsequent period of social unrest due to police brutality against Black Americans and racial injustice have posed profound challenges to college students and academic programs across the U.S. Summer research experiences (SREs) were challenged to quickly pivot research training curricula and activities to a virtual format while ramping up activities that would bolster support and safety networks for student trainees, many of whom were socially isolated, fearful of the health and safety of themselves and their loved ones, while worried about the uncertainty of their academic future. This study investigates the nature of the concerns and worries that undergraduate trainees experienced during the tumultuous summer of 2020 and what program factors, if any, were able to potentially mitigate the impact of COVID-19 and racial injustice on these concerns. Specifically, this study evaluates basic needs, mental health, and academic concerns in the context of a National Institutes of Health (NIH) funded SRE, examining how student concerns and their relationship to program factors vary by demographic and academic characteristics. 

Methods. Forty-five undergraduate trainees participated in a NIH-funded 8-week summer research training program in 2020 (64% female; 67% Under-Represented Minority [URM] students) and completed an online post-survey. Items assessed student perceptions of the virtual SRE, program activities, student experiences related to COVID-19 (Latkin & Dayton, 2020), and impacts of the racial strife and related social justice activism. Closed-ended, Likert scale-type items assessed students’ time management skills and sense of belonging. Exploratory factor analysis (EFA) established four outcome domains: students’ personal needs concerns (α=.90), stress and mental health concerns (α=.89), academic concerns (α=.83) and impacts of anti-Black racism (single item measure). Independent t-tests (correlations) assessed differences (relationships) of student concerns by demographics and key variables of interest. Due to the paucity of research on these topics and exploratory nature of our questions, an alpha of .10 was selected. 

Results. Intercorrelations revealed that the outcomes were moderately related, yet distinct (rs = .24, -.56). Students were somewhat concerned about their personal needs (M = 2.05, SD = 0.81), very concerned about their stress and mental health (M = 2.79, SD = 0.80), and extremely concerned about their academics (M = 3.90, SD = 0.88). In addition, anti-Black racism, and society’s response to it, impacted students to a moderate extent (M = 3.60, SD = 1.10). Students who received financial aid and transferred from a two-year institution had higher personal concerns than those who did not (ps = .001, .030, ds = 0.86, 0.71). Additionally, URM students indicated higher personal needs concerns than non-URM students (p = .093; d = 0.54). Mental health concerns differed by gender such that women reported greater mental health concerns than men (p = .098, d = 0.52). Biomedical majors had higher academic concerns than behavioral majors (p = .056; d = 0.58). Students indicating poverty experienced higher anti-Black racism impacts than those that did not (p = .099, d = 0.62). 

Students who felt stronger relationships with research group members indicated lower stress/mental health concerns and academic concerns (rs = -.29, -.45; ps = .089, .007 respectively). Additionally, positive working relationships with training director(s) were negatively related to academic concerns (r = -.32, p = .040) such that higher quality of relationship was related to lower academic concerns. Feeling part of the SRE community was inversely related to personal needs concerns, academic concerns, and anti-Black racism impacts (rs =-.29,-.50,-.27; ps = .059, .001, .068), and feeling connected to members of the SRE community was also inversely related to both personal needs and academic concerns (rs = -.25,-.55; ps = .097, <.001). With respect to time management, meeting deadlines was inversely related to all outcomes. Students who were highly likely to meet deadlines and hand work in on time had lower personal needs concerns, stress/mental health concerns, academic concerns, and anti-Black racism impacts (rs = -.43, -.31, -.57, -.30; ps = .003, .042, <.001, .045). Students who indicated they were able to balance responsibilities (e.g., research, classes, school, home, family, friends) also indicated lower stress/mental health concerns and academic concerns (rs = -.34, -.63; ps = .022, < .001). On the other hand, students with lower prioritization skills and self-awareness of study habits had higher academic concerns (rs = -.37, -.26; ps = .013, .084). 

Discussions and Conclusions. This study provides a snapshot of worries and concerns of undergraduate students who participated in a SRE during the summer of COVID-19 and racial unrest. As expected, SRE students experienced a wide range of concerns related to COVID and racial injustice, similar to students at other U.S institutions (e.g., Nelson et al., 2020). Results reinforce the need for academic programs like SREs to help mitigate student concerns across domains (basic needs, mental health and academic). SRE students who felt connected to their research groups and SRE program directors, as well as to the SRE community, fared better both personally and academically. While the cross-sectional nature renders the directional causality inconclusive, the associations underscore the importance of programs fostering a safe and supportive academic research environment to undergraduate trainees (e.g., Urlaub, 2020). As many programmatic modifications were made instantaneously in reaction to the societal factors such as the pandemic and social unrest, understanding the value of academic based support, as evidenced by this study, is critical. Future research in this area should focus on investigating the impact and effectiveness of specific program modifications.

Psychosocial Impacts of COVID-19 and Racial Injustice on Undergraduate Summer Research Experience Trainees

Promoting Academic Success of Economically Disadvantaged, STEM-Interested, First- and Second-Year Undergraduate Students via the ACCESS in STEM Program at University of Washington Tacoma 

Erica Cline1, Elin Bjӧrling2, Emily Cilli-Turner3, Joyce Dinglasan-Panlilio1, Jutta Heller1, Edward Kolodziej1, Annie Camey Kuo4, Marc Nahmani1, Amanda Sesko1, Mary Pat Wenderoth5, Ka Yee Yeung6

1Interdisciplinary Arts and Sciences, University of Washington Tacoma, Tacoma WA 98402, 
2Human Centered Design and Engineering, University of Washington, Seattle WA 98195, 3Mathematics, University of La Verne, La Verne CA, 91750, 
4Education, Stanford University, Stanford CA 94305, 
5Biology, University of Washington, Seattle WA 98195,
6Engineering and Technology, University of Washington Tacoma, Tacoma WA 98402

Email: ecline@uw.edu
Keywords: S-STEM, mentoring, course-based undergraduate research, learning communities

Introduction

We are facing a national challenge of low retention rates for STEM-interested students. At the University of Washington Tacoma (UWT), a public, predominantly undergraduate, minority-serving institution (Asian-American, Native American, Pacific Islander, AANAPISI), only 28% of high achieving (high school GPA>3.0), STEM-interested at entry, Pell-eligible, first-time-in-college (FTIC) students undergraduates have entered a STEM major by the beginning of their 2nd yr, and the proportion is significantly lower for PEERs (persons excluded from STEM due to ethnicity or race [Asai, 2020]) at only 16%, representing a substantial equity gap. To address this problem, we developed the Achieving Change in our Communities for Equity and Student Success (ACCESS) in STEM Program. Supported by an NSF S-STEM grant since 2018, the program supports low-income, STEM-interested students by providing focused mentoring, a living learning community, a course-based research experience in their first year, and scholarships in their first two years of college. Based on the Student Persistence model of Graham et al. (2013), we hypothesized that these interventions would increase retention, academic performance, and progress into and through STEM majors. Our approach builds upon existing research demonstrating the importance of early research experiences (Thiry et al., 2012) and intensive mentoring and community building, particularly in the context of AANAPISI institutions (Nguyen et al., 2018). 

Methods

Quarterly and cumulative GPA and enrollment status were obtained from registrar data for program participants (N=22) and a comparison group of students who met eligibility requirements (e.g., FTIC, STEM interest at entry, Pell-eligible, and high school GPA > 3.0) but did not participate in the program (N=96). We assessed course grades in individual STEM courses for program participants and for students from the comparison group who consented to be part of the research study (N=16) and were closely matched to participant demographics.

Results

In the first three years, the program supported 44 students. Of these, 59% were female, 74% First Gen, 8% dependents of veterans, and 33% identified as PEERs. In all, 90% held at least one minoritized identity in addition to low-income status. First year cumulative GPA was significantly higher for ACCESS scholars than the comparison group, at 3.44 vs. 3.07 (p=0.027); likewise, second year cumulative GPA was significantly higher at 3.57 for ACCESS scholars vs. 3.09 for the comparison group (p=0.032). At the start of their 2nd year, 35% of program participants had entered a STEM major, vs. 27% of other eligible students. The program did not have a significant effect on first year retention, which was 78% for program participants vs. 82% for other eligible students, but most of the participants who left UWT did so to pursue STEM degrees not available at UWT and were retained in college; in some cases, they reported that mentoring greatly helped them to pursue their new discipline.

Conclusions and Discussion

These results suggest that programs that provide scholarship support and intensive mentoring coupled with opportunities for a living learning community and early research experiences may promote academic performance and entry into STEM majors for economically disadvantaged, first- and second-year STEM-interested students. This may be of particular interest to other AANAPISI institutions, as there are relatively few published studies within this particular context (Nguyen et al. 2018). In future work, we will analyze student responses to on-going longitudinal surveys and focus group interviews to gain greater insight into how the program impacts our students. We will also explore how the ACCESS program contributes synergistically to ongoing institutional change initiatives on our campus promoting faculty development to build capacity for creating equitable and inclusive environments for our students.

Acknowledgments

This program was supported by NSF S-STEM, DUE, Award #1741595. KYY has equity interest in and is employed by Biodepot LLC. This has been reviewed and approved by UW in accordance with its policies governing outside work and financial conflicts of interest.

References

Asai D. (2020). Excluded. Journal of Microbiology & Biology Education, 21(2):1-2. 
Graham, M.J., Frederick, J., Byars-Winston, A., Hunter, A.-B., Handelsman, J. (2013). Increasing persistence of college students in STEM. Science, 341:1455–1456.
Nguyen, T., Nguyen, M.H., Nguyen, B.M.D., Gasman, M., & Conrad, C. (2018). From Marginalized to Validated: An In-depth Case Study of an Asian American, Native American and Pacific Islander Serving Institution. The Review of Higher Education 41(3), 327-363.
Thiry, H., Weston, T. J., Laursen, S. L., & Hunter, A. B. (2012). The benefits of multi-year research experiences: differences in novice and experienced students’ reported gains from undergraduate research. CBE—Life Sciences Education, 11(3), 260-272. 


Promoting Academic Success of Economically Disadvantaged, STEM-Interested, First- and Second-Year Undergraduate Students via the ACCESS in STEM Program at University of Washington Tacoma

Introduction: The proportions of African and Black and Hispanic and Latino Americans grew from 24.8 to 30.6% between the years 2000 and 2014 alone. This proportion is projected to increase further in the coming decades. Despite this population growth and urgent known and projected needs in biomedical research and healthcare workforce diversity, presently, under-represented students (UR) comprise only 11.5% of enrollees in PhD programs. Moreover, although UR medical school graduates comprised 11.8% of the total in 2010, alarmingly this already low percentage dropped to 10.3% in 2015. Lastly, of all MD-PhD program matriculants in 2017, only 9.9% were UR. To address this crisis, in 2016, University of Maryland Baltimore developed STAR-PREP (Science Training for Advancing Biomedical Research ((STAR)-PREP) to provide a year-long rigorous, immersive and supportive experience designed to build students in academic and research rigor and to both individually and holistically to engage in good emotional and mental health practices that will help them to thrive once they enroll in PhD and MD-PhD programs. 

Methods: STAR-PREP was built on a cohort model whose premise was that providing support and intense academic and research preparation, built on an individualized student development plan (IDP) would prepare STAR-PREP Scholars who would be highly competitive to rise to high quality graduate programs. UR participant scholars were diverse and were competitively recruited from geographically diverse US locations and institutions. Institutional and familial resources were varied as well and scholars were selected based on assessed passion for research and demonstration of research commitment and potential. Incoming Scholars each received an individualized development plan (IDP), prepared collaboratively between each Scholar and mentor and STAR-PREP Leadership. In addition to sharpening lab and research skills and experiences (75% of program time), Scholars interacted with near peers and diverse mentors and mentees across the UMB campus and engaged in rigorous graduate courses, critical thinking, writing and presentation workshops, and were both presenters and attenders in research conferences (25% of program time). These intentional and rigorous writing and communication modules prepared the students for communication in their presentations, publications and ensuing graduate school applications and interviews. Furthermore, STAR-PREP Scholars added to and benefitted from UMB’s innovatively developed pipeline and continuum and ecosystem of mentored research education programs and outreach activities on campus and with our partners, creating a strong sense of belonging. Moreover, outreach service as mentors in the pipelines and ecosystem and in the local community enhanced their sense of value in the form of giving back. 

Results: In just 4 full academic years, 14 of 18 (77.8%) of STAR-PREP scholars engaging in the program have ascended to enrollment in strong PhD and MD-PhD programs. One (1) remains in training and 3 (16.7%) ascended to MD and MD-MS programs. 

Conclusion: UM SOM’s I3 approach: intensity, immersion and integration which underpin its STAR-PREP program, demonstrates a highly effective training model and approach to address the urgent and growing need for UR biomedical scientists and physician-scientists. This success was further enhanced by the Scholars’ sense of belonging to UMB and its academic environment and, networking and service in the local and pipeline communities. 

References: 
National Institutes of Health data and statistics (https://www.edi.nih.gov/data/demographics) 

National Science Foundation (https://www.nsf.gov/statistics/) 
Acknowledgements Funding Sources 
NIH/NIGMS R25GM113262; PIs Carey and Antalis 
Grant Title: UMB Post baccalaureate Research Education Program (STAR-PREP) 

NIH/NCI P30CA134274 PI Kevin Cullen, MD Director Greenebaum Comprehensive Cancer Center 
Grant Title: UMB CURE Scholars Program. P30 supplement to Greenebaum Cancer Center Grant

STAR-PREP: An Innovative Holistic and Research-based Immersion Program to Increase Under-represented minority Representation in Science and Medicine


Introduction. Disparities in STEM undergraduate degree achievement has been a prevalent topic of concern in the current higher education landscape, particularly across race, gender and socio-economic status (Griffith, 2010; NCES, 2018). Despite shifts in population demographics, and an increase in the need for STEM workforce diversity to maintain the nation’s economic and technological prominence across the globe, minority student representation in STEM fields still falls far behind current population demographics (Eagan et al., 2013; Lane, 2016). In recent years, STEM enrichment programs have been implemented across higher education institutions in an effort to improve the student success outcomes of under-represented minority students in STEM (Carpi et al., 2017; Jackson & Winfield, 2014). Several studies have found positive outcomes associated with program participation, specifically in graduation, persistence, and math academic preparation across under-represented minority students (e.g., Chang et al., 2014; Lee & Harmon, 2013). Despite these promising results, few results investigate the effects of program participation after accounting for confounding variables that may cause bias in interpreting results (Gonyea & Miller, 2007). For this reason, this study aims to expand research on the long-term effects of participation in STEM intervention programs, in particular the University of Houston’s Scholar Enrichment Program (SEP) a program funded by the National Science Foundation as part of the broader Houston-Louis Stokes Alliance Minority Participation (H-LSAMP). Considering the significance of STEM intervention programs at increasing the completion and persistence rates of URM students enrolled in STEM fields, this study examined the extent to which participation in the UH-SEP program impacts the graduation and persistence rates of minority students after controlling for socio-demographic and pre-college baseline characteristics. The study posed the following questions:
1) To what degree does program participation in the UH-SEP program have an effect on degree completion and persistence?
2) To what degree does program participation in the UH-SEP program have an effect on long-term student dropout and persistence trends over time? 
Methods. This study examined the 2013 and 2014 SEP freshman student cohorts (n=102), as well as a matched sample of non-SEP freshmen students enrolled in STEM fields across the same cohort years (n = 1,459). Student records contained information concerning academic progress, major, GPA, and demographics, allowing the in-depth analysis of persistence, drop-out and graduation rates over time. 
The first step of the analysis involved propensity score matching techniques that matched similar students across a series of baseline covariates that are related to persistence in STEM fields, specifically pre-college standardized math scores, race, gender and major (Ashford et al., 2016; Murphy, Murphy, Gaughan, Hume & Moore; 2010). The resulting sample contained an equal number of SEP and non-SEP participants (n=131). The second step of the analysis involved logistic regression techniques to estimate program participation effects on graduation and persistence rates, using the propensity score matched sample. 
Finally, to estimate program participation effects on dropout and completion rates over-time, discrete-time survival analysis was conducted using a competing risks model to analyze patterns of student dropout while controlling for program participation and the competing risk of STEM persistence. 

Results. Tables 1 and 2 present results of the logistic regression models estimating the effect of program participation and retention in a STEM field. Logistic regression results demonstrated that participation in the UH-SEP program had a significant positive effect on graduation and retention in a STEM field. After controlling for estimated propensity scores, participants of the UH-SEP program were 3.46 times more likely to graduate and 2.55 times more likely to persist in a STEM field, compared to non-SEP students (p<0.05). 

Table 1. Logistic Regression: UH-SEP Participation Effects on Graduation 
Variables Matched Students (n = 238)
Exp (B) 95% C.I for Exp (B)
Estimated Propensity Scores 28.53 .33 2500.65
Program Participation (SEP1) 2.51* 1.08 5.82

Cumulative GPA 4.81** 1.86 12.47
Accumulated Credit Hours 1.06** 1.04 1.08
Model Summary
N 248
% of cases predicted correctly 86.6%
Nagelkerke R2 .707
-2 Log likelihood (df) 154.447
Source: Ghazzawi, D., Pattison, D.L., Horn, C., Hardy, J. and Brown, B. (2021). Impact of an intensive multi-disciplinary STEM enrichment program on underrepresented minority student success. Journal of Applied Research in Higher Education. https://doi.org/10.1108/JARHE-12-2020-0452

Table 2. Logistic Regression: UH-SEP Participation Effects on STEM Retention

Variables Matched Students (n = 238)
Exp (B) 95% C.I for Exp (B)
Estimated Propensity Scores 5.61 0.13 235.46
Program Participation (SEP1) 1.49 0.74 2.99
Cumulative GPA 2.07 0.96 4.46
Accumulated Credit Hours 1.05** 1.03 1.06
Model Summary
N 248
% of cases predicted correctly 80.9%
Nagelkerke R2 .577
-2 Log likelihood (df) 201.204
Source: Ghazzawi, D., Pattison, D.L., Horn, C., Hardy, J. and Brown, B. (2021). Impact of an intensive multi-disciplinary STEM enrichment program on underrepresented minority student success. Journal of Applied Research in Higher Education. https://doi.org/10.1108/JARHE-12-2020-0452
Results of the competing risks model indicated that compared to non-SEP students, SEP participants had a decreased likelihood of dropout and increased likelihood of persistence in a STEM field, evident through the significant sub-hazard ratios for dropout. Table 3 presents the findings of the competing risks regression model. 


Table 3. Competing Risks Regression: UH-SEP Program Participation

Variable Sub hazard Ratios SE 95% Confidence Interval
Program Participation 0.541* 0.129 0.337 0.866
Calculus I Year Taken (Reference: never taken) 
First Year 1.04 0.321 0.567 1.90
Second Year 0.97 0.331 0.503 1.90
Cumulative GPA 0.992 0.216 0.647 1.52
Total Credits Accumulated 0.952** 0.005 0.941 0.963
Source: Ghazzawi, D., Pattison, D., & Horn, C.L. (2020). Persistence of Under-represented minorities in STEM fields: Are summer bridge programs sufficient? Unpublished manuscript.

Conclusions and Discussion. Findings from this study highlight the significant role that STEM intervention programs play in addressing inadequate academic preparation and barriers to the success of under-represented minority students in STEM (Chang et al., 2014; Crisp et al., 2009). Results add to the existing body of literature concerning racial disparities in STEM degree achievement by presenting program participation effects after controlling for pre-college and socio-demographic variables significantly associated with success in a STEM field. In addition, results from this study demonstrate the long-term significant effects of intervention programs on reducing the likelihood of dropout and increasing persistence among minority students. These findings support continuous calls for intervention programs that specifically target minority students from disadvantaged backgrounds. Given that findings from this study were limited to data from a single institution, recommendations for future studies include conducting similar analyses on a broader set of institutions within the H-LSAMP alliance as well as similar national intervention programs to gain a more comprehensive view of intervention effects on minority student success. 

Acknowledgments. Funding for this work is provided by the National Science Foundation: NSF Award HRD-1911310, 1407736, and 0903948. Additional funding was provided by Halliburton, Conoco Phillips, private donors, and the University of Houston’s Provost’s Office. 
References. 

Eagan Jr, M. K., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., & Garibay, J. C. (2013). Making a difference in science education: the impact of undergraduate research programs. American educational research journal, 50(4), 683-713.
Carpi, A., Ronan, D. M., Falconer, H. M., & Lents, N. H. (2017). Cultivating minority scientists: Undergraduate research increases self‐efficacy and career ambitions for underrepresented students in STEM. Journal of Research in Science Teaching, 54(2), 169-194.
Chang, M. J., Sharkness, J., Hurtado, S., & Newman, C. B. (2014). What matters in college for retaining aspiring scientists and engineers from underrepresented racial groups. Journal of Research in Science Teaching, 51(5), 555-580.
Crisp, G., Nora, A., & Taggart, A. (2009). Student characteristics, pre-college, college, and environmental factors as predictors of majoring in and earning a STEM degree: An analysis of students attending a Hispanic serving institution.
Ghazzawi, D., Pattison, D.L., Horn, C., Hardy, J. and Brown, B. (2021). Impact of an intensive multi-disciplinary STEM enrichment program on underrepresented minority student success. Journal of Applied Research in Higher Education. https://doi.org/10.1108/JARHE-12-2020-0452
Ghazzawi, D., Pattison, D., & Horn, C.L. (2021). Persistence of Under-represented minorities in STEM fields: Are summer bridge programs sufficient? Unpublished manuscript.
Gonyea, R. M., & Miller, A. (2011). Clearing the AIR about the use of self‐reported gains in institutional research. New Directions for Institutional Research, 2011(150), 99-111.
Griffith, A. L. (2010). Persistence of women and minorities in STEM field majors: Is it the school that matters? Economics of Education Review, 29(6), 911-922.
Jackson, K. M., & Winfield, L. L. (2014). Realigning the crooked room: Spelman claims a space for African American women in STEM. Peer review: emerging trends and key debates in undergraduate education, 16(2), 9.
Lane, T. B. (2016). Beyond academic and social integration: Understanding the impact of a STEM enrichment program on the retention and degree attainment of underrepresented students. CBE—Life Sciences Education, 15(3), ar39.
Lee, D. M., & Harmon, K. (2013). The Meyerhoff Scholars Program: Changing Minds, Transforming a Campus. Metropolitan Universities, 24(2), 55-70.
National Center for Science and Engineering Statistics. 2018. “Science and Engineering Degrees, by Race and Ethnicity of Recipients.” https://www.nsf.gov/statistics/degreerecipients/.


Long-term Impacts of STEM Enrichment Programs on Under-represented Minority Students

The two-semester sequence of General Chemistry is a challenging course too many first-year students. One goal I have is for students to become better problem solvers, able to apply material they learn to synthesize solutions to new and challenging problems. To help develop problem solving skills I have restructured class time to allow for collaborative problem-solving sessions. I have also restructured weekly one-hour recitations associated with the course to include collaborative problem-solving sessions. I have trained a cohort of Learning Assistants (LAs) to help facilitate these in both lecture and recitations. Students who have successfully completed the General Chemistry sequence, return to the class to help as near-peer mentors. 
I want students to leave General Chemistry as better problem solvers, as this skill goes beyond just General Chemistry, this is a skill that will help students in any course or any career. During my first year of teaching, I observed that many students who struggle in General Chemistry, struggle because of the math-based problem solving. This was clear from the assessments that were done throughout the semester. This was also a compounding problem because students who struggled with math-based problem solving within the first semester of General Chemistry, but passed first semester, and struggled even more the subsequent semester. 
To help students become better problem solvers, and to help students solve math-based problems, I restructured class time to allow for mini-problem sessions. Sessions where students are presented with a problem and allowed to work on the problem with students in their area for 4-6 minutes. The LAs circulate through the room, with me, helping to facilitate the group work. Students are then polled on their response to the problem, and we review the problem with a methodical, step-by-step approach. 
Associated with the General Chemistry lecture is a weekly, one-hour recitation section that is run by a Teaching Assistant (TA). To give students extra practice with problem solving, I redesigned the entire structure of recitation. Recitation became a mandatory part of the course that counted toward the overall course grade. I designed structured problem sets for each weekly recitation section to reinforce the content that was covered in lecture. These problem sets were designed in the structure of Process Oriented Guided Inquiry Learning (POGIL) to help create a structure for group work. Each TA is given the same module for the week, ensuring that content is uniform across all recitation sections.

Downloads

Next from 13th Conference on Understanding Interventions 2021

The Role of International Research Experiences in the Development of Minority Undergraduate Scientists

Similar lecture

Using Long-term Follow-up Data to Understand and Refine Utility-Value Interventions in the Biomedical Sciences

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from 13th Conference on Understanding Interventions 2021

The Role of International Research Experiences in the Development of Minority Undergraduate Scientists

Similar lecture

Using Long-term Follow-up Data to Understand and Refine Utility-Value Interventions in the Biomedical Sciences

Downloads