United States

**Introduction**&lt;br&gt;
Graduate bioscience programs in the United States are rapidly evolving to meet national priorities focusing on the holistic development and support for doctoral students aligned with rigorous evaluation (NASEM, 2018, AAMC 2021). Program evaluation centered on goals or objectives, coupled with visual representations such as logic models or program theory, can be a useful way to ensure that the appropriate measures are being taken. However, these approaches may be limited in unmasking “blind-spots” not linked to intended goals. 
&lt;br&gt;As part of measuring the effectiveness our prior evaluation protocols and instruments, we set out to (a) establish a general (or generalizable) set of priorities for doctoral life sciences graduate programs; (b) propose an aligned set of measures to capture data on the extent to which institutions meet those priorities; and, (c) identify gaps in our current evaluation protocols. In short, we wanted an objective approach to evaluate our evaluations.
&lt;br&gt;&lt;br&gt;
**Methods**&lt;br&gt;
In order to outline a generalizable list of inputs that could be evaluated, we reviewed reports and recommendations (“white papers”), as well as invitations for funding applications (“request for applications” or RFAs), from some of the largest and most influential organizations impacting graduate bioscience education and training. We reasoned that these documents (a) reflected an integration of many published research studies and other work emanating from bioscience, STEM, or (more broadly) graduate education, and, (b) would have transformative impact given their influence on leaders of graduate bioscience education training programs. In our analysis, we also included relevant “internal” documents from our own institution, to assess the similarities and disparities between recommendations stemming from “external” and “internal” documents. In this way, we used a document analysis for “stakeholder mapping,” a process which could be conducted through other methods such as interviews (Ramadoss D, Bolgioni AF et al., 2022).&lt;br&gt;
For each document, we identified priorities and interventions and proposed measures. We identified major themes across these documents, consolidated and tabulated information before developing a logic model for each theme (Holliday 2014); thus generating useful template logic models that could be customized by institutions.&lt;br&gt;
For themes encompassing student-centered training, we also outlined approaches to conduct training evaluation built on the framework of Kirkpatrick’s model and considered how our institutional efforts met ideal standards (Praslova 2010).
&lt;br&gt;&lt;br&gt;**Results**&lt;br&gt;
We identified six major organizations (e.g. the Council of Graduate Schools, CGS) relevant to our scope and evaluated their reports touching on STEM, bioscience, graduate and general education. We also included three additional reports worth considering - a conference proceeding and two institutional reports. In total, we surveyed 18 documents, with the largest number coming from CGS.&lt;br&gt;
Based on this analysis, we identified eight major themes (areas of focus) in graduate education. For this presentation, we focus on the single theme of “professional development” which was the largest&lt;br&gt;
By comparing these general evaluation metrics to our previously developed instruments, we identified areas worthy of expanding evaluation. For example, if it is a priority to de-stigmatize withdrawing from PhD programs, then our evaluations should ask about students’ perceived sense of this option, access to counseling on this, and the presence of administrative mechanisms to enable this (e.g. leaving with a terminal master’s degree).&lt;br&gt;&lt;br&gt;
**Discussion**&lt;br&gt;
The approach of identifying general(izable) goals and evaluation metrics bolsters our central administrative team’s capacity to support data reporting and training for multiple stakeholders and has yielded templates which may be useful for others conducting an internal analysis of their efforts and interventions to identify “blind spots.”

15th Conference on Understanding Interventions 2023 

Evaluation as Program Innovation Tool: Insights from a Post-Baccalaureate Research Education Program

The AAAI 2025 presidential panel on the future of AI research aims to help all AI stakeholders navigate the recent significant transformations in AI capabilities, as well as AI research methodologies, environments, and communities. It includes 17 chapters, each covering one topic related to AI research, and sketching its history, current trends and open challenges. The study has been conducted by 25 AI researchers and supported by 15 additional contributors and 475 respondents to a community survey.

The Future of AI Research - Webinar Series

**Introduction** 
Graduate bioscience programs in the United States are rapidly evolving to meet national priorities focusing on the holistic development and support for doctoral students aligned with rigorous evaluation (NASEM, 2018, AAMC 2021). Program evaluation centered on goals or objectives, coupled with visual representations such as logic models or program theory, can be a useful way to ensure that the appropriate measures are being taken. However, these approaches may be limited in unmasking “blind-spots” not linked to intended goals. 
 As part of measuring the effectiveness our prior evaluation protocols and instruments, we set out to (a) establish a general (or generalizable) set of priorities for doctoral life sciences graduate programs; (b) propose an aligned set of measures to capture data on the extent to which institutions meet those priorities; and, (c) identify gaps in our current evaluation protocols. In short, we wanted an objective approach to evaluate our evaluations.
 
**Methods** 
In order to outline a generalizable list of inputs that could be evaluated, we reviewed reports and recommendations (“white papers”), as well as invitations for funding applications (“request for applications” or RFAs), from some of the largest and most influential organizations impacting graduate bioscience education and training. We reasoned that these documents (a) reflected an integration of many published research studies and other work emanating from bioscience, STEM, or (more broadly) graduate education, and, (b) would have transformative impact given their influence on leaders of graduate bioscience education training programs. In our analysis, we also included relevant “internal” documents from our own institution, to assess the similarities and disparities between recommendations stemming from “external” and “internal” documents. In this way, we used a document analysis for “stakeholder mapping,” a process which could be conducted through other methods such as interviews (Ramadoss D, Bolgioni AF et al., 2022). 
For each document, we identified priorities and interventions and proposed measures. We identified major themes across these documents, consolidated and tabulated information before developing a logic model for each theme (Holliday 2014); thus generating useful template logic models that could be customized by institutions. 
For themes encompassing student-centered training, we also outlined approaches to conduct training evaluation built on the framework of Kirkpatrick’s model and considered how our institutional efforts met ideal standards (Praslova 2010).
 **Results** 
We identified six major organizations (e.g. the Council of Graduate Schools, CGS) relevant to our scope and evaluated their reports touching on STEM, bioscience, graduate and general education. We also included three additional reports worth considering - a conference proceeding and two institutional reports. In total, we surveyed 18 documents, with the largest number coming from CGS. 
Based on this analysis, we identified eight major themes (areas of focus) in graduate education. For this presentation, we focus on the single theme of “professional development” which was the largest 
By comparing these general evaluation metrics to our previously developed instruments, we identified areas worthy of expanding evaluation. For example, if it is a priority to de-stigmatize withdrawing from PhD programs, then our evaluations should ask about students’ perceived sense of this option, access to counseling on this, and the presence of administrative mechanisms to enable this (e.g. leaving with a terminal master’s degree). 
**Discussion** 
The approach of identifying general(izable) goals and evaluation metrics bolsters our central administrative team’s capacity to support data reporting and training for multiple stakeholders and has yielded templates which may be useful for others conducting an internal analysis of their efforts and interventions to identify “blind spots.”

symposium

### Welcome to the 2023 Understanding Interventions/NSF CREST Annual Conference
Welcome to our annual conference, a gathering where the knowledge of our scientific communities share what they have researched since we last met. If it is your first time attending, then thank you for stepping into our world! If you are a returning guest, then thank you for coming back! Whether you are attending for the first or the fourteenth, we treasure your trust in what we have to offer. We promise to make it worth your while.

As in years past, we have put together an energetic collection of seminars, deeper dives, speeches, posters, and workshops to whet your appetite for learning. 

We are delighted that you have chosen to attend the 2023 Annual Understanding Interventions (UI) Conference, in partnership with the NSF Centers for Research Excellence in Science and Technology (CREST) Program. 

### About the Conference 
This conference was established to facilitate dissemination and exchange of hypothesis-based research on interventions and initiatives that broaden participation in science and engineering research careers. The conference is designed to create a dialogue among behavioral/social science and education researchers, evaluators, and faculty in STEM (science, technology, engineering, and mathematics) fields who participate in intervention programs. Workshops, sessions, and posters will communicate effective strategies from successful STEM intervention programs, present results from empirical research studies, and synthesize the translation of research findings into practice on STEM learning, undergraduate research, graduate education, and student retention.

Thank you again, and welcome! 
**[Download Program Agenda](https://drive.google.com/file/d/1rTKTrWC2DOqnIF06po5xo4buAL3mPTqZ/view?usp=sharing)**

This conference was established to facilitate dissemination and exchange of hypothesis-based research on interventions and initiatives that broaden participation in science and engineering research careers. The conference is designed to create a dialogue among behavioral/social science and education researchers, evaluators, and faculty in STEM (science, technology, engineering, and mathematics) fields who participate in intervention programs.

**Introduction:** The well-being of academics at universities, including students, staff, and faculty is a topic of increasing concern. The lack of well-being may be attributed to the current academic context which does not consistently provide cues that affirm social inclusion to all members of the academic population (Estrada et al., 2018). The lack of cues that affirm the dignity of others has consequences and undermines a basic need that people are more likely to survive and experience well-being when feeling socially connected (Baumeister & Leary, 1995; Dunbar & Barrett, 2007). Further, literature shows that social connection can reduce experiences of stress in pressured environments such as academia (Holt-Lunstad et al., 2015; USPHS, 2023; Valtorta et al., 2016). This study examines the role of kindness (defined as actions that affirm dignity and social inclusion) in promoting connectivity and well-being among academics. 
**Methods:** The Social Influence of Kindness Study was conducted through a one-time anonymous 10-minute online survey via Qualtrics. Participants were recruited using the snowball method via email and social media based on interest in taking a 10-minute survey about Kindness and Dignity during Spring 2020. For this study, 182 participants were selected for analysis based on their occupation designation of studying or working within an academic institution. Each survey included a written reflective micro-narrative in response to the prompt "recall an experience of kindness" and consisted of measures of Kindness Given, Kindness Received, Institutional Identity, Stress, and Well-Being. 
**Results:** We found engaging in acts of kindness, as measured by two novel psychometric rating scales Kindness Received (alpha=.927) and Kindness Given (alpha=.859), was significantly associated with increased institutional identity, well-being, and reduced stress. Our structural equation model (SEM) further validated the correlation results with receiving kindness. Interestingly through the SEM we found conducting acts of kindness had a significant negative relationship with institutional identity and reduced stress. Qualitative analysis of micronarratives on kindness showed that feeling safe and being acknowledged are the most common ways people describe experiencing kindness, an affirmation of their dignity. 
**Conclusions and Discussion:** This study shows that receiving kindness among academics has the potential to improve institutional identity, well-being, and reduce stress. Further, the results show that receiving kindness and giving kindness can be reliably measured amongst students and faculty in academia. We provide evidence that reflecting on experiences of kindness promotes positivity, with the potential to encourage people to conduct further acts of kindness. Together these findings provide knowledge that can inform workshops and interventions to improve social connection and inclusion amongst diverse academics. This study contributes to broader research highlighting positive contextual factors that promote the persistence of diverse students and faculty in academia. 
**Acknowledgments:** The project was funded by NIH grant #R01GM138700

The Influence of Kindness on Academic Well-Being

Engaging undergraduate students in biomedical research has been shown to improve retention rates, promote student engagement and increase the likelihood of pursuing science degrees (Gregerman 1999, Ishiyama 2001, Jones et al. 2010, Fechheimer, et al. 2011). As a pilot intervention strategy to support undergraduate students, especially rural and Alaska Native students who are pursuing biomedical science research and career trajectories, we have developed a unique, mid-level Research, Advising, and Mentoring Professional (RAMP) position. RAMPs work with their mentees to create an individual development plan, assist in comprehensive advising, help with mentee placement in research laboratories, provide psychosocial support, and meet regularly to monitor progress and troubleshoot problems. Survey and focus group data from students mentored directly by RAMPs indicated that the comprehensive, holistic approach has had a positive impact on student experiences by supporting persistence in degree programs and providing psychosocial support of both personal and professional development. Students also indicated that RAMP mentoring was helpful in connecting with research projects of interest, a valuable source for academic advising, and important in providing guidance in academic as well as non-academic challenges (Gildehaus et al. 2019). RAMP mentoring is likely to increase underrepresented student success in research and in the retention and completion of their undergraduate biomedical academic programs. Data from recently conducted focus groups with students mentored by RAMPs throughout the pandemic will be presented.

Research, Advising, and Mentoring Professionals: A novel position for mentoring undergraduate students in STEM research

The Centers of Research Excellence in Science and Technology (CREST) program provides support to enhance the research capabilities of minority-serving institutions (MSI). Southern University and A&M College in Baton Rouge (SUBR) established its Next Generation Composites CREST Center (NextGenC3) by a unifying theme of multiscale and multifunctional composites and related technology development of advanced fiber reinforced polymer composites with capabilities to address varying technological needs and challenges in material applications. Dr. Jahan in this colloquia session will provide an introduction and overview of her projects focusing on food wastes and byproducts. Her team at SUBR provides educational and research training opportunities to traditionally underrepresented minority (URM) students in Science, Technology, Engineering, and Mathematics (STEM) disciplines by using enhanced non-structural functionalities of the bio-composites towards a potential replacement for their inorganic-based counterparts in automotive, aerospace industries, and researchable fuel cell applications.

Potential Multifunctional Composite using carbonized coconut shell particles

Expanding and Scaling Research Opportunities for Computing Students at HSIs

How do we measure evolving collaborative relationships, and their impact on systemic change? The National Science Foundation-funded Computing Alliance of Hispanic-Serving Institutions (CAHSI) Eddie Bernice Johnson INCLUDES community uses the collective impact framework to bring together stakeholders across sectors to tackle the problem of the underrepresentation of Hispanics in computing. This deep dive will share practical lessons learned to guide discussion with attendees who may be interested in developing mechanisms to measure social networks and the way they support change efforts.

Evaluating Collective Impact- Focusing on Social Networks for Developing Inclusive Excellence

HBCU Digital Learning Infrastructure Initiative

Opportunities for Investments in Digital Learning Infrastructure

"Introduction. Undergraduate research experiences (UREs) are increasingly valued as high impact experiential educational activities that have broad and long-standing positive impacts on college students (Kuh, 2008). UREs are beneficial for undergraduate students since they increase students’ research self-efficacy, science identity and impact graduate school enrollment (Carpi et al., 2016). In this study, we look at how faculty perceived benefits and barriers may mediate the effect of funding on undergraduate and graduate research mentees.

Methods. Faculty included in this study are biomedical researchers at a large Hispanic-serving institution that answered a survey in fall of 2015 and 2017. The surveys included information about faculty characteristics and number of student mentees and duration of mentoring relationships as well as information about the students. A set of questions also gathered faculty responses on a set of items stating research-based barriers and benefits to research mentoring. We identified three benefits and two barriers to engaging undergraduates in research among research active biomedical faculty (Monarrez et al, 2020). The factors influencing the number of undergraduate and graduate students mentored in research are assessed using causal models incorporating counterfactuals. For all results, the number of undergraduate or graduate students mentored has ordinal categories 0, 1-4, 5-9, 10+. This allows for use of a quasi-Poisson link for the causal model.

Results. Undergraduate students. There is evidence that having any level of funding versus no funding increases the odds of more undergraduate mentees (OR(less than 50K)=1.11, p-value< 0.001; OR(50K - 150K)=1.06, p-value< 0.001; OR(more than 150K)=1.07, p-value< 0.001). With regard to faculty characteristics, white faculty are more likely than Hispanic faculty to mentor undergraduate students (OR(white)=1.19, p-value< 0.001) and males are also more likely to mentor more students than females (OR(male)=1.06, p-value< 0.001). Being an associate or full professor decreases the odds of mentoring undergraduate students as well (OR(Assoc)=0.88, p-value< 0.001; OR(Full)=0.87, p-value< 0.001). Finally, being in the college of liberal arts (OR(LA)=1.13, p-value< 0.001) or college of science (OR(SCI)=1.28, p-value< 0.001) vs engineering increases the odds of mentoring more undergraduate students.

Graduate students. There is a different pattern on mentoring graduate students than undergraduate students. In general, higher levels of funding results in greater odds of additional graduate student mentees in the natural direct effect (OR(50K – 150K)=1.09, p-value< 0.001; OR(more than 150K)=1.16 p-value< 0.001). White faculty tend to mentor more graduate students (OR(white)=1.13, p-value< 0.001) and females do as well (OR(males)=0.95, p-value< 0.001). Finally, being a full professor increases odds of more graduate mentees (OR(full)=1.06, p-value< 0.001) and College of Engineering faculty have higher odds of more graduate mentees than any other college.

Conclusion and Discussions. Overall, we see that funding levels affect the number of students (undergraduate and graduate) mentored and that the self-reported perceptions about undergraduate mentoring do not strongly affect the number of mentees at either the undergraduate or graduate level. Thus, we found no to little effect of faculty perception on actual number of research mentees. However, for both models, we do identify faculty characteristics, related to the benefits and barriers that affect the number of research mentees. The evidence provided in this study takes one step further to suggest that even small amounts of funding can have consistent and uniform effect on enabling biomedical research faculty to engage in and support undergraduate research mentees in projects. This is a practical issue and could be due to the higher salaries and time investment required for mentoring graduate students. Otherwise, both models (for undergraduate and graduate students) indicate that faculty-level characteristics do affect number of mentored students. Our results sheds light on the importance of funding support in interventions related to undergraduate research experiences.

Acknowledgments. Research reported in this paper was supported by the National Institute of General Medical Sciences of the National Institutes of Health under linked Award Numbers RL5GM118969, TL4GM118971, and UL1GM118970."

Premium content

Downloads

Next from 15th Conference on Understanding Interventions 2023

The Influence of Kindness on Academic Well-Being

Similar lecture

AI Evaluation