United States

Previous work has shown that item response theory may be used to rank incorrect response options to multiple-choice items on commonly used assessments. This work has shown that, when the correct response to each item is specified, a nominal response model (NRM) may be used to rank the incorrect options. We seek to determine the robustness of these results by applying the NRM to all response choices, without specifying the correct response. We apply these analyses to multiple data sets (each with more than 9,000 response sets), including pre-instruction and post-instruction responses. We find that the rankings generated without specifying the correct response are consistent with the previously published rankings for one data set; however, we find noticeable differences between rankings generated from different data sets. We provide evidence that discrepancies may result from differences in response rates for less commonly chosen responses.

PERC SM 21

Comparing item response theory models for ranking incorrect response options

introductory physics

item response theory

fmce

assessment

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

poster

This study is situated within possible selves theory to explore how senior undergraduate students in STEM develop and envision their future possible selves after graduation. We interviewed senior students at an urban institution in the US and encouraged them to think about who they will be in the future associated with their future career choices. This study presents two case studies: Ricky and Francisco. These two students discuss a variety of academic contexts through which they negotiate and explore career-related possible selves: conferences; experiences from classes and research; maintained aspirations for science; and dissatisfaction with education from high school. However, we found that in some cases academic contexts alone are not enough to help students develop possible future selves. Rather, they intersect with social contexts (social identity, out-of-school experiences) through which possible selves emerge. Our findings suggest that students might need more than just classroom-based lessons to explore their possible selves. Implications from this research can benefit the ongoing curriculum development aiming at increasing students' interest in STEM-related careers and building a meaningful professional life after graduation.

Senior undergraduate students developing and envisioning possible selves after graduation

In this study, we adapted a prior identity framework to investigate how students’ perception of learning environment (including sense of belonging, peer interaction and perceived recognition) in an introductory physics course predicts their course grades and physics motivational beliefs (including self-efficacy, interest and identity) at the end of this course. We found signatures of inequitable and non-inclusive learning environment in that female students’ mean scores for sense of belonging, peer interaction and perceived recognition were all lower than male students’ in the course. In addition, we found that female students had lower average course grades than male students. Using structural equation modeling, we found that students’ perception of the learning environment predicts their self-efficacy, interest, identity and grades at the end of the course. In particular, students’ perceived recognition, e.g., by instructors and teaching assistants, played a major role in predicting students’ physics identity, and students’ sense of belonging in physics played an important role in explaining the change in students’ physics self-efficacy. Our findings can be helpful for creating an inclusive and equitable learning environment in which all students can excel.

How learning environment predicts students’ physics grades and motivational beliefs in introductory physics courses

Physics is a degree that supports many career paths, and students often develop preferences for particular methods, such as theoretical, computational or experimental. However, it is not well understood how those preferences develop and affect students' later career decisions. We used Social Cognitive Career Theory (SCCT) as the basis for interpreting students' decision-making processes. SCCT provides a framework for connecting learning experiences, self-efficacy, and outcome expectations with students' interests, goals, and decisions. Semi-structured interviews with 8 physics students were conducted. This analysis focuses primarily on a single student to provide space to explore all three method specializations (theory, computation, and experiment) in more depth. We find that the availability of resources and learning opportunities had a significant impact on students' career choices. Theoretical and computational experiences were readily available through classwork, undergraduate research, and could be worked on at home and in peer study groups. Students lacked the ability to work on experimental physics outside of infrequent classroom opportunities and could not build peer networks that supported their experimental skill growth, which was linked to lower interest and self-efficacy in regards to experimental physics.

Access to opportunities affects physics majors interest and choice of methods specialization

Theory on teacher-student relationship development in higher education is scarce despite the empirical work linking positive relationships to important student outcomes. To advance research in this area, this study explores using Epistemic Network Analysis, a technique that characterizes patterns in data by visualizing them as connected graphs, to characterize patterns in social behaviors exhibited by Teaching Assistants in a large-enrollment algebra-based physics course for non-majors. We recorded classroom interactions between teaching assistants and students in synchronous remote active-learning problem-solving sessions and coded the interactions for the types of social behaviors exhibited by the Teaching Assistants (e.g., addressing students by name or empathizing with student difficulties) as they engage groups of 2-3 students in breakout rooms. Using a validated survey measuring a construct related to teacher-student relationship quality, we identified the two instructors with the highest-rated and lowest-rated scores and compared the connected graphs describing the patterns in their social practices. We present the graphs and generate hypotheses about which social behaviors may contribute most to building or damaging relationships between Teaching Assistants and students. Our results can guide future research in Teaching Assistant-student relationships in physics classrooms and have implications for training Teaching Assistants to build rapport with students.

Characterizing social behavior patterns in teaching assistant interactions with students

In this project, we sought to uncover the cognitive processes and skills that are involved in completing a theoretical physics project. Theoretical physics is often portrayed as a field requiring individual genius and can seem inaccessible to undergraduate students, as well as the public. We drew upon the foundations of Cognitive Task Analysis and completed semi-structured interviews with eleven theoretical physics faculty members from several different research institutions who specialized in subfields including quantum optics, biophysics, computational astrophysics, and string theory. We analyzed the processes and skills of these physicists, focusing on an analysis of idea origin, collaboration, and motivation. We used concept maps to organize these codes and portray the factors that influence the creation of project ideas. We found that motivation and collaboration are fundamental determinants of project ideas and their origins, which contradicts the “lone genius” stereotype. These findings on cognitive processes and skills can help us understand how to better prepare students to do theoretical physics research. Finally, the information gathered during this project may be useful for improving the public understanding of theoretical physics, dispelling the belief that the field requires “genius,” and making it accessible to more students. (Supported by NSF DGE-1846321 and REU-1757477)

When the light bulb turns on: motivation and collaboration spark the creation of ideas for theoretical physicists

Previous work has shown that many physics students will draw unprompted diagrams while solving physics problems. These diagrams are distinct from prompted student-generated diagrams in that they tend to be smaller and contain fewer details, an unsurprising finding given that these two types of diagrams are generated for different reasons. In this study, we expand and elaborate on reasons physics problem-solvers generate diagrams and the differences between unprompted and prompted diagrams by analyzing a set of 10 interviews with undergraduates, graduates, postdocs, and faculty in physics. Interviewees were given six physics problems and three diagramming tasks to complete, followed by a set of explicit questions about diagramming while problem solving. Participants reported that their unprompted diagrams help them: understand the situation and question; store, organize, and prioritize information; and support solution strategies; and communicate information. Participants also shared their beliefs about the limitations of diagrams, including when they are not useful or stop being useful, and difficulties with generating diagrams.

Physics Problem-Solvers on Why They Generate Unprompted Diagrams

Matrix algebra is a commonly applied mathematical tool in physics that can be used in classical mechanics, quantum mechanics, optics, and more. A recent study arising from the Research in Undergraduate Mathematics Education community suggests that students use three mathematical interpretations of matrix multiplication: linear combination, system of equations, and transformations [Larson & Zandieh, 2013], but there are no corresponding studies in PER. We have investigated student written responses [N=43] and student interviews [N=3] to examine how physics students interpret matrix equations using the three interpretations to classify their responses. In this poster, we evaluate both types of data with the primary emphasis on the student interviews. Preliminary results suggest that physics students prefer using the transformations interpretation, which corresponds to interpreting a matrix transforming, or ‘acting on’ a vector, giving a new vector.

Physics students and matrix multiplication: evaluating their interpretations

As a method to shorten the test time of the Force Concept Inventory (FCI), a previous study suggested the use of Computerized Adaptive Testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. As a step to develop a CAT–based version of the FCI (FCI-CAT), the previous study examined the optimal test length of the FCI-CAT such that accuracy and precision [which were measured in terms of root-mean-square error (RMSE)] of Cohen’s d would be comparable to that of the full FCI for a given class size. The objective of this paper is to address an issue in the previous study and, in so doing, to improve the FCI-CAT. We consider content balancing ensuring that the same set of concepts assessed in the original test is covered in the CAT administration for each respondent. To balance content in CAT, the percentage of items to be administered from each subgroup is defined in advance. Doing so ensures that items from each subgroup are administered. We conducted a Monte Carlo simulation to analyze how implementing an algorithm of content balancing affects the RMSE of Cohen’s d. As a result, we found that, for a class size of 40, the increase of the RMSE due to content balancing is 6%-7% for test lengths of 2-5 items and less than 1% if the test length is larger than 13 items. This result indicates that for a sufficiently large test length (say, larger than 13 items), content balancing does not compromise the accuracy and precision of the FCI-CAT. Hence, we recommend that the FCI-CAT incorporate content balancing, provided the test length is larger than 13 items.

Balancing content of computerized adaptive testing for the Force Concept Inventory

Education researchers often compare performance across race and gender on research-based assessments of physics knowledge to investigate the impacts of racism and sexism on physics student learning. These investigations' claims rely on research-based assessments providing reliable, unbiased measures of student knowledge across social identity groups. We used classical test theory and differential item functioning (DIF) analysis to examine whether the items on the Force Concept Inventory (FCI) provided unbiased data across social identifiers for race, gender, and their intersections. The data was accessed through the Learning About STEM Student Outcomes platform and included responses from 4,848 students posttests in 152 calculus-based introductory physics courses from 16 institutions. The results indicated that the majority of items (22) on the FCI were biased towards a group. These results point to the need for instrument validation to account for item bias and the identification or development of fair research-based assessments.

Bias on the Force Concept Inventory across the intersection of gender and race

Ethical training for scientists is not effective in improving scientists' ethical reasoning. In this study, we had fourteen scientists from several disciplines participate in a fellowship designed to have the participants discuss the roles both scientific and non-scientific values play in science. We conducted pre- and post-interviews with the participants about how they viewed ethics in their research and what they thought the appropriate action would be in vignettes posing ethical dilemmas about research conduct. We analyzed these interviews by focusing on the goals and values the faculty express in their ethical reasoning. Our results show that the values scientists' invoke did change when talking about ethics in their own research, but did not change when looking at vignettes. We discuss the implications of this finding for future work regarding the reason for these shifts in reasoning.

Premium content

Comparing item response theory models for ranking incorrect response options

Next from PERC SM 21

Senior undergraduate students developing and envisioning possible selves after graduation

Similar lecture

Balancing content of computerized adaptive testing for the Force Concept Inventory