United States

Conceptual inventories such as the FCI are widely used to assess the impact of course reform efforts in physics education. Gains on these instruments, however, are known to correlate with other student variables such as students’ SAT score. I make use of this type of underlying correlation to communicate the impact of course reforms to broad audiences outside of physics education for which the FCI is not a useful benchmark. To illustrate this approach, I present an analysis of FCI learning gains that are disaggregated by ACT score before and after a major course reform in an introductory physics course. In this sample, normalized learning gains shifted from 0.28 to 0.46 were found to be equivalent to a shift in average ACT score from 24 to 29. Implications for the communicating research and evaluation results with different audiences are discussed.

PERC SM 21

Making use of FCI–ACT Correlations to Communicate the Impact of Course Reform

course reform

assessment

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

poster

In this project, we sought to uncover the cognitive processes and skills that are involved in completing a theoretical physics project. Theoretical physics is often portrayed as a field requiring individual genius and can seem inaccessible to undergraduate students, as well as the public. We drew upon the foundations of Cognitive Task Analysis and completed semi-structured interviews with eleven theoretical physics faculty members from several different research institutions who specialized in subfields including quantum optics, biophysics, computational astrophysics, and string theory. We analyzed the processes and skills of these physicists, focusing on an analysis of idea origin, collaboration, and motivation. We used concept maps to organize these codes and portray the factors that influence the creation of project ideas. We found that motivation and collaboration are fundamental determinants of project ideas and their origins, which contradicts the “lone genius” stereotype. These findings on cognitive processes and skills can help us understand how to better prepare students to do theoretical physics research. Finally, the information gathered during this project may be useful for improving the public understanding of theoretical physics, dispelling the belief that the field requires “genius,” and making it accessible to more students. (Supported by NSF DGE-1846321 and REU-1757477)

When the light bulb turns on: motivation and collaboration spark the creation of ideas for theoretical physicists

Previous work has shown that many physics students will draw unprompted diagrams while solving physics problems. These diagrams are distinct from prompted student-generated diagrams in that they tend to be smaller and contain fewer details, an unsurprising finding given that these two types of diagrams are generated for different reasons. In this study, we expand and elaborate on reasons physics problem-solvers generate diagrams and the differences between unprompted and prompted diagrams by analyzing a set of 10 interviews with undergraduates, graduates, postdocs, and faculty in physics. Interviewees were given six physics problems and three diagramming tasks to complete, followed by a set of explicit questions about diagramming while problem solving. Participants reported that their unprompted diagrams help them: understand the situation and question; store, organize, and prioritize information; and support solution strategies; and communicate information. Participants also shared their beliefs about the limitations of diagrams, including when they are not useful or stop being useful, and difficulties with generating diagrams.

Physics Problem-Solvers on Why They Generate Unprompted Diagrams

Matrix algebra is a commonly applied mathematical tool in physics that can be used in classical mechanics, quantum mechanics, optics, and more. A recent study arising from the Research in Undergraduate Mathematics Education community suggests that students use three mathematical interpretations of matrix multiplication: linear combination, system of equations, and transformations [Larson & Zandieh, 2013], but there are no corresponding studies in PER. We have investigated student written responses [N=43] and student interviews [N=3] to examine how physics students interpret matrix equations using the three interpretations to classify their responses. In this poster, we evaluate both types of data with the primary emphasis on the student interviews. Preliminary results suggest that physics students prefer using the transformations interpretation, which corresponds to interpreting a matrix transforming, or ‘acting on’ a vector, giving a new vector.

Physics students and matrix multiplication: evaluating their interpretations

As a method to shorten the test time of the Force Concept Inventory (FCI), a previous study suggested the use of Computerized Adaptive Testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. As a step to develop a CAT–based version of the FCI (FCI-CAT), the previous study examined the optimal test length of the FCI-CAT such that accuracy and precision [which were measured in terms of root-mean-square error (RMSE)] of Cohen’s d would be comparable to that of the full FCI for a given class size. The objective of this paper is to address an issue in the previous study and, in so doing, to improve the FCI-CAT. We consider content balancing ensuring that the same set of concepts assessed in the original test is covered in the CAT administration for each respondent. To balance content in CAT, the percentage of items to be administered from each subgroup is defined in advance. Doing so ensures that items from each subgroup are administered. We conducted a Monte Carlo simulation to analyze how implementing an algorithm of content balancing affects the RMSE of Cohen’s d. As a result, we found that, for a class size of 40, the increase of the RMSE due to content balancing is 6%-7% for test lengths of 2-5 items and less than 1% if the test length is larger than 13 items. This result indicates that for a sufficiently large test length (say, larger than 13 items), content balancing does not compromise the accuracy and precision of the FCI-CAT. Hence, we recommend that the FCI-CAT incorporate content balancing, provided the test length is larger than 13 items.

Balancing content of computerized adaptive testing for the Force Concept Inventory

Education researchers often compare performance across race and gender on research-based assessments of physics knowledge to investigate the impacts of racism and sexism on physics student learning. These investigations' claims rely on research-based assessments providing reliable, unbiased measures of student knowledge across social identity groups. We used classical test theory and differential item functioning (DIF) analysis to examine whether the items on the Force Concept Inventory (FCI) provided unbiased data across social identifiers for race, gender, and their intersections. The data was accessed through the Learning About STEM Student Outcomes platform and included responses from 4,848 students posttests in 152 calculus-based introductory physics courses from 16 institutions. The results indicated that the majority of items (22) on the FCI were biased towards a group. These results point to the need for instrument validation to account for item bias and the identification or development of fair research-based assessments.

Bias on the Force Concept Inventory across the intersection of gender and race

Previous work has shown that item response theory may be used to rank incorrect response options to multiple-choice items on commonly used assessments. This work has shown that, when the correct response to each item is specified, a nominal response model (NRM) may be used to rank the incorrect options. We seek to determine the robustness of these results by applying the NRM to all response choices, without specifying the correct response. We apply these analyses to multiple data sets (each with more than 9,000 response sets), including pre-instruction and post-instruction responses. We find that the rankings generated without specifying the correct response are consistent with the previously published rankings for one data set; however, we find noticeable differences between rankings generated from different data sets. We provide evidence that discrepancies may result from differences in response rates for less commonly chosen responses.

Comparing item response theory models for ranking incorrect response options

Many researchers find advantages of assigning and supporting open inquiries in physics lab education. However, objective assessment becomes more difficult as students gain more control over their own experiments because tasks are not described in operational detail. The Assessment Rubric for Physics Inquiry (ARPI) is designed to determine students’ attainment level of inquiry knowledge and to simultaneously assess the scientific quality of the inquiry. The ARPI consists of nineteen understandings of evidence divided over six phases of inquiry. It enables the objective assessment of students’ work without the need to adapt the rubric to differing research topics. In this presentation we highlight the main features of this validated instrument and demonstrate its applicability in a fully open lab course in which natural science students conceive, design, and carry out two extended experiments using accessible Maker tools.

Using the Assessment Rubric for Physics Inquiry for open inquiries in a multidisciplinary lab course

The Force Concept Inventory (FCI), one of the most widely used tools in the physics community, is commonly used as a pre and posttest assessment to gauge the effectiveness of various teaching strategies. Over the years there have been various studies on the FCI itself, and different formats have been created. In this preliminary study we are looking at the incorporation of videos that illustrate the problem statements as a way to see if these additions have positive effects on students’ responses to FCI questions. We took a subset of four questions from the FCI and varied the format of the questions and videos in quizzes administered to multiple introductory algebra-based physics courses. Though there were no quantitatively significant differences in students’ performance on questions between those who answered the as-written FCI questions versus the questions with the videos, we found significant differences on some questions with repeated responses to the same questions with and without video. Furthermore, students overall felt the videos helped them visualize the question scenarios.

Preliminary results on a video-based force concept inventory

Due to the COVID-19 pandemic, many universities have had to transition to online instruction. For large-enrollment introductory physics classes, this situation raises questions of ensuring academic honesty and fairness in assessments. Short, low-stakes quizzes have been suggested as an alternative to traditional exams, in order to minimize opportunity for students to engage in academic dishonesty by way of unauthorized online resources that can be problematic for longer online exams, and to reduce the pressure students feel that may tempt them to cheat on high-stakes exams. However, short quizzes could still be vulnerable to academic dishonesty by way of student sharing of assessment content with each other through various means that may be difficult to detect. Moreover, students may not perceive that short quizzes are meant to be low-stakes. In this investigation, the prevalence of academic dishonesty via unauthorized information sharing was explored in two remote large-enrollment introductory physics courses. In addition, student perception of assessments was investigated. It was found that students learning remotely had a preference for less-frequent, longer assessments, compared to frequent, short quizzes. It was also found that evidence for academic dishonesty was reduced when students perceived the assessments as fair, even if the online format was perceived as more difficult than an in-class experience.

Creating honest and fair remote introductory physics assessments

We use quantitative measures and visual inspection to compare the item response curves (IRCs) of matched pre-/post-instruction Force Concept Inventory (FCI) data. We find that the IRCs are not static; the pre-instruction IRCs differ from the post-instruction IRCs by more than can be explained by random chance. This result is also the case for a subpopulation consisting of students who make little or no gains on the FCI, suggesting that learning is taking place even when scores do not change appreciably. We consider three items where students make substantial progress (item 4) or little progress (items 14 and 21) compared to overall changes in FCI scores.

Next from PERC SM 21

When the light bulb turns on: motivation and collaboration spark the creation of ideas for theoretical physicists

Similar lecture

Bias on the Force Concept Inventory across the intersection of gender and race