United States

Qualitative data coding is a powerful research tool for analyzing text statements; however, having human researchers manually code can be time-consuming and hindering to a project with a large data set. Modern machine learning programs have advanced such that they currently play large roles in automating data categorization and may be useable as a method of semi-automating categorization in PER: human coders can prepare a moderate sample of the data for training, and machine learning software can be left to sort the remaining data, leaving only complex responses to be finally resolved by humans. We have attempted this process using data from a study seeking to code several hundred answers to a single free-response question. Our preliminary work provides Google’s AutoML API with a minimal training sample. In this poster, we compare the accuracy and efficiency of their software to human coders’ interrater reliability to demonstrate how we need to continue developing our training sets moving forward.

PERC SM 21

Understanding the Efficacy of Machine Learning for Coding Practices

qualitative data analysis

coding

machine learning

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

poster

Education researchers often compare performance across race and gender on research-based assessments of physics knowledge to investigate the impacts of racism and sexism on physics student learning. These investigations' claims rely on research-based assessments providing reliable, unbiased measures of student knowledge across social identity groups. We used classical test theory and differential item functioning (DIF) analysis to examine whether the items on the Force Concept Inventory (FCI) provided unbiased data across social identifiers for race, gender, and their intersections. The data was accessed through the Learning About STEM Student Outcomes platform and included responses from 4,848 students posttests in 152 calculus-based introductory physics courses from 16 institutions. The results indicated that the majority of items (22) on the FCI were biased towards a group. These results point to the need for instrument validation to account for item bias and the identification or development of fair research-based assessments.

Bias on the Force Concept Inventory across the intersection of gender and race

Previous work has shown that item response theory may be used to rank incorrect response options to multiple-choice items on commonly used assessments. This work has shown that, when the correct response to each item is specified, a nominal response model (NRM) may be used to rank the incorrect options. We seek to determine the robustness of these results by applying the NRM to all response choices, without specifying the correct response. We apply these analyses to multiple data sets (each with more than 9,000 response sets), including pre-instruction and post-instruction responses. We find that the rankings generated without specifying the correct response are consistent with the previously published rankings for one data set; however, we find noticeable differences between rankings generated from different data sets. We provide evidence that discrepancies may result from differences in response rates for less commonly chosen responses.

Comparing item response theory models for ranking incorrect response options

Many researchers find advantages of assigning and supporting open inquiries in physics lab education. However, objective assessment becomes more difficult as students gain more control over their own experiments because tasks are not described in operational detail. The Assessment Rubric for Physics Inquiry (ARPI) is designed to determine students’ attainment level of inquiry knowledge and to simultaneously assess the scientific quality of the inquiry. The ARPI consists of nineteen understandings of evidence divided over six phases of inquiry. It enables the objective assessment of students’ work without the need to adapt the rubric to differing research topics. In this presentation we highlight the main features of this validated instrument and demonstrate its applicability in a fully open lab course in which natural science students conceive, design, and carry out two extended experiments using accessible Maker tools.

Using the Assessment Rubric for Physics Inquiry for open inquiries in a multidisciplinary lab course

The Force Concept Inventory (FCI), one of the most widely used tools in the physics community, is commonly used as a pre and posttest assessment to gauge the effectiveness of various teaching strategies. Over the years there have been various studies on the FCI itself, and different formats have been created. In this preliminary study we are looking at the incorporation of videos that illustrate the problem statements as a way to see if these additions have positive effects on students’ responses to FCI questions. We took a subset of four questions from the FCI and varied the format of the questions and videos in quizzes administered to multiple introductory algebra-based physics courses. Though there were no quantitatively significant differences in students’ performance on questions between those who answered the as-written FCI questions versus the questions with the videos, we found significant differences on some questions with repeated responses to the same questions with and without video. Furthermore, students overall felt the videos helped them visualize the question scenarios.

Preliminary results on a video-based force concept inventory

Due to the COVID-19 pandemic, many universities have had to transition to online instruction. For large-enrollment introductory physics classes, this situation raises questions of ensuring academic honesty and fairness in assessments. Short, low-stakes quizzes have been suggested as an alternative to traditional exams, in order to minimize opportunity for students to engage in academic dishonesty by way of unauthorized online resources that can be problematic for longer online exams, and to reduce the pressure students feel that may tempt them to cheat on high-stakes exams. However, short quizzes could still be vulnerable to academic dishonesty by way of student sharing of assessment content with each other through various means that may be difficult to detect. Moreover, students may not perceive that short quizzes are meant to be low-stakes. In this investigation, the prevalence of academic dishonesty via unauthorized information sharing was explored in two remote large-enrollment introductory physics courses. In addition, student perception of assessments was investigated. It was found that students learning remotely had a preference for less-frequent, longer assessments, compared to frequent, short quizzes. It was also found that evidence for academic dishonesty was reduced when students perceived the assessments as fair, even if the online format was perceived as more difficult than an in-class experience.

Creating honest and fair remote introductory physics assessments

We use quantitative measures and visual inspection to compare the item response curves (IRCs) of matched pre-/post-instruction Force Concept Inventory (FCI) data. We find that the IRCs are not static; the pre-instruction IRCs differ from the post-instruction IRCs by more than can be explained by random chance. This result is also the case for a subpopulation consisting of students who make little or no gains on the FCI, suggesting that learning is taking place even when scores do not change appreciably. We consider three items where students make substantial progress (item 4) or little progress (items 14 and 21) compared to overall changes in FCI scores.

Comparing pre/post item response curves to identify changes in misconceptions

Conceptual inventories such as the FCI are widely used to assess the impact of course reform efforts in physics education. Gains on these instruments, however, are known to correlate with other student variables such as students’ SAT score. I make use of this type of underlying correlation to communicate the impact of course reforms to broad audiences outside of physics education for which the FCI is not a useful benchmark. To illustrate this approach, I present an analysis of FCI learning gains that are disaggregated by ACT score before and after a major course reform in an introductory physics course. In this sample, normalized learning gains shifted from 0.28 to 0.46 were found to be equivalent to a shift in average ACT score from 24 to 29. Implications for the communicating research and evaluation results with different audiences are discussed.

Making use of FCI–ACT Correlations to Communicate the Impact of Course Reform

The updated Next Generation Science Standards place an emphasis on computational integration into science classrooms through the inclusion of mathematics and computational thinking; however, research focused on understanding the impacts and methods of computational integration is still in its infancy. Important questions around activity design and assessment remain unanswered and understudied. In this paper, we present an initial case study of one classroom that integrates computation into a second semester, introductory level physics class. In the course, instructional staff use formative feedback as an assessment tool to both evaluate students and help them develop computational practices. Instructor feedback for several years was coded for repeating trends, resulting in 3 categories of feedback: praise, notice and suggestion. An examination of the data revealed that the feedback is helping to develop positive norms around collaborative computational practices, but little attention is paid to the development of computational thinking practices.

Analysis of computation-based formative feedback

Three-dimensional assessments address application of knowledge and are based on a combination of core ideas, scientific practices, and crosscutting concepts. In this paper, we present one portion of a larger project aimed at developing a flexible, three-dimensional assessment for upper-division thermal physics: the Thermal and Statistical Physics Assessment (TaSPA). The TaSPA tasks are guided by ``learning performances'' (LPs), which faculty interface with to customize the assessment version they prefer to administer in their class. Here, we focus on interviews conducted with 7 physics faculty who teach upper-division thermal physics to gain insights about the three-dimensional LPs developed for the TaSPA. We found faculty rated our LPs variably, but highly overall, and their reasons for their ratings often focused on core concepts as opposed to scientific practices. This finding informs the design of LPs for the TaSPA and supports the need for a flexible assessment which can address varying course foci.

Faculty Perceptions of Three-Dimensional Learning Performances Addressing Thermal Physics

PER-developed assessments are used to track students' progress in a curriculum. If carefully designed, such assessments can also be used by instructors to inform explicit course modifications beyond using them just as tools to evaluate student learning. However, this requires a mechanism to translate assessment outcomes, which are usually presented as a score, into meaningful information about student learning. This information can in turn inform actionable feedback for instructors. In this work, we advocate for PER-developed assessments that explicitly include actionable feedback for instructors. We demonstrate a mechanism to generate feedback for coupled, multiple-response tasks in the context of developing a new standardized assessment for upper-division thermal physics. The inherent features of the coupled, multiple-response tasks such as the ability to administer online with streamlined scoring inform its broader utility in classrooms with small to large student enrollment.

Premium content

Understanding the Efficacy of Machine Learning for Coding Practices

Next from PERC SM 21

Bias on the Force Concept Inventory across the intersection of gender and race

Similar lecture

Using natural language processing to predict student problem solving performance