United States

In this work we report on a pilot study where we used machine learning to predict whether students will correctly solve the classic “ballistic pendulum” problem based on an essay written by students elucidating their approach to solving the problem. Specifically, students were asked to describe the “principles, assumptions, and approximations” they used to solve the problem. Student essays were codified using the practices of natural language processing. Essays from two non-consecutive semesters were used for training/validation (N = 1441) and testing (N=1480). The final model used to make predictions was an ensemble classification scheme using random forest, eXtreme Gradient Boosting classifier (XGBoost), and logistic regression as estimators. Our accuracy in predicting students’ correctness was around 80% with slightly higher accuracy in identifying students who incorrectly solved the problem and slightly lower in predicting student who correctly solved the problem.

PERC SM 21

Using natural language processing to predict student problem solving performance

technology

assessment

machine learning

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

poster

We use quantitative measures and visual inspection to compare the item response curves (IRCs) of matched pre-/post-instruction Force Concept Inventory (FCI) data. We find that the IRCs are not static; the pre-instruction IRCs differ from the post-instruction IRCs by more than can be explained by random chance. This result is also the case for a subpopulation consisting of students who make little or no gains on the FCI, suggesting that learning is taking place even when scores do not change appreciably. We consider three items where students make substantial progress (item 4) or little progress (items 14 and 21) compared to overall changes in FCI scores.

Comparing pre/post item response curves to identify changes in misconceptions

Conceptual inventories such as the FCI are widely used to assess the impact of course reform efforts in physics education. Gains on these instruments, however, are known to correlate with other student variables such as students’ SAT score. I make use of this type of underlying correlation to communicate the impact of course reforms to broad audiences outside of physics education for which the FCI is not a useful benchmark. To illustrate this approach, I present an analysis of FCI learning gains that are disaggregated by ACT score before and after a major course reform in an introductory physics course. In this sample, normalized learning gains shifted from 0.28 to 0.46 were found to be equivalent to a shift in average ACT score from 24 to 29. Implications for the communicating research and evaluation results with different audiences are discussed.

Making use of FCI–ACT Correlations to Communicate the Impact of Course Reform

The updated Next Generation Science Standards place an emphasis on computational integration into science classrooms through the inclusion of mathematics and computational thinking; however, research focused on understanding the impacts and methods of computational integration is still in its infancy. Important questions around activity design and assessment remain unanswered and understudied. In this paper, we present an initial case study of one classroom that integrates computation into a second semester, introductory level physics class. In the course, instructional staff use formative feedback as an assessment tool to both evaluate students and help them develop computational practices. Instructor feedback for several years was coded for repeating trends, resulting in 3 categories of feedback: praise, notice and suggestion. An examination of the data revealed that the feedback is helping to develop positive norms around collaborative computational practices, but little attention is paid to the development of computational thinking practices.

Analysis of computation-based formative feedback

Three-dimensional assessments address application of knowledge and are based on a combination of core ideas, scientific practices, and crosscutting concepts. In this paper, we present one portion of a larger project aimed at developing a flexible, three-dimensional assessment for upper-division thermal physics: the Thermal and Statistical Physics Assessment (TaSPA). The TaSPA tasks are guided by ``learning performances'' (LPs), which faculty interface with to customize the assessment version they prefer to administer in their class. Here, we focus on interviews conducted with 7 physics faculty who teach upper-division thermal physics to gain insights about the three-dimensional LPs developed for the TaSPA. We found faculty rated our LPs variably, but highly overall, and their reasons for their ratings often focused on core concepts as opposed to scientific practices. This finding informs the design of LPs for the TaSPA and supports the need for a flexible assessment which can address varying course foci.

Faculty Perceptions of Three-Dimensional Learning Performances Addressing Thermal Physics

PER-developed assessments are used to track students' progress in a curriculum. If carefully designed, such assessments can also be used by instructors to inform explicit course modifications beyond using them just as tools to evaluate student learning. However, this requires a mechanism to translate assessment outcomes, which are usually presented as a score, into meaningful information about student learning. This information can in turn inform actionable feedback for instructors. In this work, we advocate for PER-developed assessments that explicitly include actionable feedback for instructors. We demonstrate a mechanism to generate feedback for coupled, multiple-response tasks in the context of developing a new standardized assessment for upper-division thermal physics. The inherent features of the coupled, multiple-response tasks such as the ability to administer online with streamlined scoring inform its broader utility in classrooms with small to large student enrollment.

Conceptualizing Assessments in PER Informed by Explicit Feedback for Instructors

Qualitative data coding is a powerful research tool for analyzing text statements; however, having human researchers manually code can be time-consuming and hindering to a project with a large data set. Modern machine learning programs have advanced such that they currently play large roles in automating data categorization and may be useable as a method of semi-automating categorization in PER: human coders can prepare a moderate sample of the data for training, and machine learning software can be left to sort the remaining data, leaving only complex responses to be finally resolved by humans. We have attempted this process using data from a study seeking to code several hundred answers to a single free-response question. Our preliminary work provides Google’s AutoML API with a minimal training sample. In this poster, we compare the accuracy and efficiency of their software to human coders’ interrater reliability to demonstrate how we need to continue developing our training sets moving forward.

Understanding the Efficacy of Machine Learning for Coding Practices

A small number of prior studies have investigated whether instructor gender has a measurable impact on student course grades, though to our best knowledge no study has specifically examined physics courses. We explored the effect of instructor gender on student performance in calculus-based introductory physics courses at a large land grant university. The data set included the scores on midterm and final exams and final letter grades for more than 8200 students taught by 11 male and 4 female instructors between 2007-2018. We found no persistent correlation between student gendered performance and their instructor gender. While statistically significant differences in student performance were found for particular exams or in final letter grades, the effect sizes were weak or small. When comparing male and female instructors, we found that the instructor gender had a larger effect on female students than on their male classmates.

Does Instructor Gender Matter for Student Performance in Introductory Physics?

Response-shift bias explains how changes in students' criteria for self-evaluation can suppress the effect of educational interventions in pre-to-post shifts of self-reported metrics, such as the Colorado Learning Attitudes about Science Survey (CLASS). Therefore, the static or declining pre-to-post shifts seen in the CLASS might not reflect development in students' reasoning about themselves based on new knowledge of physics. We investigated possible causes of response-shift bias in the CLASS by adding a free response question after select CLASS items during a two-pass survey administration, asking students to explain their reasoning behind their expected and perceived shifts. We present the themes that occurred most clearly in students' free responses to one survey item, which highlight the established forms of response-shift bias: recalibration of the students' standards of measurement, reprioritization of the students' values, and reconceptualization of the subject's content.

Themes in student self-assessments of attitudinal development in the CLASS

Classifying students into groups that exhibit similar behavioral patterns so they can be treated with a tailored instruction is a new research direction that attracts both the Physics Education and the Learning Analytics research communities. Typically, personalization approaches center on the individual learner, and identify the knowledge components required for solving each problem based on expert task analysis or educational data mining. However, individual-level personalization is less appropriate in the reality of science classes, and may also miss the advantages of group work and discussions. Moreover, previous studies have shown that experts may miss latent knowledge structures that bottom-up analyses such as Exploratory Factor Analysis (EFA) may help to reveal. 

In this study we examined how cluster analysis can reveal additional cognitive dimensions not identified by the experts. We used cluster analysis and not EFA which is typically used for this purpose because our end goal was using the cognitive dimensions to provide the teachers with sub-groups of students sharing similar profiles. 

The research tool that we used was an instrument on Magnetism. The items were analyzed by experts and the knowledge components identified for each item were represented in a Q-matrix. The instrument was administered to ~300 12th grade Physics majors. K-means cluster analysis revealed that subgroups can be differentiated by type of knowledge structures (e.g. procedural vs. declarative), resulting in improved labeling of the questions. In follow-up interviews the teachers expressed their satisfaction with the usefulness of this representation for tailoring their teaching to each group's difficulties and strengths.

Combining expert- and cluster-analysis for Group-level personalization

As part of an effort to examine student understanding of mathematical representations in quantum mechanics (QM), online survey tasks were designed and administered to students at the end of their upper-division QM courses at multiple institutions. Tasks were designed to find expressions and concepts that were related or viewed as similar in students’ minds, and to determine what concepts and properties were ascribed to various expressions. Network analysis techniques were used to identify conceptual connections between expressions within and between Dirac, wave function, and standard vector notations. Community detection algorithms were used to study similarity between expressions, and network distance algorithms were used to study similarity between quantum mechanical concepts. Connections were found between Dirac expressions and vector expressions. Also, distinct single- and multiple-term communities were identified. Network analysis confirmed some clear conceptual connections, including between unit vectors and basis vectors, probability and probability amplitude, and eigenvectors and basis vectors.

Premium content

Using natural language processing to predict student problem solving performance

Next from PERC SM 21

Comparing pre/post item response curves to identify changes in misconceptions

Similar lecture

Supporting and Studying Introductory Physics Students’ Algebra Skills with Solve-It