United States

Low-stakes research-based assessments are opportunities for professors to understand and integrate course transformation built on pedagogical research. These assessments often come in the form of pre- and post-tests, such as the Force and Motion Conceptual Evaluation -- the usefulness of which, however, relies on voluntary student participation. A prior study by Nissen et al. investigating participation rates as well as performance outcomes in low-stakes assessments found a possible connection between student participation rates and final course grades. We explore this connection at a different large research university with pre- and post-test participation rates and course grade data from a data set of more than 2000 students using statistical methods from the previous paper.

PERC SM 21

Investigating the connection between course grades and student participation in research-based assessments

student outcome

research-based assessments

equity

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

poster

PER-developed assessments are used to track students' progress in a curriculum. If carefully designed, such assessments can also be used by instructors to inform explicit course modifications beyond using them just as tools to evaluate student learning. However, this requires a mechanism to translate assessment outcomes, which are usually presented as a score, into meaningful information about student learning. This information can in turn inform actionable feedback for instructors. In this work, we advocate for PER-developed assessments that explicitly include actionable feedback for instructors. We demonstrate a mechanism to generate feedback for coupled, multiple-response tasks in the context of developing a new standardized assessment for upper-division thermal physics. The inherent features of the coupled, multiple-response tasks such as the ability to administer online with streamlined scoring inform its broader utility in classrooms with small to large student enrollment.

Conceptualizing Assessments in PER Informed by Explicit Feedback for Instructors

Qualitative data coding is a powerful research tool for analyzing text statements; however, having human researchers manually code can be time-consuming and hindering to a project with a large data set. Modern machine learning programs have advanced such that they currently play large roles in automating data categorization and may be useable as a method of semi-automating categorization in PER: human coders can prepare a moderate sample of the data for training, and machine learning software can be left to sort the remaining data, leaving only complex responses to be finally resolved by humans. We have attempted this process using data from a study seeking to code several hundred answers to a single free-response question. Our preliminary work provides Google’s AutoML API with a minimal training sample. In this poster, we compare the accuracy and efficiency of their software to human coders’ interrater reliability to demonstrate how we need to continue developing our training sets moving forward.

Understanding the Efficacy of Machine Learning for Coding Practices

A small number of prior studies have investigated whether instructor gender has a measurable impact on student course grades, though to our best knowledge no study has specifically examined physics courses. We explored the effect of instructor gender on student performance in calculus-based introductory physics courses at a large land grant university. The data set included the scores on midterm and final exams and final letter grades for more than 8200 students taught by 11 male and 4 female instructors between 2007-2018. We found no persistent correlation between student gendered performance and their instructor gender. While statistically significant differences in student performance were found for particular exams or in final letter grades, the effect sizes were weak or small. When comparing male and female instructors, we found that the instructor gender had a larger effect on female students than on their male classmates.

Does Instructor Gender Matter for Student Performance in Introductory Physics?

Response-shift bias explains how changes in students' criteria for self-evaluation can suppress the effect of educational interventions in pre-to-post shifts of self-reported metrics, such as the Colorado Learning Attitudes about Science Survey (CLASS). Therefore, the static or declining pre-to-post shifts seen in the CLASS might not reflect development in students' reasoning about themselves based on new knowledge of physics. We investigated possible causes of response-shift bias in the CLASS by adding a free response question after select CLASS items during a two-pass survey administration, asking students to explain their reasoning behind their expected and perceived shifts. We present the themes that occurred most clearly in students' free responses to one survey item, which highlight the established forms of response-shift bias: recalibration of the students' standards of measurement, reprioritization of the students' values, and reconceptualization of the subject's content.

Themes in student self-assessments of attitudinal development in the CLASS

Classifying students into groups that exhibit similar behavioral patterns so they can be treated with a tailored instruction is a new research direction that attracts both the Physics Education and the Learning Analytics research communities. Typically, personalization approaches center on the individual learner, and identify the knowledge components required for solving each problem based on expert task analysis or educational data mining. However, individual-level personalization is less appropriate in the reality of science classes, and may also miss the advantages of group work and discussions. Moreover, previous studies have shown that experts may miss latent knowledge structures that bottom-up analyses such as Exploratory Factor Analysis (EFA) may help to reveal. 

In this study we examined how cluster analysis can reveal additional cognitive dimensions not identified by the experts. We used cluster analysis and not EFA which is typically used for this purpose because our end goal was using the cognitive dimensions to provide the teachers with sub-groups of students sharing similar profiles. 

The research tool that we used was an instrument on Magnetism. The items were analyzed by experts and the knowledge components identified for each item were represented in a Q-matrix. The instrument was administered to ~300 12th grade Physics majors. K-means cluster analysis revealed that subgroups can be differentiated by type of knowledge structures (e.g. procedural vs. declarative), resulting in improved labeling of the questions. In follow-up interviews the teachers expressed their satisfaction with the usefulness of this representation for tailoring their teaching to each group's difficulties and strengths.

Combining expert- and cluster-analysis for Group-level personalization

As part of an effort to examine student understanding of mathematical representations in quantum mechanics (QM), online survey tasks were designed and administered to students at the end of their upper-division QM courses at multiple institutions. Tasks were designed to find expressions and concepts that were related or viewed as similar in students’ minds, and to determine what concepts and properties were ascribed to various expressions. Network analysis techniques were used to identify conceptual connections between expressions within and between Dirac, wave function, and standard vector notations. Community detection algorithms were used to study similarity between expressions, and network distance algorithms were used to study similarity between quantum mechanical concepts. Connections were found between Dirac expressions and vector expressions. Also, distinct single- and multiple-term communities were identified. Network analysis confirmed some clear conceptual connections, including between unit vectors and basis vectors, probability and probability amplitude, and eigenvectors and basis vectors.

Using network analysis techniques to probe student connections between Dirac notation and wave function expressions

In this work we report on a pilot study where we used machine learning to predict whether students will correctly solve the classic “ballistic pendulum” problem based on an essay written by students elucidating their approach to solving the problem. Specifically, students were asked to describe the “principles, assumptions, and approximations” they used to solve the problem. Student essays were codified using the practices of natural language processing. Essays from two non-consecutive semesters were used for training/validation (N = 1441) and testing (N=1480). The final model used to make predictions was an ensemble classification scheme using random forest, eXtreme Gradient Boosting classifier (XGBoost), and logistic regression as estimators. Our accuracy in predicting students’ correctness was around 80% with slightly higher accuracy in identifying students who incorrectly solved the problem and slightly lower in predicting student who correctly solved the problem.

Using natural language processing to predict student problem solving performance

The Conceptual Survey of Electricity and Magnetism (CSEM) is a multiple-choice survey that contains a variety of electricity and magnetism concepts at the level of introductory physics used to help inform instructors of student mastery of those concepts. Prior studies suggest that many concepts on the survey are challenging for introductory physics students and the average student scores after traditional instruction are low. The research presented here compares the performance of students in introductory, upper-level, and graduate-level physics courses on five CSEM questions to understand the cross-sectional evolution of student understanding of these concepts from the introductory to upper-level to graduate-level. We discuss five CSEM questions that remain challenging for many upper-level and graduate students.

Evolution in student conceptual understanding of electricity and magnetism

We evaluated the use of student peer review in an introductory physics laboratory as a central part of writing instruction. During a recent revision of a physics laboratory course the number of written reports was reduced from weekly to three per semester but each report went through a process of review by a peer and then revision based on the feedback before final submission for grading by the instructor. Copies of laboratory reports from all sections of the laboratory were collected and evaluated following a coding scheme that had previously been developed to evaluate the quality of student writing on different facets of the written reports. The results were then compared to student reports from before the revision. It was found that the quality of the reports undergoing peer review was comparable or better than those without, although the former submitted only a fourth of the number of reports. The use of peer review resulted in comparable to improved student writing while significantly reducing the workload for students and instructors.

Evaluating the Effectiveness of Peer Review in Introductory Physics Labs

Many physics education research studies aim to assess the impact of curriculum on student learning and experience. Fewer studies examine how the specific assessment structure or the grading approach of the course impact student learning and experience. In a typical university physics course, assignments are given a letter or numerical grade and returned to the student. This potentially focuses the student on obtaining high marks and discourages them from more productive epistemological stances. In this study, we aim to employ and assess the impact of an ungrading approach where grades are not given on individual assignments and a final letter grade is determined through a scaffolded negotiation between teacher and student. Preliminary data from different sections of a 2021 Spring semester college physics course are presented in this poster, and further work is needed.

Premium content

Investigating the connection between course grades and student participation in research-based assessments

Next from PERC SM 21

Conceptualizing Assessments in PER Informed by Explicit Feedback for Instructors

Similar lecture

What students gain outside the classroom: Using Social Network Analysis to understand the role of affinity-based professional groups