United States

We evaluated the use of student peer review in an introductory physics laboratory as a central part of writing instruction. During a recent revision of a physics laboratory course the number of written reports was reduced from weekly to three per semester but each report went through a process of review by a peer and then revision based on the feedback before final submission for grading by the instructor. Copies of laboratory reports from all sections of the laboratory were collected and evaluated following a coding scheme that had previously been developed to evaluate the quality of student writing on different facets of the written reports. The results were then compared to student reports from before the revision. It was found that the quality of the reports undergoing peer review was comparable or better than those without, although the former submitted only a fourth of the number of reports. The use of peer review resulted in comparable to improved student writing while significantly reducing the workload for students and instructors.

PERC SM 21

Evaluating the Effectiveness of Peer Review in Introductory Physics Labs

introductory

physics education research

peer review

Join AAPT and Physics Educators worldwide in developing and strengthening your professional connections by attending the 2021 AAPT Virtual Summer Meeting. AAPT meetings are a great way to reflect on your teaching, recommit to best practices and re-energize your career with new research, products, and techniques.

Listen to internationally known plenary speakers, attend sessions, workshops, and special activities. Attending the virtual summer meeting is an excellent opportunity to share your expertise and learn from and network with leaders in the field.

Please log in with your AAPT account or register for the event if you haven't already.

If you are registered please note that you need to be logged in with your AAPT email / credentials in order to get access.

AAPT SM 21

poster

**Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics**

As a gateway discipline to STEM majors and careers, physics plays a central role in deciding who does, and who doesn't, get to discover and pursue scientific careers. Technical and scientific professionals are rewarded with economic privilege and status which affords them both social and cultural power. Therefore, social justice will be realized only when the opportunity, and the choice, to excel in physics is open to all students. The culture and practices of the physics classroom reflect the culture of physics at large, which raises the question of how our professional physics communities can embrace, rather than inadvertently repel, a diverse group of academics that can help improve physics culture.  To this end, the 2021 Physics Education Research Conference will focus on defining inclusive spaces and practices. Additionally, we will explore strategies for eliminating exclusionary practices in our classrooms, research spaces, and professional organizations. We call on the community to propose sessions that will drive conversations related to these initiatives throughout PERC 2021.

Please download the full program [here. ](https://drive.google.com/file/d/1TRMC_SYJ3PUqXyP7sskA7LRSA8ow0eRE/view?usp=sharing)

August 4, 2021 - August 5, 2021 in a Virtual Conference Format

Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics

The updated Next Generation Science Standards place an emphasis on computational integration into science classrooms through the inclusion of mathematics and computational thinking; however, research focused on understanding the impacts and methods of computational integration is still in its infancy. Important questions around activity design and assessment remain unanswered and understudied. In this paper, we present an initial case study of one classroom that integrates computation into a second semester, introductory level physics class. In the course, instructional staff use formative feedback as an assessment tool to both evaluate students and help them develop computational practices. Instructor feedback for several years was coded for repeating trends, resulting in 3 categories of feedback: praise, notice and suggestion. An examination of the data revealed that the feedback is helping to develop positive norms around collaborative computational practices, but little attention is paid to the development of computational thinking practices.

Analysis of computation-based formative feedback

Three-dimensional assessments address application of knowledge and are based on a combination of core ideas, scientific practices, and crosscutting concepts. In this paper, we present one portion of a larger project aimed at developing a flexible, three-dimensional assessment for upper-division thermal physics: the Thermal and Statistical Physics Assessment (TaSPA). The TaSPA tasks are guided by ``learning performances'' (LPs), which faculty interface with to customize the assessment version they prefer to administer in their class. Here, we focus on interviews conducted with 7 physics faculty who teach upper-division thermal physics to gain insights about the three-dimensional LPs developed for the TaSPA. We found faculty rated our LPs variably, but highly overall, and their reasons for their ratings often focused on core concepts as opposed to scientific practices. This finding informs the design of LPs for the TaSPA and supports the need for a flexible assessment which can address varying course foci.

Faculty Perceptions of Three-Dimensional Learning Performances Addressing Thermal Physics

PER-developed assessments are used to track students' progress in a curriculum. If carefully designed, such assessments can also be used by instructors to inform explicit course modifications beyond using them just as tools to evaluate student learning. However, this requires a mechanism to translate assessment outcomes, which are usually presented as a score, into meaningful information about student learning. This information can in turn inform actionable feedback for instructors. In this work, we advocate for PER-developed assessments that explicitly include actionable feedback for instructors. We demonstrate a mechanism to generate feedback for coupled, multiple-response tasks in the context of developing a new standardized assessment for upper-division thermal physics. The inherent features of the coupled, multiple-response tasks such as the ability to administer online with streamlined scoring inform its broader utility in classrooms with small to large student enrollment.

Conceptualizing Assessments in PER Informed by Explicit Feedback for Instructors

Qualitative data coding is a powerful research tool for analyzing text statements; however, having human researchers manually code can be time-consuming and hindering to a project with a large data set. Modern machine learning programs have advanced such that they currently play large roles in automating data categorization and may be useable as a method of semi-automating categorization in PER: human coders can prepare a moderate sample of the data for training, and machine learning software can be left to sort the remaining data, leaving only complex responses to be finally resolved by humans. We have attempted this process using data from a study seeking to code several hundred answers to a single free-response question. Our preliminary work provides Google’s AutoML API with a minimal training sample. In this poster, we compare the accuracy and efficiency of their software to human coders’ interrater reliability to demonstrate how we need to continue developing our training sets moving forward.

Understanding the Efficacy of Machine Learning for Coding Practices

A small number of prior studies have investigated whether instructor gender has a measurable impact on student course grades, though to our best knowledge no study has specifically examined physics courses. We explored the effect of instructor gender on student performance in calculus-based introductory physics courses at a large land grant university. The data set included the scores on midterm and final exams and final letter grades for more than 8200 students taught by 11 male and 4 female instructors between 2007-2018. We found no persistent correlation between student gendered performance and their instructor gender. While statistically significant differences in student performance were found for particular exams or in final letter grades, the effect sizes were weak or small. When comparing male and female instructors, we found that the instructor gender had a larger effect on female students than on their male classmates.

Does Instructor Gender Matter for Student Performance in Introductory Physics?

Response-shift bias explains how changes in students' criteria for self-evaluation can suppress the effect of educational interventions in pre-to-post shifts of self-reported metrics, such as the Colorado Learning Attitudes about Science Survey (CLASS). Therefore, the static or declining pre-to-post shifts seen in the CLASS might not reflect development in students' reasoning about themselves based on new knowledge of physics. We investigated possible causes of response-shift bias in the CLASS by adding a free response question after select CLASS items during a two-pass survey administration, asking students to explain their reasoning behind their expected and perceived shifts. We present the themes that occurred most clearly in students' free responses to one survey item, which highlight the established forms of response-shift bias: recalibration of the students' standards of measurement, reprioritization of the students' values, and reconceptualization of the subject's content.

Themes in student self-assessments of attitudinal development in the CLASS

Classifying students into groups that exhibit similar behavioral patterns so they can be treated with a tailored instruction is a new research direction that attracts both the Physics Education and the Learning Analytics research communities. Typically, personalization approaches center on the individual learner, and identify the knowledge components required for solving each problem based on expert task analysis or educational data mining. However, individual-level personalization is less appropriate in the reality of science classes, and may also miss the advantages of group work and discussions. Moreover, previous studies have shown that experts may miss latent knowledge structures that bottom-up analyses such as Exploratory Factor Analysis (EFA) may help to reveal. 

In this study we examined how cluster analysis can reveal additional cognitive dimensions not identified by the experts. We used cluster analysis and not EFA which is typically used for this purpose because our end goal was using the cognitive dimensions to provide the teachers with sub-groups of students sharing similar profiles. 

The research tool that we used was an instrument on Magnetism. The items were analyzed by experts and the knowledge components identified for each item were represented in a Q-matrix. The instrument was administered to ~300 12th grade Physics majors. K-means cluster analysis revealed that subgroups can be differentiated by type of knowledge structures (e.g. procedural vs. declarative), resulting in improved labeling of the questions. In follow-up interviews the teachers expressed their satisfaction with the usefulness of this representation for tailoring their teaching to each group's difficulties and strengths.

Combining expert- and cluster-analysis for Group-level personalization

As part of an effort to examine student understanding of mathematical representations in quantum mechanics (QM), online survey tasks were designed and administered to students at the end of their upper-division QM courses at multiple institutions. Tasks were designed to find expressions and concepts that were related or viewed as similar in students’ minds, and to determine what concepts and properties were ascribed to various expressions. Network analysis techniques were used to identify conceptual connections between expressions within and between Dirac, wave function, and standard vector notations. Community detection algorithms were used to study similarity between expressions, and network distance algorithms were used to study similarity between quantum mechanical concepts. Connections were found between Dirac expressions and vector expressions. Also, distinct single- and multiple-term communities were identified. Network analysis confirmed some clear conceptual connections, including between unit vectors and basis vectors, probability and probability amplitude, and eigenvectors and basis vectors.

Using network analysis techniques to probe student connections between Dirac notation and wave function expressions

In this work we report on a pilot study where we used machine learning to predict whether students will correctly solve the classic “ballistic pendulum” problem based on an essay written by students elucidating their approach to solving the problem. Specifically, students were asked to describe the “principles, assumptions, and approximations” they used to solve the problem. Student essays were codified using the practices of natural language processing. Essays from two non-consecutive semesters were used for training/validation (N = 1441) and testing (N=1480). The final model used to make predictions was an ensemble classification scheme using random forest, eXtreme Gradient Boosting classifier (XGBoost), and logistic regression as estimators. Our accuracy in predicting students’ correctness was around 80% with slightly higher accuracy in identifying students who incorrectly solved the problem and slightly lower in predicting student who correctly solved the problem.

Using natural language processing to predict student problem solving performance

The Conceptual Survey of Electricity and Magnetism (CSEM) is a multiple-choice survey that contains a variety of electricity and magnetism concepts at the level of introductory physics used to help inform instructors of student mastery of those concepts. Prior studies suggest that many concepts on the survey are challenging for introductory physics students and the average student scores after traditional instruction are low. The research presented here compares the performance of students in introductory, upper-level, and graduate-level physics courses on five CSEM questions to understand the cross-sectional evolution of student understanding of these concepts from the introductory to upper-level to graduate-level. We discuss five CSEM questions that remain challenging for many upper-level and graduate students.

Next from PERC SM 21

Analysis of computation-based formative feedback

Similar lecture

Effectively Implementing Peer Review into the Introductory Physics Laboratory Course