United States

A previous study shows that incentivizing students to correct mistakes on unit exam problems within an upper-division quantum mechanics course improved students&#39; problem-solving efforts on those same problems in a final exam environment, relative to a comparison group of students who were not incentivized. We attempt to replicate the quantitative portion of this study within a first-semester upper-division electromagnetism course, specifically examining students&#39; invoking correct concepts and applying those concepts correctly. A statistical comparison of students who accepted the offer to rework unit exam problems for partial credit, versus students who declined the offer, demonstrates a better improvement for students who chose to rework relative to students who declined. As the results suggested that unit exam performance might provide a covariate within the comparison of choice to rework between groups, the results were analyzed using ANCOVA; they showed a result for one of the three featured exam problems that may be hidden by regular 1-way ANOVA analysis of pre-post normalized gains. The authors also investigate whether a focus on conceptual understanding versus a focus on algorithmic calculation for different exam problems affects the degree of benefit for students who rework problems. Results appear to show that incentivization works more specifically for invoking correct concepts on a primarily conceptual problem, and more specifically for applying concepts correctly on a primarily algorithmic problem. Future plans include a more complete analytical framework using think-aloud protocol interviews for students from the sample, as well as more statistical detail to determine the interaction between unit exam score and choice to rework problems.

PERC SM 22

Reworking exam problems to incentivize improved performance in upper-division electrodynamics

PERC 2022 is focused on "queering physics education" through an intersectional lens. By "queering," we are centering queer theory which breaks down social constructions and hierarchies to unpack normalized assumptions. In the context of physics education, we want to apply the theory to unpack who has power in physics to control the production of future physicists, how physics policies and practices are sometimes built on ideas of punishment and power, and how PER embeds binaries in both its content and sociological research. Further, we want to encourage a dialogue that is both a critique and an imagining of a queer future for PER.

Explicit attention will be given to constructions of race, (dis)ability, gender, sexuality and how they interact and impact lived experiences.

We have had such a fun time organizing this year’s PERC. You might be aware that we were committed to transparency and documenting our processes and decision making which we outlined here. This year has also been challenging to be a queer or otherwise marginalized person in the US. And we continue to seek and provide support for those individuals.

As such, we recommend that folks visiting Grand Rapids who have the means consider giving to some local organizations who are working to advocate for those impacted by historic and ongoing marginalization and violence.


PERC 2022 focused on "queering physics education" through an intersectional lens. By "queering," we centered queer theory which breaks down social constructions and hierarchies to unpack normalized assumptions. In the context of physics education, we apply the theory to unpack who has power in physics to control the production of future physicists, how physics policies and practices are sometimes built on ideas of punishment and power, and how PER embeds binaries in both its content and sociological research. Further, we wanted to encourage a dialogue that is both a critique and an imagining of a queer future for PER. Explicit attention was given to constructions of race, (dis)ability, gender, sexuality and how they interact and impact lived experiences.

A previous study shows that incentivizing students to correct mistakes on unit exam problems within an upper-division quantum mechanics course improved students' problem-solving efforts on those same problems in a final exam environment, relative to a comparison group of students who were not incentivized. We attempt to replicate the quantitative portion of this study within a first-semester upper-division electromagnetism course, specifically examining students' invoking correct concepts and applying those concepts correctly. A statistical comparison of students who accepted the offer to rework unit exam problems for partial credit, versus students who declined the offer, demonstrates a better improvement for students who chose to rework relative to students who declined. As the results suggested that unit exam performance might provide a covariate within the comparison of choice to rework between groups, the results were analyzed using ANCOVA; they showed a result for one of the three featured exam problems that may be hidden by regular 1-way ANOVA analysis of pre-post normalized gains. The authors also investigate whether a focus on conceptual understanding versus a focus on algorithmic calculation for different exam problems affects the degree of benefit for students who rework problems. Results appear to show that incentivization works more specifically for invoking correct concepts on a primarily conceptual problem, and more specifically for applying concepts correctly on a primarily algorithmic problem. Future plans include a more complete analytical framework using think-aloud protocol interviews for students from the sample, as well as more statistical detail to determine the interaction between unit exam score and choice to rework problems.

poster

Recent years have seen a movement within the research-based assessment development community towards item formats that go beyond simple multiple-choice formats. Some have moved towards free-response questions, particularly at the upper-division level; however, free-response items have the constraint that they must be scored by hand.  To avoid this limitation, some assessment developers have moved toward formats that maintain the closed-response format, while still providing more nuanced insight into student reasoning.  One such format is known as coupled, multiple response (CMR).  This format pairs multiple-choice and multiple-response formats to allow students to both commit to an answer in addition to selecting options that correspond with their reasoning. In addition to being machine-scorable, this format allows for more nuanced scoring than simple right or wrong.  However, such nuanced scoring presents a potential challenge with respect to utilizing certain testing theories to construct validity arguments for the assessment.  In particular, Item Response Theory (IRT) models often assume dicotomously scored items.  While polytomous IRT models do exist, each brings with it certain constraints and limitations.  Here, we will explore multiple IRT models and scoring schema using data from an existing CMR test, with the goal of providing guidance and insight for possible methods for simultaneously leveraging the affordances of both the CMR format and IRT models in the context of constructing validity arguments for research-based assessments.

Methods for utilizing Item response theory with Coupled, Multiple-Response assessments

Researchers across all scientific disciplines routinely face ethical decisions in their work, from addressing conflicts of interest to deciding whether and how to make data available for reproducibility. To help strengthen their ethical reasoning skills, they are encouraged to take online training programs like the CITI program. Ethics training is insufficient for improving ethical behavior. Better understanding of how scientists make decisions and reason about ethics is needed. To develop that understanding, we need expanded, asset-based measures of ethical reasoning that can be applied to open-ended responses and discussions. As part of a year-long intervention on a group of fifteen scientists' value-based reasoning, we conducted pre/post interviews that included open-ended questions about ethical scenarios. For this paper, we explore an application of three theories of ethical and stakeholder reasoning to those answers, and determine that we can use them to examine quality, principles, and subjects of their reasoning in open responses.

Examining Physicists' Ethical Reasoning: A Case Study

We identify three conceptual resources that introductory physics students in our sample commonly use when reasoning microscopically about thermal physics topics: 1) differences will eventually even out, 2) macroscopic changes connect to microscopic collisions, and 3) when something is hotter (colder), its molecules are moving faster (slower). We report the prevalence of these resources, as well as the prevalence of microscopic thinking, in 624 written responses to three different heat and temperature questions administered to introductory physics students at four different colleges and universities. This work complements past research identifying common student difficulties in using microscopic models for heat and temperature, and it adds to the small but growing body of literature that focuses on student resources for heat and temperature, identifying ideas in student thinking that are sensible and continuous with formal physics. By reporting common student resources, we aim to assist instructors in noticing, appreciating, and building on student ideas during introductory thermal physics instruction.

Identifying student resources for reasoning microscopically about heat and temperature

Research-Practice Partnership (RPP) is recognized as a valuable tool for generating actionable research and improving educational practices by involving practitioners and researchers in both research and the implementation of research findings. However, creating and maintaining such partnerships is challenging; it requires time and financial resources, and a team with diverse expertise and high commitment. Despite their wide recognition in educational fields, RPPs are under-studied in physics education research and literature in informal physics programs is sparse. In this paper, we present a case study of RPP between a physics education researcher and an informal physics practitioner. We describe the development and maintenance of this partnership and use qualitative and ethnographic methods to investigate practitioner-driven concerns about self-evaluation practices and attendance in the program under study, an informal conference for physics teachers. We found evidence that surveys and ethnographic interviews provided complementary data: surveys allowed the practitioner to learn about their audience's motivations, needs, and perception of the program's value. Ethnographic interviews were particularly useful as a means to elicit actionable ideas for how conference organizers might better support teacher attendance. In addition, participating in the RPP provided the practitioner with skills that they can carry forward into their future assessment work and that this partnership yielded actionable insights beneficial to both practitioners and audience. Our research team was able to overcome inherent challenges of RPP by exploring each side's language and cultural norms, and maintained a long-term partnership despite limitations. Our experience suggests that RPPs in informal physics spaces face many of the same challenges as those conducted in formal education settings, and that best practices documented for formal education RPPs will also be of benefit to them.

You Pick My Brain, I’ll Pick Yours: Research-Practice Partnership in an Informal Physics Space

Changing basis is a common task when solving quantum mechanical problems.  As part of a research project investigating student understanding of basis and change of basis in quantum mechanics, we developed a tutorial to support students in learning about basis in the context of spin-1/2 systems.  We have since created an interactive online version of the basis tutorial as part of a freely available suite of online quantum tutorials called ACE Physics (https://acephysics.net).  The ACE Physics tutorials include dynamic guidance elements and, unlike other tutorials, are intended for use outside the classroom without instructor facilitation.  After extensive study in an instructor-supported environment, we assigned the ACE Physics basis tutorial as homework in two semesters of upper-division quantum mechanics, and we report on the effectiveness of the activity based on pre-/post-testing and comparison of student exam performance with a similar semester that did not include the activity.  We find that the tutorial produces sufficient learning gains to justify continued assignment as a homework problem in our classes.

Effectiveness of an online homework tutorial about changing basis in quantum mechanics

In this paper, we analyze video recordings of students working on tutorials in Zoom breakout rooms in an upper-division quantum mechanics course. We investigate group behaviors in this virtual environment, including the effects of instructor presence.
To this end, we modify the Color Frames coding scheme introduced by Scherr to suit the virtual nature of the interactions. By broadening the frames and allowing for multiple overlapping frames, we are able to describe some group behaviors not otherwise captured.  For example, in some instances, students take on an authoritative role in the group, and in other instances, groups engage in overtly casual behavior while nonetheless having on-topic discussions.  We observe significant variation in how much time each group spends in each frame, but find that all groups spend some time in all frames.  Instructors can be present without dominating or eliminating discussion between students, and their presence need not significantly impact the time students spent in an ``informal/friendly'' frame. However, instructor presence significantly reduces time spent working individually.  
Our findings will support additional research into the dynamics of student discussions during tutorials and aid ongoing development of online tutorials that can, e.g., be assigned for use outside of class.

Modified color frames for analyzing group interactions during an online quantum tutorial

This research focuses on the experiences of physics and astronomy graduate women of color. We conducted semi-structured, empathetic interviews to understand their experiences in their undergraduate and graduate programs and how they navigate the physics departments at predominantly white institutions (PWI). The interviews serve as counter stories, which is a central tenet of critical race theory. We use this framework to examine how racial identities play a role in the obstacles faced by these women, including interactions with peers and faculty members. In this paper, we focus on the experiences of a Black woman in physics, Linda, to understand how her marginalized identities affected her experiences in physics during her undergraduate and graduate programs.

Investigating experiences of graduate women of color in physics & astronomy

Scientists are encouraged to engage in some form of ethical training. A common criticism of these training modules is that they are not effective. We formed a fellowship of fifteen scientists to investigate different modes of engagement with value-laden topics to see how scientists would engage in more ethically-driven discussions. These scientists met for a series of group sessions over the course of an academic year. We examined these discussions through the lens of epistemic frames, and characterize these frames through a number of characteristics including behaviors, verbal cues, and epistemic forms. We use these characteristics to identify the frames present in the fellowship, and what factors influenced changes in frames. Categorizing discussions with epistemic frames offers an opportunity for identifying what is driving more ethically-minded discussions, and this knowledge could then be used to better design ethical training modules.

Identifying Epistemic Games in Value-Laden Discussions

Covariational reasoning--considering how changes in one quantity affect another, related quantity--is a foundation of quantitative modeling in physics. Understanding quantitative models is a learning objective of introductory physics instruction at the college level. Prior work suggests that covariational reasoning in physics contexts differs from the reasoning about functions and graphs in purely mathematical contexts that students develop in math courses; this reasoning is effortful in physics even for mathematically well-prepared students. In order to improve physics students' covariational reasoning, we must first characterize covariational reasoning with physics quantities. To this end, we present a framework of covariational reasoning in physics contexts, to describe the ways that covariational reasoning is used in physics modeling. The framework can be used as a lens through which to analyze student reasoning, and can help inform instructional interventions. We describe an application of this framework in the development of a set of computer-based training assignments.

Characterizing covariational reasoning in physics modeling

Physics identity is an essential topic in Physics Education Research, PER. Historically this research has examined how physics identities form for primarily white, male undergraduate students in calculus-based physics classes at Tier I institutions. More recent research has begun to consider the experiences of marginalized populations, such as students of color, women, and members of the LGBTQIA+ community. Nepantla is a theoretical framework that explicitly explores how individuals navigate opposing identities and realities. Physics students from marginalized groups are navigating multiple, often conflicting identities. Therefore, this paper asks how applying the theoretical framework of Nepantla to PER contributes to the physics identity conversation by providing insight into how individuals in marginalized populations are able to navigate their multiple identities. This paper looks at how Nepantla has been used as a theoretical framework in other areas of STEM-Ed, then applies Nepantla to existing research within PER. The results suggest that the theoretical framework of Nepantla contributes to understanding how Nepantler@s both navigate the system and change the system.

Premium content

Reworking exam problems to incentivize improved performance in upper-division electrodynamics

Downloads

Next from PERC SM 22

Methods for utilizing Item response theory with Coupled, Multiple-Response assessments

Similar lecture

Beyond the Binary: Critically Queer Approaches to Quantitative Research in Education