United States

Research characterizing common student ideas about particular physics topics has significantly impacted university-level physics teaching by providing knowledge that supports instructors to target their instruction and by informing curriculum development. In this work, we utilize a Natural Language Processing algorithm (Latent Dirichlet Allocation, or LDA) to identify distinct student ideas in a set of written responses to a conceptual physics question, with the goal of significantly expediting the process of characterizing student ideas. We preliminarily test the LDA approach by applying the algorithm to a collection of introductory physics student responses to a conceptual question about circuits, specifically attending to whether it is useful for characterizing instructionally-relevant student ideas. We find that for a large enough collection of student responses (N ≈ 500), LDA can be useful for characterizing the ideas students used to answer conceptual physics questions. We discuss some considerations that researchers may take into account as they interpret the results of the LDA algorithm for characterizing student&#39;s physics ideas.

PERC SM 22

Developing a natural language processing approach for analyzing student ideas in introductory calculus-based physics

PERC 2022 is focused on "queering physics education" through an intersectional lens. By "queering," we are centering queer theory which breaks down social constructions and hierarchies to unpack normalized assumptions. In the context of physics education, we want to apply the theory to unpack who has power in physics to control the production of future physicists, how physics policies and practices are sometimes built on ideas of punishment and power, and how PER embeds binaries in both its content and sociological research. Further, we want to encourage a dialogue that is both a critique and an imagining of a queer future for PER.

Explicit attention will be given to constructions of race, (dis)ability, gender, sexuality and how they interact and impact lived experiences.

We have had such a fun time organizing this year’s PERC. You might be aware that we were committed to transparency and documenting our processes and decision making which we outlined here. This year has also been challenging to be a queer or otherwise marginalized person in the US. And we continue to seek and provide support for those individuals.

As such, we recommend that folks visiting Grand Rapids who have the means consider giving to some local organizations who are working to advocate for those impacted by historic and ongoing marginalization and violence.


PERC 2022 focused on "queering physics education" through an intersectional lens. By "queering," we centered queer theory which breaks down social constructions and hierarchies to unpack normalized assumptions. In the context of physics education, we apply the theory to unpack who has power in physics to control the production of future physicists, how physics policies and practices are sometimes built on ideas of punishment and power, and how PER embeds binaries in both its content and sociological research. Further, we wanted to encourage a dialogue that is both a critique and an imagining of a queer future for PER. Explicit attention was given to constructions of race, (dis)ability, gender, sexuality and how they interact and impact lived experiences.

Research characterizing common student ideas about particular physics topics has significantly impacted university-level physics teaching by providing knowledge that supports instructors to target their instruction and by informing curriculum development. In this work, we utilize a Natural Language Processing algorithm (Latent Dirichlet Allocation, or LDA) to identify distinct student ideas in a set of written responses to a conceptual physics question, with the goal of significantly expediting the process of characterizing student ideas. We preliminarily test the LDA approach by applying the algorithm to a collection of introductory physics student responses to a conceptual question about circuits, specifically attending to whether it is useful for characterizing instructionally-relevant student ideas. We find that for a large enough collection of student responses (N ≈ 500), LDA can be useful for characterizing the ideas students used to answer conceptual physics questions. We discuss some considerations that researchers may take into account as they interpret the results of the LDA algorithm for characterizing student's physics ideas.

poster

Students' sense of belonging contributes to success at universities. Studies hint that students' sense of belonging in their introductory STEM courses may be tied to the course structures. In this study, we compared students' sense of belonging and letter grade between four semesters of an introductory calculus-based electricity and magnetism course. The course structures varied throughout the four semesters with hybrid versus in-person instruction and midterm exams versus quizzes for assessment, but all implementations used research-based instruction. Here, we compare students' sense of belonging and letter grades within these different course structures. Students expressed a stronger sense of belonging and earned higher letter grades with the lower-stakes quiz structure than in prior semesters with midterm exams. However, students' sense of belonging did not change when attending hybrid compared to in-person.

Investigating the effects of course structure on students' sense of belonging and course performance

Many of the activities and cognitive processes that physicists use while solving problems are "invisible" to students, which can hinder their acquisition of important expert-like skills.  Whereas the detailed calculations performed by researchers are often published in journals and textbooks, other activities such as those undertaken while planning how to approach a problem are rarely discussed in published research.  Hence, these activities are especially hidden from students.  To better understand how physicists solve problems in their professional research, we leveraged the framework of cognitive task analysis to conduct semi-structured interviews with theoretical physicists (N=11).  Here we elucidate the role of planning and preliminary analysis in theorists' work.  Theorists described using a variety of activities in order to decide if their project was doable while also generating possible solution paths.  These actions included doing cursory calculations, reflecting on previous knowledge, gaining intuition and understanding by studying prior work, and reproducing previous results.  We found that theorists typically did not pursue projects unless they had a clear idea of what the outcome of their project would be, or at least knew that they would be able to make progress on the problem.  Thus, this preliminary design and analysis phase was highly important for theorists despite being largely hidden from students.  We conclude by suggesting potential ways to incorporate our findings into the classroom to give students more numerous opportunities to engage in these expert-like practices.

Making expert cognitive processes visible: planning and preliminary analysis in theoretical physics research

Although social identity, as it relates to physics identity, is not a new area of research, disability identity is woefully understudied in physics. Disability is political, and ableism is real, ordinary, and pervasive in physics. Disabled people find themselves as the subject of sociopolitical violence both in and out of the classroom, simply for existing in their body. In a climate charged with politicality and identity discourse in the classroom,
it is surprising that a framework for understanding disability and the transactional relationship disabled persons have with their environment in a political way has yet to be used in physics education. In this paper, I introduce Political Disability Identity (Putnam,2005) as a framework for studying disability identity in Physics Education Research. I outline the domains of Political Disability Identity, and the underlying themes which should guide research using this framework. Finally, I call for its use both in the physics classroom, and in Physics Education Research.

Introducing political disability identity as a framework for studying disability in physics

The field of quantum computing is presently gaining tremendous attention from governments, researchers, engineers, academics, and investors. There is an ongoing push for the development of quantum computers, advances in information technology, and the development of a quantum workforce. This needs to be accompanied by the development of quantum computing courses and curricula and the development and adoption of evidence-based materials and pedagogies to support the education of the next generation of quantum information scientists. We have introduced a course in Quantum Computing and have begun to research student understanding of topics in this field. Our goal is to develop evidence-based materials for the course. We did a series of interviews to identify students' strengths and difficulties in these topics.  We report on the results of these interviews and our initial work on the development of supplementary materials for the course.

Investigating Students' Strengths and Difficulties in Quantum Computing

Collaborative learning with peers can lead to students learning from each other and solving physics problems correctly not only in situations in which one student knows how to solve the problems but also when none of the students can solve the problems alone. In the latter situation, students are co-constructing knowledge that helps them solve the problems, while in the former, one student helps the other construct knowledge. In this study, we investigated student learning measured by student performance on a validated quantum mechanics survey and frequencies of construction and co-construction of knowledge when students first worked individually after lecture-based instruction in relevant concepts and then worked with peers during class without receiving any feedback from the course instructor. We find that construction of knowledge consistently occurred at a high rate during peer collaboration. However, rates of co-construction were more varied. High rates of co-construction were generally achieved when approximately half of the students knew the correct answers initially.  We also conducted an analysis of some of the survey questions with high rates of co-construction to gain insight into what students converged on after peer interaction and what types of difficulties were reduced. Our findings can be valuable for physics instructors who want to provide in-class and out-of-class opportunities for peer collaboration, e.g., in their quantum mechanics courses.

Peer interaction facilitates co-construction of knowledge related to quantum mechanics formalism and postulates

A previous study shows that incentivizing students to correct mistakes on unit exam problems within an upper-division quantum mechanics course improved students' problem-solving efforts on those same problems in a final exam environment, relative to a comparison group of students who were not incentivized. We attempt to replicate the quantitative portion of this study within a first-semester upper-division electromagnetism course, specifically examining students' invoking correct concepts and applying those concepts correctly. A statistical comparison of students who accepted the offer to rework unit exam problems for partial credit, versus students who declined the offer, demonstrates a better improvement for students who chose to rework relative to students who declined. As the results suggested that unit exam performance might provide a covariate within the comparison of choice to rework between groups, the results were analyzed using ANCOVA; they showed a result for one of the three featured exam problems that may be hidden by regular 1-way ANOVA analysis of pre-post normalized gains. The authors also investigate whether a focus on conceptual understanding versus a focus on algorithmic calculation for different exam problems affects the degree of benefit for students who rework problems. Results appear to show that incentivization works more specifically for invoking correct concepts on a primarily conceptual problem, and more specifically for applying concepts correctly on a primarily algorithmic problem. Future plans include a more complete analytical framework using think-aloud protocol interviews for students from the sample, as well as more statistical detail to determine the interaction between unit exam score and choice to rework problems.

Reworking exam problems to incentivize improved performance in upper-division electrodynamics

Recent years have seen a movement within the research-based assessment development community towards item formats that go beyond simple multiple-choice formats. Some have moved towards free-response questions, particularly at the upper-division level; however, free-response items have the constraint that they must be scored by hand.  To avoid this limitation, some assessment developers have moved toward formats that maintain the closed-response format, while still providing more nuanced insight into student reasoning.  One such format is known as coupled, multiple response (CMR).  This format pairs multiple-choice and multiple-response formats to allow students to both commit to an answer in addition to selecting options that correspond with their reasoning. In addition to being machine-scorable, this format allows for more nuanced scoring than simple right or wrong.  However, such nuanced scoring presents a potential challenge with respect to utilizing certain testing theories to construct validity arguments for the assessment.  In particular, Item Response Theory (IRT) models often assume dicotomously scored items.  While polytomous IRT models do exist, each brings with it certain constraints and limitations.  Here, we will explore multiple IRT models and scoring schema using data from an existing CMR test, with the goal of providing guidance and insight for possible methods for simultaneously leveraging the affordances of both the CMR format and IRT models in the context of constructing validity arguments for research-based assessments.

Methods for utilizing Item response theory with Coupled, Multiple-Response assessments

Researchers across all scientific disciplines routinely face ethical decisions in their work, from addressing conflicts of interest to deciding whether and how to make data available for reproducibility. To help strengthen their ethical reasoning skills, they are encouraged to take online training programs like the CITI program. Ethics training is insufficient for improving ethical behavior. Better understanding of how scientists make decisions and reason about ethics is needed. To develop that understanding, we need expanded, asset-based measures of ethical reasoning that can be applied to open-ended responses and discussions. As part of a year-long intervention on a group of fifteen scientists' value-based reasoning, we conducted pre/post interviews that included open-ended questions about ethical scenarios. For this paper, we explore an application of three theories of ethical and stakeholder reasoning to those answers, and determine that we can use them to examine quality, principles, and subjects of their reasoning in open responses.

Examining Physicists' Ethical Reasoning: A Case Study

We identify three conceptual resources that introductory physics students in our sample commonly use when reasoning microscopically about thermal physics topics: 1) differences will eventually even out, 2) macroscopic changes connect to microscopic collisions, and 3) when something is hotter (colder), its molecules are moving faster (slower). We report the prevalence of these resources, as well as the prevalence of microscopic thinking, in 624 written responses to three different heat and temperature questions administered to introductory physics students at four different colleges and universities. This work complements past research identifying common student difficulties in using microscopic models for heat and temperature, and it adds to the small but growing body of literature that focuses on student resources for heat and temperature, identifying ideas in student thinking that are sensible and continuous with formal physics. By reporting common student resources, we aim to assist instructors in noticing, appreciating, and building on student ideas during introductory thermal physics instruction.

Identifying student resources for reasoning microscopically about heat and temperature

Research-Practice Partnership (RPP) is recognized as a valuable tool for generating actionable research and improving educational practices by involving practitioners and researchers in both research and the implementation of research findings. However, creating and maintaining such partnerships is challenging; it requires time and financial resources, and a team with diverse expertise and high commitment. Despite their wide recognition in educational fields, RPPs are under-studied in physics education research and literature in informal physics programs is sparse. In this paper, we present a case study of RPP between a physics education researcher and an informal physics practitioner. We describe the development and maintenance of this partnership and use qualitative and ethnographic methods to investigate practitioner-driven concerns about self-evaluation practices and attendance in the program under study, an informal conference for physics teachers. We found evidence that surveys and ethnographic interviews provided complementary data: surveys allowed the practitioner to learn about their audience's motivations, needs, and perception of the program's value. Ethnographic interviews were particularly useful as a means to elicit actionable ideas for how conference organizers might better support teacher attendance. In addition, participating in the RPP provided the practitioner with skills that they can carry forward into their future assessment work and that this partnership yielded actionable insights beneficial to both practitioners and audience. Our research team was able to overcome inherent challenges of RPP by exploring each side's language and cultural norms, and maintained a long-term partnership despite limitations. Our experience suggests that RPPs in informal physics spaces face many of the same challenges as those conducted in formal education settings, and that best practices documented for formal education RPPs will also be of benefit to them.

Premium content

Developing a natural language processing approach for analyzing student ideas in introductory calculus-based physics

Next from PERC SM 22

Investigating the effects of course structure on students' sense of belonging and course performance

Similar lecture

Navigating group work in computational projects:Case studies using self-assessments