United States

Assessment scores can be used to inform instructors about their students’ learning, ideally to inform subsequent course modifications to better facilitate students’ learning. However, explicit course modifications require interpretations about what assessment scores convey instructors about the extent to which students meet the learning goals instructors have for their students, i.e., actionable feedback for instructors (hereinafter feedback). As part of developing a standardized assessment for upper-division thermal physics, we articulate a theory-based approach to generating feedback that goes beyond simply reporting students’ scores. The extent to which students met the learning goals instructors have for them guides our feedback generation. In this talk/poster, we present our theory-based approach to feedback design by integrating effective feedback practices that promote instructors’ agency. This work emphasizes the development of feedback for instructors to better facilitate students’ learning in physics classrooms. 


AAPT SM 21

Research-based Assessment Feedback for Instructors

assessments

upper-division

feedback

AAPT® is a strong professional physics science society dedicated to the pursuit of excellence in physical science education.

Winter and Summer National Meetings:
These National Meetings, held each winter and summer, are opportunities for members, colleagues, and future physicists from around the world to:

- participate in physics workshops
- meet and greet other physics educators
- form networks nationally and locally
- engage exhibitors and learn about the latest physics resources
- discuss innovations in teaching methods
- share the results of research about teaching and learning.

If you're not able to log in, you're most likely using **incorrect login credentials**. Double-check that you are using the same email and password that you used to register for the conference, submit your abstract or access your AAPT membership. There may also be **an issue with your AAPT meeting registration**. Please email aapt2021@underline.io and programs@aapt.org and we will work quickly to resolve this issue.

AAPT WM 21

technical paper

Join AAPT and Physics Educators worldwide in developing and strengthening your professional connections by attending the 2021 AAPT Virtual Summer Meeting. AAPT meetings are a great way to reflect on your teaching, recommit to best practices and re-energize your career with new research, products, and techniques.

Listen to internationally known plenary speakers, attend sessions, workshops, and special activities. Attending the virtual summer meeting is an excellent opportunity to share your expertise and learn from and network with leaders in the field.

Please log in with your AAPT account or register for the event if you haven't already.

If you are registered please note that you need to be logged in with your AAPT email / credentials in order to get access.

Although the technology that enables the modern Information Age looks very complicated, it has several basic physics ideas at its core that are simple enough to explain at the high school level. One can use the same simple physics arguments to explain the fundamental limits of current information technologies and the bottlenecks we are already experiencing when we try to send, receive, or process information. Perhaps most importantly, after learning these basic physics principles, the students should be able to think about ways to overcome the challenges and propose their own solutions.

Information technology: challenges and bottlenecks

Many events can drive change but one hopes that a tragic event is not a primary motivator. Varied ideas about change regarding the graduate program circulated the Department of Physics and Astronomy at the University of Utah prior to 2017. But along with those ideas came resistance to change among the faculty, little sense of urgency, and an inability to settle or agree on the best course of action. However, when tragedy struck in the fall of 2017, change occurred at a much more rapid pace. In this talk we discuss the different aspects of the change process that transpired: spearheading events, decisions, departmental reaction, long-term impacts, challenges, guiding messages, and sense of community. One key finding we present is that, while policy change was widely implemented and accepted, there is still some resistance to cultural change within the department.

Graduate Reform in Physics and Astronomy: Tragedy, Policy, and Culture

In physics discipline, women are globally underrepresented at the graduate level as well as at the academic and top leadership positions in higher education institutes. It is argued that multiple stakeholders, including academic departments, management at higher education institutes, government, and scientific societies need to work in tandem on developing effective programs to recruit, retain, and advance women in physics in all under-represented areas. My presentation describes one such national programmatic effort - “Vigyan Vidushi” (“a learned woman scientist”) recently launched in India by the Tata Institute of Fundamental Research and Homi Bhabha Centre for Science Education. It is a three-week summer school for the postgraduate physics degree pursuing women with a goal to enhance their preparedness towards graduate education and bring awareness of possible hindrances in their career paths. The presentation will explore specific features of the Vigyan Vidushi Summer School in a case study model to understand its widescale utility and impact.

Vigyan Vidushi- A programmatic effort towards addressing under-representation

Advisors and graduate students are sometimes stuck and not moving their research forward. We use Deliberate Innovation(DI) techniques to understand the specific situations centered around research that graduate students and their advisors find themselves in that may be problematic. A critical element of this is determining whether advisors and advisees show indifference in particular situations. We understand indifference by noticing what people are and are not drawn to in the research setting, and will be focusing on ways in which we have seen it show up. Finding this indifference allows us to know where to focus our attention, with the ultimate goal of finding situations in which we can help graduate students and advisors get unstuck and move their research forward.

Using Deliberate Innovation to Understand Indifference in Graduate Advisor/Advisee Relationships

Abstract. Large Hadron Collider (LHC) and Higgs boson discovery reached audience levels unprecedented for a particle physics event. The LHC can create almost a billion proton-proton collisions per second. A non-zero Higgs Field was determined to alter the properties of these particles. Understanding how this field works is one of the central particle physics research areas and justified LHC construction. From a science education research perspective, how the particulate nature of matter is introduced in the classroom is an important. The purpose of this paper explores an alternate context for which subatomic particles may be introduced to K-12 students—space weather. NASA solar probes have studied space weather and the behavior of solar energetic particles since the 1960s. The perilous impact of space weather on global commerce has motivated greater research into its particulate nature and behavior.

Instructional Context for Astrophysics: Space Weather

The Next Generation Physics Science and Everyday Thinking (NGP) curriculum materials [1] were developed to give students learning experiences that integrate all three dimensions of the Next Generation Science Standards (NGSS) [2]. Analysis of a subset of the materials with EQUIP rubric [3] confirmed this alignment and integration. However, analysis of the test bank questions that accompany the materials (using the 3D-LAP [4]) revealed that most items being used for assessment did not integrate the three dimensions. We are therefore developing assessment tasks that follow the pattern of NGP activities in integrating all three dimensions. In this presentation, we will describe the path that led us to this point, the process we are using to develop these assessment tasks, and preliminary results.

[1] https://nextgenpet.activatelearning.com
[2] NGSS Lead States. (2013). Next Generation Science Standards: For States, By
States. Washington, DC: The National Academies Press.
[3] https://www.nextgenscience.org/resources/equip-rubric-science
[4] Laverty, J. T., et. al., (2016). PloS ONE,11(9): e0162333.


Developing NGSS-aligned assessment tasks for the Next Gen PET curriculum

The Colorado Learning Attitudes about Science Survey (CLASS) assesses students’ perceptions of physics through self-reported agreement with attitudinal statements. Previous work shows that CLASS results may be subject to response-shift bias, in which knowledge gained during an intervention causes learners’ perceptions of their pre-instructional state to become more novice-like. We explored possible sources of this response-shift bias in the CLASS by asking students to (1) complete the survey twice at the beginning of the semester (providing their pre-instruction responses and expected end-of-semester responses) and twice at the end of the semester (providing their post-instruction responses and retrospective beginning-of-semester responses), and (2) comment on their concurrent responses. By comparing beginning-of-semester and end-of-semester comments with survey shifts from a broad sample of students, we illuminate an underexplored dimension of student thinking. Preliminary results show that our students expect their perceptions of the applicability of physics will grow directly from acquiring physics knowledge.

Sources of Response-Shift Bias in the CLASS’s Real-World Connection

Educators and researchers often use research-based assessments to measure the efficacy of their courses. Limited resources exist, however, for interpreting assessment results. We will present analyses and representations created with data from more than 500 introductory physics courses that provide a context for interpreting assessment results. The data set comes from the online Learning About STEM Student Outcomes (LASSO) platform and the scientific literature. The representations include scatterplots of pretest and posttest scores and distributions of effect sizes, Cohen’s d with Hedges’ correction. These representations show how courses compare to the larger database before and after instruction. The results show that common rules of thumb for effect sizes are not useful for interpreting effect sizes on research-based assessments. Excel and R files are available at https://tinyurl.com/stemequityresources for readers to create their own visualizations overlaying their course data on the larger data set.

Tools for Identifying Effective Courses

As a method to shorten the test time of the Force Concept Inventory (FCI), we suggest the use of Computerized Adaptive Testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. In so doing, the test length can be significantly shortened. As a step to develop a CAT–based version of the FCI (FCI-CAT), we examined the optimal test length of the FCI-CAT such that accuracy and precision ([which we measure in terms of root-mean-square error (RMSE)] of Cohen’s d would be comparable to that of the full FCI for a given class size. We conducted a Monte Carlo simulation to analyze how the RMSE of Cohen’s d depend upon the test-length. As a result, we found, that for a class size of 40, we may reduce the test length of the FCI-CAT to 15-19 items, thereby reducing the test time of the FCI to 50% - 63%, with an accompanying decrease in accuracy and precision of only 5-10%.

Optimizing the length of computerized adaptive testing for the FCI

The presented study investigates the use of mathematical procedures to solve items in a kinematic context. For this purpose, a validated test instrument by Ceuppens et al. (2019) was administered to high school students which included pairs of items in mathematics and kinematics contexts that are isomorphic to each other, i.e., have the same surface features. 
While solving the tasks, eye movements of N=35 learners were recorded by a stationary eyetracker. For selected items, we demonstrate that gaze behavior and thus visual strategies differ between solving the mathematical and kinematic items. 
The results show that specific solution strategies are induced by the task context, which means that procedures that can be applied in the mathematical context cannot be transferred to the kinematic context. The talk presents the study and discuss the results in details.


Premium content

Downloads

Next from AAPT SM 21

Information technology: challenges and bottlenecks

Similar lecture

Designing Research-based Assessment Feedback for Instructors