United States

Large language models can generate feedback on free-form student writing, but it is unclear whether such feedback is correct and pedagogically useful. We evaluate LLM-generated feedback on 65 undergraduate proof-writing exercises using Hattie and Timperley&#39;s feedback framework and a grade agreement metric, comparing two models (GPT-4.1, GPT-5) under two workflow configurations graded by two independent LLM evaluators. A mark-scheme-augmented workflow improves grade correlation with human experts for both models, and its precomputed mark schemes allow instructors to audit the system before deployment. GPT-5 produces higher-quality feedback across all dimensions. The metrics we collect give some evidence that in the setting studied, feedback quality is high, and several sanity checks on our experiments support this finding. However, providing meaningful self-regulation support and controlled tests with students remain to be done. The results in this contribution show that feedback theory provides a useful lens for evaluating automated mathematical feedback.

ACL 2026

Evaluating LLM-Generated Formative Feedback for Undergraduate Mathematics Through the Lens of Feedback Theory

undergraduate mathematics teaching

feedback

Welcome to the **largest national, multi-specialty research** event featuring research from nearly 700 medical students, residents, fellows, and international medical graduates.

Use the buttons below or navigation menu to access the event.
 


[![](https://assets.underline.io/uploads/markdown_image/1/image/3e33a8c754cc49610b30950617d813e7.png)](https://underline.io/events/434/sessions?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/551404d429c4e274f9a54cbdd7081918.png)](https://underline.io/events/434/posters?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/677d92f566b880cebd36905fc738fc10.png)](https://underline.io/events/434/lounge)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/c7392833c7261c3874c5b43fa2f4df19.png)](https://underline.io/events/434/help-desk)
 

#### Event Details

The 2023 American Medical Association Research Challenge includes a poster symposium hall and a semifinals hall with the top scored research. **Log in at any time throughout the event to:**

* **View posters and presentations** on a variety of topics and specialties
* **Score posters in the semifinals** to help decide the top five finalists (AMA member exclusive opportunity)
* **Connect with participants** and provide feedback to help advance their research
* **Attend education sessions** with physician experts on key research topics

At the conclusion of this event, the AMA will announce the top five finalists who will advance to the AMA Research Challenge finals, present their research to an elite panel of judges, and compete to win a $10,000 grand prize sponsored by Laurel Road. 

### Grand Prize Sponsor

AMA would like to thank Laurel Road for sponsoring a $10,000 grand prize for the winner of the AMA Research Challenge. Laurel Road is not involved in the judging or scoring process to select the winner. The winner will be announced on February 4, 2024 during the AMA Research Challenge finals. Payment of the grand prize will be directly from Laurel Road. See the AMA Research Challenge Official Rules for terms and conditions.

![](https://assets.underline.io/uploads/markdown_image/1/image/611e331d4a0eba0920f240cfc04b4d77.png)

**There is no registration fee for this event! **

Register to explore research in a variety of topics. AMA members get exclusive access to score posters in the semifinals.

**Please use this link [here](https://www.ama-assn.org/about/events/ama-research-challenge) to register for free.**

Registration Is Required

AMA Research Challenge 2023

Join the AMA Research Challenge poster symposium and semifinals to explore top research posters, score posters in the semifinals (an AMA member exclusive opportunity) and attend education sessions with experienced physicians to learn tips to advance your research and career. 

Large language models can generate feedback on free-form student writing, but it is unclear whether such feedback is correct and pedagogically useful. We evaluate LLM-generated feedback on 65 undergraduate proof-writing exercises using Hattie and Timperley's feedback framework and a grade agreement metric, comparing two models (GPT-4.1, GPT-5) under two workflow configurations graded by two independent LLM evaluators. A mark-scheme-augmented workflow improves grade correlation with human experts for both models, and its precomputed mark schemes allow instructors to audit the system before deployment. GPT-5 produces higher-quality feedback across all dimensions. The metrics we collect give some evidence that in the setting studied, feedback quality is high, and several sanity checks on our experiments support this finding. However, providing meaningful self-regulation support and controlled tests with students remain to be done. The results in this contribution show that feedback theory provides a useful lens for evaluating automated mathematical feedback.

workshop paper

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. 
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) 
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

**Online Registration Form**: https://acl.swoogo.com/acl2026

Registration Required

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

Effective tutoring requires distinguishing optimal (expert-path), valid-alternative (valid but suboptimal), and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) yet untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS for scalable feedback, evaluating their diagnostic precision is essential. We benchmark seven LLMs as feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution-feedback pairs \footnote{\url{https://github.com/anonymouspaper64-debug/anonymous_paper.git}} and three prompt-level feedback conditions. Our analysis show that models achieve near-ceiling performance on optimal steps but systematically over-reject valid-alternative reasoning and over-validate incorrect solutions, precisely where adaptive tutoring matters most. These failure modes persist regardless of solution context or model capability, indicating architectural rather than informational limits. Accurate classification does not reliably translate into pedagogically actionable feedback, exposing a gap between diagnostic judgment and instructional effectiveness. LLMs thus cannot resolve the assistance dilemma without ITS-grounded mechanisms; they are better positioned within hybrid systems that combine LLM flexibility with ITS student modeling.

Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Algorithmic problem-solving is often taught through course-specific representations, conventions, and examples, making it important to provide instructional support that remains closely aligned with the curriculum while also being timely and available when students need help. In this paper, we present KITE (Knowledge-Informed Tutoring Engine), a Retrieval-Augmented Generation (RAG)-based intelligent tutoring system designed to serve as a classroom teaching assistant by supporting learning through dialogue in university AI courses. KITE uses an intent-aware Socratic response strategy to tailor its support to different student needs, responding with targeted hints, guiding questions, and progressive scaffolding intended to support independent reasoning. To keep responses aligned with course content, KITE uses a multimodal RAG pipeline that retrieves relevant information from lecture slides and course PDFs. We evaluate KITE using three forms of assessment: RAGAS-based metrics for response grounding and quality, human evaluation of pedagogical quality, and a simulated student pipeline in which a weaker language model interacts with KITE across two-turn dialogues and produces revised answers after receiving feedback. Preliminary results suggest that KITE produces contextually grounded and pedagogically appropriate responses, with especially promising performance on algorithm validation, debugging, and tracing tasks. This work contributes a replicable architecture and evaluation framework for intent-aware, curriculum-grounded tutoring systems in STEM education.

Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving in AI Education

Grammatical error correction using large language models often suffers from the over-correction issue. To mitigate this, we propose a training-free inference method that performs edit-level majority voting over multiple candidates generated by a single model, without requiring model modifications or additional training. Across nine benchmarks covering English, Czech, German, Ukrainian, Korean, Hindi, and Romanian, the proposed method outperforms both greedy and MBR decoding in most cases. Moreover, it yields stable correction quality regardless of the instruction prompts used. The code will be released upon acceptance.

Edit-level Majority Voting Mitigates Over-Correction in LLM-based Grammatical Error Correction

Phonics stories are essential for early literacy, requiring controlled repetition of grapheme-phoneme (GP) patterns while maintaining simplicity, suitability, and quality. Generating such texts poses a challenge for large language models (LLMs), which must balance multiple phonological and pedagogical constraints. We evaluate six LLMs in a zero-shot setting across 16 prompt configurations, producing 8,688 outputs and 39,096 stories. Outputs are assessed using a multi-dimensional framework covering phonological alignment, developmental lexical appropriateness, readability, and narrative quality. Results show that while LLMs generate highly readable and age-appropriate text, they exhibit variability in phoneme control and narrative coherence. Prompt design significantly affects performance, revealing trade-offs between focusing on multiple phonological, linguistic, and pedagogical constraints, while model choice also leads to significant differences. These findings highlight the challenges of controllable educational text generation and the importance of prompt design in balancing instructional objectives. We release our prompts, generated stories, and evaluation framework to support future work in phonics-based story generation for early readers.

Zero Shot Phonics: Evaluating Constraint-Adherent Phonics Story Generation in Large Language Models

We present a framework for evaluating adaptive personalization of educational reading materials with theory-grounded simulated learners. Unlike typical intelligent tutoring systems that adapt questions or feedback, we treat reading as the primary intervention and use question answering only as an observation channel for Bayesian Knowledge Tracing (BKT). This enables controlled comparison of LLM-powered adaptive and non-adaptive reading policies before classroom deployment.
The framework links open educational content to a shared ontology of learning objectives and knowledge components, which is used to generate aligned reading--assessment pairs targeting one objective at a time. Simulated learners update their knowledge through a comprehension-and-memory process that models encoding, integration with prior knowledge, and misconception revision.
The learner model combines established theories of reading with constrained answer selection, ensuring responses are generated only from information the learner has plausibly retained. Together, these components provide an interpretable offline testbed for studying whether adaptive reading improves learning outcomes.

Evaluating Adaptive Personalization of Educational Readings with Simulated Learners

Constructive, personalized, and timely feedback is essential to student learning. However, providing such feedback in large classes remains a major challenge. Large language models (LLMs) offer alternatives to support instructors by generating relevant feedback that highlights both student strengths and areas for improvement. Nevertheless, most existing LLM-based feedback systems rely on proprietary APIs and are often tailored to specific tasks, which limits their accessibility, flexibility, and applicability in resource-constrained educational settings. In this study, we investigate the potential of two open-source LLMs to support automated feedback generation for varied forms of student assignments across three disciplines (e.g., programming assignments, essays, and mathematics problems). We evaluate two prompting strategies (unified and multi-agent) across these domains and use human judgment criteria to assess feedback quality. Through this analysis, we examine whether open-source models can serve as practical, scalable alternatives to proprietary API-based systems for developing freely accessible feedback-generation tools. Our results provide initial evidence for cross-domain applicability, rather than establishing full generalizability. We conclude that open-source models feedback quality varies significantly across domains, and that the effectiveness of prompting strategies depends on the nature of the task.

Toward Cross-Domain Automated Feedback: A Comparative Evaluation of Open-Source Models across Diverse Student Assessment Types

We propose a two-stage framework for vocabulary difficulty prediction for L2 learners, addressing representation-dependent variation and learner-specific effects such as L1 transfer. Stage 1 constructs diverse representations using multiple pretrained encoders with different pooling and prediction strategies, capturing complementary aspects of lexical and contextual complexity. Stage 2 models systematic residual errors using psycholinguistic and cross-lingual features, enabling explicit correction of learner-dependent biases. Experiments on the BEA 2026 Shared Task (KVL dataset) show that the proposed approach outperforms strong baselines, achieving substantial improvements in both RMSE and correlation. These results demonstrate the effectiveness of combining representation diversity with structured residual calibration for accurate difficulty prediction.

A Two-Stage Framework for L1-Aware Vocabulary Difficulty Prediction with Representation Diversity and Residual Calibration

What makes a word difficult to learn, and how does the difficulty depend on the learner's native language? We computationally model vocabulary difficulty for English learners whose first language is Spanish, German, or Chinese with gradient-boosted models trained on familiarity, meaning, surface-form, and cross-linguistic transfer features. Using Shapley values, we identify the importance of feature groups. Word familiarity is the dominant feature group shared by all three languages. However, Spanish- and German-L1 predictions rely additionally on orthographic transfer and predictions for easy words generally rely on one strong clue, either familiarity or transfer. This transfer mechanism is unavailable to Chinese learners, whose difficulty is shaped by a combination of familiarity and surface features alone. Our models provide interpretable L1-tailored difficulty estimates applicable to the design of vocabulary curricula and readability measures.

What Makes a Word Hard to Learn? Modeling L1 Influence on English Vocabulary Difficulty

In this paper, we present our team submissions to the BEA 2026 Shared Task 2:
Rubric-based Short Answer Scoring for German. We systematically explored how fine-tuned language models can be reliably employed for short answer scoring, for which three aspects turn out to be particularly beneficial: supplementing the finetuning process with generated domain expertise, restructured rubrics, and thinking traces. To increase the robustness of the scoring, we combine distinct approaches in an ensemble. Our best submissions finished in first place across all tracks, indicating promise for the further application of these strategies in automatic scoring.

IWM-DKM at BEA 2026 Shared Task 2: Supplementing Supervised Fine-Tuning for Rubric-Based Short Answer Scoring

In this paper, we present the RETUYT-INCO participation at the BEA 2026 shared task "Rubric-based Short Answer Scoring for German". Our team participated in track 1 (Unseen answers three-way), track 3 (Unseen answers two-way) and track 4 (Unseen questions two-way). Since these tracks required scoring short student answers using specific rubrics, we looked for ways to handle the changing nature of the task. We created a method called Meta-prompting. In this approach, an LLM creates a custom prompt based on examples from the Train set. This prompt is then used to grade new student answers. Along with this method, we also describe other approaches we used, such as classic machine learning, fine-tuning open-source LLMs, and different prompting techniques. According to the official results, our team placed 6th out of 8 participants in Track 1 with a QWK of 0.729. In Track 3, we secured 4th place out of 9 with a QWK of 0.674, and we also placed 4th out of 8 in Track 4 with a QWK of 0.49.

Downloads

Next from ACL 2026

Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Similar lecture

"Let's Unpack My Feedback!" - An Approach to Enhance Medical Students' Peer Feedback