Austria

This paper presents automatic evaluation systems for assessing the pedagogical capabilities of LLM-based AI tutors. Drawing from a shared task, our systems specifically target four key dimensions of tutor responses: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. These dimensions capture the educational quality of responses from multiple perspectives, including the ability to detect student mistakes, accurately identify error locations, provide effective instructional guidance, and offer actionable feedback. We propose GPT-4.1-based automatic evaluation systems, leveraging their strong capabilities in comprehending diverse linguistic expressions and complex conversational contexts to address the detailed evaluation criteria across these dimensions. Our systems were quantitatively evaluated based on the official criteria of each track. In the Mistake Location track, our evaluation systems achieved an Exact macro F1 score of 58.80\% (ranked in the top 3), and in the Providing Guidance track, they achieved 56.06\% (ranked in the top 5). While the systems showed mid-range performance in the remaining tracks, the overall results demonstrate that our proposed automatic evaluation systems can effectively assess the quality of tutor responses, highlighting their potential for evaluating AI tutor effectiveness.

ACL 2025

K-NLPers at BEA 2025 Shared Task: Evaluating the Quality of AI Tutor Responses with GPT-4.1

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

workshop paper

As AI's presence in educational environments grows, it becomes critical to evaluate how its feedback may impact students' learning processes. Pedagogical theory, with decades of effort into understanding how human instructors give good-quality feedback to students, may provide a rich source of insight into feedback automation. In this paper, we propose a novel architecture based on pedagogical-theory feature extraction from the conversation history and tutor response to predict pedagogical guidance on MRBench. Such features are based on Brookhart's canonical work in pedagogical theory, and extracted by prompting the language model LearnLM. The features are then used to train a random-forest classifier to predict the ‘providing guidance' dimension of the MRBench dataset. Our approach ranked 8\textsuperscript{th} in the dimension's leaderboard with a test Macro F1-score of ~0.54. Our work provides some evidence in support of using pedagogical theory qualitative factors treated separately to provide clearer guidelines on how to improve low-scoring intelligent tutoring systems. Finally, we observed several inconsistencies between pedagogical theory and MRBench's inherent relaxation of the tutoring problem implied in evaluating on a single-conversation basis, calling for the development of more elaborate measures which consider student profiles to serve as true heuristics of AI tutors' usefulness.

IALab UC at BEA 2025 Shared Task: LLM-Powered Expert Pedagogical Feature Extraction

We present MSA-MathEval, our submission to the BEA 2025 Shared Task on evaluating AI tutor responses across four instructional dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. Our approach uses a unified training pipeline to fine-tune a single instruction-tuned language model across all tracks, without any task-specific architectural changes. To improve prediction reliability, we introduce a disagreement-aware ensemble inference strategy that enhances coverage of minority labels. Our system achieves strong performance across all tracks, ranking 1st in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location. These results demonstrate the effectiveness of scalable instruction tuning and disagreement-driven modeling for robust, multi-dimensional evaluation of LLMs as educational tutors.

MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

In light of the growing adoption of large language models (LLMs) as educational tutors, it is crucial to effectively evaluate their pedagogical capabilities across multiple dimensions. Toward this goal, we address the Mistake Identification sub-task of the BEA 2025 Shared task, aiming to assess the accuracy of tutors in detecting and identifying student errors. We experiment with several LLMs, including GPT-4o-mini, Mistral-7B, and Llama-3.1-8B, evaluating them in both zero-shot and fine-tuned settings. To address class imbalance, we augment the training data with synthetic examples, targeting underrepresented labels, generated by Command R+. Our GPT-4o model finetuned on the full development set achieves a strict macro-averaged F1 score of 71.63%, ranking second in the shared task. Our work highlights the effectiveness of fine-tuning on task-specific data and suggests that targeted data augmentation can further support LLM performance on nuanced pedagogical evaluation tasks.

TutorMind at BEA 2025 Shared Task: Leveraging Fine-Tuned LLMs and Data Augmentation for Mistake Identification

With the rapid advancement of global digitaliza-
tion, users from different countries increasingly
rely on social media for information exchange.
In this context, multilingual multi-label emo-
tion detection has emerged as a critical research
area. This study addresses SemEval-2025 Task
11: Bridging the Gap in Text-Based Emotion
Detection. Our paper focuses on two sub-tracks
of this task: (1) Track A: Multi-label emo-
tion detection, and (2) Track B: Emotion in-
tensity. To tackle multilingual challenges, we
leverage pre-trained multilingual models and
focus on two architectures: (1) a fine-tuned
BERT-based classification model and (2) an
instruction-tuned generative LLM. Addition-
ally, we propose two methods for handling
multi-label classification: the base method,
which maps an input directly to all its cor-
responding emotion labels, and the pairwise
method, which models the relationship between
the input text and each emotion category indi-
vidually. Experimental results demonstrate the
strong generalization ability of our approach in
multilingual emotion recognition. In Track A,
our method achieved Top 4 performance across
10 languages, ranking 1st in Hindi. In Track B,
our approach also secured Top 5 performance
in 7 languages, highlighting its simplicity and
effectiveness1
.

JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models

This paper presents a multi-step zero-shot system for SemEval-2025 Task 1 on Advancing Multimodal Idiomaticity Representation (AdMIRe). The system employs two state-of-the-art multimodal language models, Claude Sonnet 3.5 and OpenAI GPT-4o, to determine idiomaticity and rank images for relevance in both subtasks. A hybrid approach combining o1-preview for idiomaticity classification and GPT-4o for visual ranking produced the best overall results. The system demonstrates competitive performance on the English extended dataset for Subtask A, but faces challenges in cross-lingual transfer to Portuguese. Comparing Image+Text and Text-Only approaches reveals interesting trends and raises questions about the role of visual information in multimodal idiomaticity detection.

daalft at SemEval-2025 Task 1: Multi-step Zero-shot Multimodal Idiomaticity Ranking

We developed an ensemble learning system that integrates multiple transformer-
based models, including RoBERTa-large and DeBERTa-v3-large. These models were trained using diverse data augmentation strategies, both lightweight and intensive, to increase generalization and robustness. The ensemble approach allowed us to
combine the strengths of different models through weighted soft voting.

Anastasia at SemEval-2025 Task 9: Subtask 1, Ensemble Learning with Data Augmentation and Focal Loss for Food Risk Classification.

In this paper we present our participation in Subtask 2 of SemEval-2025 Task 10, focusing on the identification and classification of narratives in news of multiple languages, on climate change and the Ukraine-Russia war. To address this task, we employed a Zero-Shot approach using a generative Large Language Model (LLM) without prior training on the dataset. Our classification strategy is based on two steps: first, the system classifies the topic of each news item; subsequently, it identifies the sub-narratives directly at the finer granularity. We present a detailed analysis of the performance of our system compared to the best ranked systems on the leaderboard, highlighting the strengths and limitations of our approach.

UNEDTeam at SemEval-2025 Task 10: Zero-Shot Narrative Classification

This paper presents our system designed for Subtask 1 of SemEval-2025 Task 10, which focuses on multilingual entity framing in news articles. Given the complexity of the task—multi-label, multi-class classification across five languages—we propose an approach based on large language models (LLMs). Our method combines multilingual text translation, data augmentation, multi-model fine-tuning, and ensemble classification. First, we translate all texts into English to unify the datasets and apply synonym-based augmentation to address class imbalances. We then fine-tune multiple LLMs on the augmented data. Finally, a state-of-the-art LLM aggregates the individual model predictions for ensemble classification, yielding robust and accurate results. Our system achieved top positions in three languages (English, Portuguese, and Russian) and second place in Bulgarian.

DUTIR at SemEval-2025 Task 10: A Large Language Model-based Approach for Entity Framing in Online News

We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs). Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task.

iai_MSU at SemEval-2025 Task-3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes in English

This paper presents a zero-shot system for fact-checked claim retrieval. 
We employed several state-of-the-art large language models to obtain text embeddings. The models were then combined to obtain the best possible result.
Our approach achieved 7th place in monolingual and 9th in cross-lingual subtasks.
We used only English translations as an input to the text embedding models since multilingual models did not achieve satisfactory results. We identified the most relevant claims for each post by leveraging the embeddings and measuring cosine similarity. Overall, the best results were obtained by the NVIDIA NV-Embed-v2 model. For some languages, we benefited from model combinations (NV-Embed & GPT or Mistral).

Downloads

Next from ACL 2025

IALab UC at BEA 2025 Shared Task: LLM-Powered Expert Pedagogical Feature Extraction

Similar lecture

Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2025

IALab UC at BEA 2025 Shared Task: LLM-Powered Expert Pedagogical Feature Extraction

Similar lecture

Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors

Downloads