Austria

We present our submission to SemEval 2025 Task 8: Question Answering on Tabular Data, which challenges participants to develop systems capable of answering natural language questions on real-world tabular datasets. Our approach aims at generating Pandas code that can be run on such datasets to produce the desired answer. 
The approach consists in fine-tuning a Small Language Model (SLM) through Preference Optimization on both positive and negative examples generated by a teacher model. A base SLM is first elicited to produce the code to compute the answer to a question through a Chain of Thought (CoT) prompt. We performed extensive testing on the DataBench development set, exploring a variety of prompts, eventually settling on a detailed instruction prompt, followed by two-shot examples. 
Due to hardware constraints, the base model was an SLM with ≤ 8 billion parameters. We then fine-tuned the model through Odds Ratio Preference Optimization (ORPO) using as training data the code produced by a teacher model on the DataBench training set. The teacher model was GPT-4o, whose code was labeled preferred, while the code generated by the base model was rejected. This increased the accuracy on the development set from 71% to 85%.
Our method demonstrated robust performance in answering complex questions across diverse datasets, highlighting the effectiveness of combining small LLMs with supervised fine-tuning and automated code execution for tabular question answering.

ACL 2025

Dataground at SemEval-2025 Task 8: Small LLMs and Preference Optimization for Tabular QA

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

workshop paper

In today's digital era, individuals convey their feelings, viewpoints, and perspectives across various platforms in nuanced and intricate ways. At times, these expressions can be challenging to articulate and interpret. Emotion recognition aims to identify the most relevant emotions in a text that accurately represent the author's psychological state. Despite its substantial impact on natural language processing (NLP), this task has primarily been researched only in high-resource languages. To bridge this gap, SemEval-2025 Task 11 introduces a multilingual emotion recognition challenge encompassing 32 languages, promoting broader linguistic inclusivity in emotion recognition. This paper presents our participation in this task, where we introduce a language-specific fine-tuned transformer-based system for emotion recognition and emotion intensity prediction. To enhance generalization, we incorporate a multi-sample dropout strategy. Our approach is evaluated across 11 languages, and experimental results demonstrate its competitive performance, achieving top-tier results in certain languages.

CSECU-Learners at SemEval-2025 Task 11: Multilingual Emotion Recognition and Intensity Prediction with Language-tuned Transformers and Multi-sample Dropout

Food safety incidents cause serious threats to public health, requiring efficient detection systems. This study contributes to SemEval 2025 Task 9: Food Hazard Detection by leveraging insights from existing literature and using multiple BERT-based models for multi-label classification of food hazards and product categories. Using a dataset of food recall notifications, we applied preprocessing techniques
to prepare data and address challenges like class imbalance. Experimental results show strong hazard classification performance on ensembled models such as DistilBERT, SciBERT, and DeBERTa but highlight product classification variability. Building on Tyagi et al. (Tyagi et al., 2023) and Madry et al.’s (Rebuffi et al., 2021) work, we explored strategies like ensemble modeling and data augmentation to improve accuracy and explainability, paving the way for scalable food safety solutions.

Habib University at SemEval-2025 Task 9: Using Ensemble Models for Food Hazard Detection

As the Large Language Model (LLM) gains widespread adoption, increasing attention has been given to the challenge of making LLM forget non-compliant data memorized during its pre-training. Machine Unlearning focuses on efficiently erasing sensitive information from LLM under limited computational resources. To advance research in this area, SemEval 2025 Task 4: "Unlearning Sensitive Content from Large Language Models" introduces three unlearning datasets and establishes a benchmark by evaluating both forgetting effectiveness and the preservation of standard capabilities. In this work, we propose a more controllable forgetting loss, Effective Unlearning Loss, and explore its integration with various techniques to achieve more efficient and controlled unlearning. Our system ultimately ranked 5th on the competition leaderboard.

iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss

This paper describes our approach to the SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval on cross-lingual data. In this study, we developed a system to tackle the verified claim retrieval task, where the objective is to identify relevant, previously fact-checked claims from multilingual data based on a given input claim. We leveraged LLaMA, utilizing its ability to evaluate the relevance of retrieved claims within a retrieval-based fact-checking framework. This approach aimed to explore the impact of large language models (LLMs) on retrieval tasks and assess their effectiveness in enhancing fact-checking accuracy. Additionally, we integrated various embeddings, including e5-large, Jina embeddings, and the MPNet multilingual sentence transformer, to filter and rank a set of 500 candidate claims. These refined claims were then used as input for LLaMA, ensuring that only the most contextually relevant ones were assessed. Our team in the cross-lingual track scored s@10.

CAISA at SemEval-2025 Task 7: Multilingual and Cross-lingual Fact-Checked Claim Retrieval

The Unlearning Sensitive Content from Large Language Models task aims to remove targeted datapoints from trained models while minimally affecting their general knowledge. In our work, we leverage parameter-efficient, gradientbased unlearning using low-rank (LoRA) adaptation and layer-focused fine-tuning. To further enhance unlearning effectiveness, we employ data chunking, splitting forget data into disjoint
partitions and merging them with cyclically sampled retain samples at a pre-defined
ratio. Our task-agnostic method achieves an outstanding forget-retain balance, ranking first on leaderboards and significantly outperforming baselines and competing systems.

AILS-NTUA at SemEval-2025 Task 4: Parameter-Efficient Unlearning for Large Language Models using Data Chunking

This study introduces a methodology centred
on Llama 3 fine-tuning for the classification of
entities mentioned within news articles, based
on a predefined role taxonomy. The research
is conducted as part of SemEval-2025 Task 10,
which focuses on the automatic identification
of narratives, their classification, and the deter-
mination of the roles of the relevant entities in-
volved. The developed system was specifically
used within Subtask 1 on Entity Framing. The
approach used is based on parameter-efficient
fine-tuning, in order to minimize the computa-
tional costs while maintaining reasonably good
model performance across all datasets and lan-
guages involved. The model achieved promis-
ing results on both the development and test
sets. Specifically, during the final evaluation
phase, it attained an average accuracy of 0.84
on the main role and an average Exact Match
Ratio of 0.41 in the prediction of fine-grained
roles across all the five languages involved, i.e.
Bulgarian, English, Hindi, Portuguese and Rus-
sian. The best performance was observed for
English (3rd place out of 32 participants), on
a par with Hindi and Russian. The paper pro-
vides an overview of the system adopted for the
task and discusses the results obtained.

DEMON at SemEval-2025 Task 10: Fine-tuning LLaMA-3 for Multilingual Entity Framing

Idioms and figurative language are nuanced linguistic phenomena that transport semanticity and culture via non-compositional multi-word expressions. This type of figurative language remains difficult for small and large language models to handle. For language models to become more valuable as translators and even companions, natural language must be understood and generated. A portion of this is idiomatic and figurative language. Various attempts have been made to identify idiomaticity in text. The approach presented in this paper represents an intuitive attempt to accurately address Task 1: AdMIRe Subtask A to correctly order a series of images and captions by concatenating the image captions as a sequence. The methods employ the reliability of a pre-trained vision and language model for the image-type task and a large language model with instruction fine-tuning for a causal language model approach to handle the caption portion of the task. The models chosen for development in the pipeline were based on their respective reliability in captioning and instruction fine-tuning.

JNLP at SemEval-2025 Task 1: Multimodal Idiomaticity Representation with Large Language Models

This paper describes the machine translation system submitted to the SemEval-2025 Entity-Aware Machine Translation Task by the SheffieldGATE Team. We proposed a multi-agent entity-aware machine translation system that operates through three distinct reasoning stages: entity recognition, knowledge enhancement, and translation decision-making. The innovation in our approach lies in leveraging large language models to generate contextually relevant queries during the knowledge enhancement stage, extracting candidate entities and their translations from external knowledge bases. In the final translation decision-making stage, we employ fine-tuned large language models to denoise the retrieved knowledge, selecting the most relevant entity information to ensure accurate translation of the original text. Experimental results demonstrate our system's effectiveness. In SemEval-2025 Task 2, our system ranks first among all systems in Spanish entity translation metrics and third in Italian. For systems that do not use gold standard entity IDs during test set inference, ours achieves the highest overall scores across four language pairs: German, French, Italian, and Spanish.

SheffieldGATE at SemEval-2025 Task 2: Multi-Stage Reasoning with Knowledge Fusion for Entity Translation

This paper presents our system for SemEval-
2025 Task 8: DataBench, Question-Answering
over Tabular Data. The primary objective of
this task is to perform question answering on
given tabular datasets from diverse domains un-
der two subtasks: DataBench QA (Subtask I)
and DataBench Lite QA (Subtask II). To tackle
both subtasks, we developed a zero-shot solu-
tion with a particular emphasis on leveraging
Large Language Model (LLM)-based code gen-
eration. Specifically, we propose a Python code
generation framework utilizing state-of-the-art
open-source LLMs to generate executable Pan-
das code via optimized prompting strategies.
Our experiments reveal that different LLMs ex-
hibit varying levels of effectiveness in Python
code generation. Additionally, results show
that Python code generation achieves supe-
rior performance in tabular question answering
compared to alternative approaches. Although
our ranking among zero-shot systems is un-
known at the time of this paper’s submission,
our system achieved eighth place in Subtask
I and sixth place in Subtask II among the 30
systems that outperformed the baseline in the
open-source models category.

ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generation

Food safety is a critical global concern, and automating the detection of food hazards from recall reports can improve public health monitoring and regulatory compliance. This paper presents our submission for SemEval-2025 Task 9: The Food Hazard Detection Challenge. We tackle the inherent class imbalance in this task by leveraging advanced data augmentation techniques, including LLM-based synthetic data generation, synonym replacement, and back-translation. We employ transformer-based models such as DistilBERT, fine-tuned with these augmented datasets, to enhance performance. Our system achieves significant improvements, obtaining a Macro-F1 score of 0.7882 in ST1 and 0.5099 in ST2. Additionally, we analyze the impact of augmentation strategies and compare multiple architectures, highlighting challenges in handling implicit hazards. Our findings underscore the effectiveness of LLM-based augmentation in addressing extreme class imbalance while demonstrating the strengths and limitations of transformer models in food safety applications.

Downloads

Next from ACL 2025

CSECU-Learners at SemEval-2025 Task 11: Multilingual Emotion Recognition and Intensity Prediction with Language-tuned Transformers and Multi-sample Dropout

Similar lecture

ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2025

CSECU-Learners at SemEval-2025 Task 11: Multilingual Emotion Recognition and Intensity Prediction with Language-tuned Transformers and Multi-sample Dropout

Similar lecture

ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generation

Downloads