Austria

The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings.

Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

ACL 2025

Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

workshop paper

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Toxic language in digital environments presents significant challenges with implications for user well-being and the structure of online discourse. The spread of harmful text has been linked to consequences such as effects on mental health, the reinforcement of social biases, and the exclusion of specific groups. These concerns have led to the development of systems to detect and reduce toxicity across different contexts. Despite progress in toxic content detection, most publicly available datasets rely on binary annotations (e.g., “toxic” vs. “non-toxic”). This simplification limits the representation of linguistic and social attributes, such as the targeted identity group, the tone or intensity of the message, and the subtype of harmful expression. Consequently, models trained on such datasets often struggle to generalize to more implicit manifestations of toxicity, including covert hate speech and indirect discriminatory language. In this work, we investigate classifier-guided and prompt-based strategies for generating synthetic toxicity datasets. We compare their effects on linguistic variation, demographic coverage, and generalization across benchmarks, focusing on existing frameworks such as ToxiGen and ToxEvol.

Prompt-Based Evolution for Diverse and Generalizable Toxic Language Datasets

Evaluation of intermediate language model checkpoints during training is critical for effective model development and selection. However, reliable evaluation using the popular multiple-choice question (MCQ) format is challenging, as small and non-instruction-tuned models often lack the symbolic reasoning required for the task, despite the fact that MCQ evaluation is often used and needed to distinguish between the performance of different training runs. In particular, when prompted with a question and a set of labeled answer choices (e.g., "A. …, B. …, C. …''), many models struggle to emit the correct label (e.g., "C''), even when they can select the correct string answer choice. We propose an alternative evaluation method: fine-tuning the model on an auxiliary MCQ dataset prior to outputting labels. We validate this approach empirically and analyze its generalization across datasets and varying amounts of training data, finding that the model is indeed able to generalize the format from one dataset to another, potentially providing a better indication of the strength of intermediate model checkpoints.

Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints

Aligning large language models (LLMs) with human preferences remains a key challenge in AI. Preference-based optimization methods, such as Reinforcement Learning with Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on human-annotated datasets to improve alignment. In this work, we identify a crucial property of the existing learning method: the distinguishing signal obtained in preferred responses is often concentrated in the early tokens. We refer to this as $\textit{shallow preference signals}$.

To explore this property, we systematically truncate preference datasets at various points and train both reward models and DPO models on the truncated data. $\textbf{Surprisingly}$, models trained on truncated datasets, retaining only the first half or fewer tokens, achieve comparable or even superior performance to those trained on full datasets. For example, a reward model trained on the Skywork-Reward-Preference-80K-v0.2 dataset outperforms the full dataset when trained on a 40% truncated dataset. This pattern is consistent across multiple datasets, suggesting the widespread presence of $\textit{shallow preference signals}$.

We further investigate the distribution of the reward signal through decoding strategies. We consider two simple decoding strategies motivated by the shallow reward signal observation, namely Length Control Decoding and KL Threshold Control Decoding, which leverage shallow preference signals to optimize the trade-off between alignment and computational efficiency. The performance is even better, which again validates our hypothesis.

The phenomenon of $\textit{shallow preference signals}$ highlights potential issues in LLM alignment: existing alignment methods often focus on aligning only the initial tokens of responses, rather than considering the full response. This could lead to discrepancies with real-world human preferences, resulting in suboptimal alignment performance.

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

Human evaluation is indispensable in natural language processing (NLP), as automatic metrics are known to not always align well with human judgments.
However, the reproducibility of human evaluations can be problematic since results are susceptible to many factors, the details of which are often missing from the respective works.
As part of the ReproHum project, this work aims to reproduce the human evaluation of a single criterion in the paper "MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes" (Gu et al, 2022).
The results of our reproduction differ noticeably from those of the original study. To explain this discrepancy, we discuss differences in the experimental setup, as well as more general characteristics of the selected domain and the generated summaries.

ReproHum #0729-04: Human Evaluation Reproduction Report for "MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes"

The reproducibility of results is the foundation on which scientific credibility is built. In Natural Language Processing (NLP) research, human evaluation is often seen as the gold standard of evaluation. This paper presents the replication of a human evaluation of a Natural Language Generation (NLG) system that generates pairs of questions and answers based on children's stories that was originally conducted by Yao et al. (2022). Specifically, it replicates the evaluation of readability, one of the most commonly evaluated criteria for NLG systems. The results of the reproduction are aligned with the original findings and all major claims of the original paper are confirmed by the reproduction.

ReproHum #0031-01: Reproducing the Human Evaluation of Readability from "It is AI’s Turn to Ask Humans a Question"

Reproducibility remains a fundamental challenge for human evaluation in Natural Language Processing (NLP), particularly due to the inherent subjectivity and variability of human judgments. This paper presents a reproduction study of the human evaluation protocol introduced by Hosking and Lapata (2021), which assesses semantic preservation in paraphrase generation models. By faithfully replicating the original experiment with careful adaptation and applying the Quantified Reproducibility Assessment framework (Belz and Thomson, 2024a; Belz, 2022), we demonstrate strong agreement with the original findings, confirming the semantic preservation ranking among four paraphrase models. Our analyses reveal moderate inter-annotator agreement and low variability in key results, underscoring a good degree of reproducibility despite practical deviations in participant recruitment and platform.These findings highlight both the feasibility and challenges of reproducing human evaluation studies in NLP. We discuss the implications for improving methodological rigor, transparent reporting, and standardized protocols to bolster reproducibility in future human evaluation efforts. The data and analysis scripts are publicly available to support ongoing community efforts toward reproducible NLP evaluation.

ReproHum: #0744-02: Investigating the Reproducibility of Semantic Preservation Human Evaluations

We investigate cross-lingual transfer in large language models (LLMs) trained on two high-resource languages, English and Chinese. Four monolingual Chinese and four bilingual English-Chinese models are evaluated on two Chinese linguistic benchmarks. Monolingual models consistently outperform the bilingual ones on 12 out of 55 tasks, a result indicating negative transfer from English to Chinese. Additionally, we carry out a feature attribution analysis in a monolingual and a bilingual model, showing that the differences in their performance may be explained by more predictable attribution patterns in the monolingual model. Our findings have implications for the ongoing effort of training bilingual LLMs.

Curse of bilinguality: Evaluating monolingual and bilingual language models on Chinese linguistic benchmarks

In-context learning is an effective method for large pre-trained language models (LLMs) to complete various downstream tasks. LLMs are typically trained in high-resource languages, and tasks in lower-resourced languages tend to underperform the higher-resource language counterparts for in-context learning. Despite the large body of work on prompting settings, it is still unclear how LLMs should be adapted cross-lingually specifically for in-context learning in the low-resource target languages. We perform a comprehensive study spanning five diverse target languages, three base LLMs, and seven downstream tasks spanning over 4,100 GPU training hours (9,900+ TFLOPs) across various adaptation techniques such as few-shot prompting, translate-test, fine-tuning, embedding re-initialization, and instruction fine-tuning. Our experiments display that few-shot prompting and translate-test settings tend to heavily outperform gradient-based adaptation methods, but there is no trend between which of the two works best. We also provide an empirical analysis of model outputs and demonstrate that the gradient-based adaptation methods faltered due to catastrophic forgetting. To the extent of our knowledge, this is the largest study done on in-context learning for low-resource languages with respect to train compute (time and FLOPs) and number of adaptation techniques considered. We make all our trained models available for public use.

Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

The deployment of Large Language Models (LLMs) in customer support is constrained by hallucination—generating false information—and the high cost of proprietary models. To address these challenges, we propose a retrieval-augmented question-answering (QA) pipeline and explore how to balance human input and automation. Using a dataset of questions about a Samsung Smart TV user manual, we demonstrate that synthetic data generated by LLMs outperforms crowdsourced data in reducing hallucination in finetuned models. We also compare self-training (fine-tuning models on their own outputs) and knowledge distillation (fine-tuning on stronger models' outputs, e.g., GPT-4o), and find that self-training achieves comparable hallucination reduction. We conjecture that this surprising finding can be attributed to increased exposure bias issues in the knowledge distillation case and support this conjecture with post hoc analysis. We also improve robustness to unanswerable questions and retrieval failures with contextualized “I don’t know” responses. These findings show that scalable, cost-efficient QA systems can be built using synthetic data and self-training with open-source models, reducing reliance on proprietary tools or costly human annotations.

Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents

This study utilizes the game Codenames as a benchmarking tool to evaluate large language models (LLMs) with respect to specific linguistic and cognitive skills. LLMs play each side of the game, where one side generates a clue word covering several target words and the other guesses those target words. We designed various experiments by controlling the choice of words (abstract vs. concrete words, ambiguous vs. monosemic) or the opponent (programmed to be faster or slower in revealing words). Recent commercial and open-weight models were compared side-by-side to find out factors affecting their performance. The evaluation reveals details about their strategies, challenging cases, and limitations of LLMs.

Downloads

Next from ACL 2025

Prompt-Based Evolution for Diverse and Generalizable Toxic Language Datasets

Similar lecture

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2025

Prompt-Based Evolution for Diverse and Generalizable Toxic Language Datasets

Similar lecture

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

Downloads