United States

Evaluating long-form answers from retrieval-augmented generation (RAG) systems remains challenging: human evaluation is expensive, while automatic metrics must reliably capture answer completeness. The AutoNuggetizer framework addresses this by decomposing evaluation into atomic facts (nuggets) and using LLMs for both nugget creation and assignment. The original study validated this approach on open-ended TREC RAG queries; however, it remains unclear whether the same cost-quality tradeoffs hold for structurally different tasks. We reproduce AutoNuggetizer on seven RAG systems over the QAMPARI list-QA benchmark, where answers consist of discrete entities and omissions are more directly measurable. Our results directionally reproduce the original findings: fully automatic evaluation preserves run-level rankings, assignment-only automation yields stronger agreement than end-to-end automation, and LLM-based assignment is highly concordant with human labels while being modestly stricter. These findings support the use of AutoNuggetizer for comparative evaluation beyond open-ended RAG, while also identifying systematic biases in automatic nugget creation and assignment.

ACL 2026

Do Nugget-Based Evaluation Patterns Generalize to List-QA?

nugget-based evaluation

recall evaluation

long-form evaluation

It is my great pleasure and honor to welcome you to the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), held in San Diego, California, USA, July 2–7, 2026. ACL 2026 continues our fields tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.

It is exciting to see the many developments in the field reflected at our conference, such as the advancement of large language models (born out of our community) into many application directions. This is not without challenges, partly due to unsolved problems, and partly due to misuse of the technology. All this is reflected well at ACL 2026. The conference also reflects that the field is not only about advancing technology, but also the science of fundamentally understanding language and how it is use by many different speaker communities. <br>
[Full Message from General Chair Philipp Koehn](https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing) <br><br><br>
<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://drive.google.com/file/d/1QiJFOmMjvDgsDo3i1XzlQnImxe3RU_Cy/view?usp=sharing')">Conference Handbook</button></html>
<br><br><br><br>

To access this event page, you need to log in with the **email address you registered with**. <br>Access credentials will be sent to your email from Underline -  subject line "Welcome to ACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.aclweb.org/registration/) first.

**Online Registration Form**: https://acl.swoogo.com/acl2026

Registration Required

The 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) will take place in San Diego, California, from July 2 - 7, 2026.

workshop paper

Recent work in NLP has probed large language models for their understanding of cultural norms across countries. However, this work typically considers distributional patterns, ignoring group consensus or possible multicultural environments within a country. In this work, we leverage cultural consensus theory (CCT) from cultural anthropology to model such multidimensional nuance. Applying CCT to the World Values Survey (WVS) across 10 countries and 12 domains, we demonstrate that models frequently misrepresent cultural structures by either failing to form cohesive consensus or severely over-regularizing consensus. Through explicit representation of intra-group variance, CCT provides actionable diagnostics to evaluate when models reflect true human diversity versus algorithmic homogenization.

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Bangla memes are widely used on social me dia to express humor and social commentary, yet computational analysis of gender bias in Bangla memes remains largely unexplored. In this work, we present a multimodal framework for detecting gender bias in Bangla memes by jointly analyzing textual and visual con tent. We construct a dataset of 6,846 Bangla and Banglish code-mixed memes annotated into three categories: male-biased, female biased, and neutral. For textual representa tion, we use BanglishBERT, while visual fea tures are extracted using ConvNeXt, and the two modalities are fused for final classifica tion. Our best-performing model, ConvNeXt + BanglishBERT, achieves accuracy of 0.67 and an F1-score of 0.63, outperforming sev eral multimodal baselines. The results demon strate the effectiveness of multimodal learn ing for understanding culturally nuanced and code-mixed meme content in low-resource lan guages.

Behind the Laughter: Uncovering Gender Bias in Code-Mixed Bangla Memes

Bias and fairness risks in Large Language Mod els (LLMs) vary substantially across deploy ment contexts, yet existing approaches lack systematic guidance for selecting appropriate evaluation metrics. We present a decision framework that maps LLM use cases, charac terized by a model and population of prompts, to relevant bias and fairness metrics based on task type, whether prompts contain protected attribute mentions, and stakeholder priorities. Our framework addresses toxicity, stereotyp ing, counterfactual unfairness, and allocational harms, and introduces novel metrics based on stereotype classifiers and counterfactual adapta tions of text similarity measures. We release an open-source Python library, langfair, for practical adoption. Extensive experiments on use cases across five LLMs and five prompt populations demonstrate that fairness risks can not be reliably assessed from benchmark per formance alone: results on one prompt dataset likely overstate or understate risks for another, underscoring that fairness evaluation must be grounded in the specific deployment context.

Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs

Multilingual Natural Language Understanding (NLU) systems often struggle to adapt when new languages or new semantic labels are in troduced with only a few annotated examples. This challenge is particularly pronounced for low-resource languages, where limited super vision and evolving label spaces make conven tional joint-label classification approaches un stable. Most existing multilingual NLU models treat each language–semantic pair as an inde pendent class, entangling linguistic and seman tic representations and hindering few-shot adap tation. We propose Dual-Axis Compositional Few-Shot Learning, a framework that explicitly factorizes the representation space into linguis tic and semantic embedding axes, enabling in dependent modeling of language variation and domain–intent semantics. Joint representations are constructed compositionally through mul tiplicative interaction of axis-specific embed dings, allowing controlled adaptation when ei ther the language set or the semantic label space evolves. The framework integrates factorized prototype learning, axis-structured contrastive alignment, and disentanglement regularization using HSIC-based statistical independence and Jacobian-based cross-axis decorrelation. Ex periments on six low-resource Indic languages spanning Indo-Aryan and Dravidian families (Hindi, Bengali, Sanskrit, Assamese, Tamil, and Telugu) demonstrate strong performance under two structured generalization regimes. The model achieves 81.12% accuracy when adapting to few-shot languages with known semantics and 63.5% accuracy when learning new semantic classes from few-shot examples, along with an accuracy of 89.56% on known languages and seen semantics. These results show that axis-factorized representations en able stable compositional generalization, offer ing a promising direction for scalable multilin gual NLU in linguistically diverse low-resource settings.

Dual-Axis Compositional Contrastive Few-Shot Learning using Prototypes Across Linguistic and Semantic Dimensions for Indic Low-Resource Multilingual NLU

Automatic text simplification (ATS) aims to en hance language accessibility for various target groups, particularly persons with intellectual disabilities. Recent advancements in large lan guage models (LLMs) have substantially im proved the quality of machine-generated text simplifications, however, existing LLM-based ATS systems do not incorporate preference feedback during post-training, resulting in a lack of personalization tailored to the specific needs of target group persons. In this work, we propose an ATS personalization framework using direct preference optimization (DPO). Specifically, we post-trained LLM-based ATS models using human feedback collected from persons with intellectual disabilities, reflecting their preferences of paired text simplifications generated by mainstream LLMs. Our pipeline for developing personalized LLM-based ATS systems encompasses data collection, model se lection, supervised fine-tuning (SFT) and DPO post-training, and result evaluation. Our find ings underscore the necessity of active partici pation of target group persons in designing per sonalized inclusive AI solutions aligned with human preferences.

Evaluating Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities

This paper presents a methodology that uses LLMs to align multilingual offensive lexi cons at the sense level. Lexicons of differ ent structures and origins in Arabic, Bulgar ian, Modern Greek, French, and Italian have been aligned directly without pivoting through English. The Modern Greek lexicon is LLM generated, and the other four lexicons are WordNet-compatible. For inter-language align ment of senses, an LLM-as-a-judge rubric was used over lemma–definition–example triples. The LLM makes 2.87M pairwise comparisons and yields 31 strict global-sense categories. The paper discusses the challenges involved in sense alignment tasks. The resource is avail able to support downstream applications such as Machine Translation and cross-lingual hate speech detection.

From Form to Meaning: Interlingua Sense-Alignment of Offensive Language with LLMs

Large Language Models (LLMs) perform strongly on many language tasks, but their capa bility in structurally constrained, accessibility critical modalities such as Braille remains un clear. We evaluate state-of-the-art LLMs on bidirectional Korean–Braille translation using a human-annotated dataset. Despite expecta tions that multilingual, instruction-tuned mod els can generalize to Braille via text repre sentations, we find consistently poor, unstable outputs and substantial disagreement with hu man judgments. These results point to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns. In con trast, supervised fine-tuning of a small model (T5-small) on the same data yields large and stable gains over zero-shot and prompted LLM baselines across standard metrics (SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr). Our findings reveal a systematic lim itation of current LLMs and demonstrate the effectiveness of modest task-specific supervi sion.

I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs

We investigate the role of large language mod els (LLMs) in promoting gender-inclusive lan guage by evaluating their ability to rewrite bi ased text and generate counterfactual narratives across multiple languages. We introduce a shared task with two subtasks: gender-inclusive rewriting and counterfactual generation. The task covers five languages English, German, Spanish, Tamil, and Kannada reflecting diverse grammatical gender systems and sociocultural contexts. We release curated word-level and sentence-level datasets to support controlled inclusive generation. A total of 50 teams reg istered for the shared task, and around 8 teams submitted results. Submissions are evaluated using a hybrid framework combining rubric based automatic scoring with expert human judgment. Finally, we provide an overview of participating systems and discuss key findings and challenges observed across languages.

Insights from Multilingual Gender Inclusive Language Generation Shared Task

Gender bias in multilingual language gener ation systems poses serious ethical and so cial issues, especially in languages with com plex morphology. In this study, we propose a lightweight multilingual approach that employs instruction-guided fine-tuning of the mT5 small transformer model for gender-inclusive language generation. The framework accom modates five languages: English, German, Spanish, Tamil, and Kannada. The approach uses the task-prefix rewriting method to trans form gender-specific sentences to their gender neutral versions. The training data from differ ent languages is combined into a single multi lingual dataset for sequence-to-sequence fine tuning. Beam search decoding with repetition constraints is used during inference to improve the quality of the output. The system’s perfor mance is measured using GIFI, semantic simi larity, and an overall combined score across all languages. Experimental results show that the system can eliminate gender-biased language while retaining semantic meaning in part across languages.

CAI@LTEDI 2026: Multilingual Gender Inclusive Language Generation using Instruction-Guided mT5 Transformer Model

The rapid growth of social media has also led to a rise in abusive and harmful content, which negatively affects the online environment for users. The frequent use of offensive language and hate speech contributes to making these platforms increasingly hostile. In particular, homophobic and transphobic remarks target members of the LGBT+ community. Detecting such comments is therefore essential so that they can be flagged promptly and appropriate warnings can be given to users involved in such behaviour. The problem becomes more serious when such content appears in other forms of communication used by younger generations, such as memes. This work tries to address this issue. We propose a method to detect such content using the meme dataset from the LT-EDI 2026 challenge and secured 8th rank for English and 6th rank for Chinese language dataset in the shared task. Our approach uses a multimodal technique that processes both image and text information. The dataset has limited data, which creates a challenge. To handle this, we pre–fine-tune the models on a similar dataset called PrideMM. The proposed multimodal approach achieved Macro F1-scores of 0.24 and 0.57 for English and Chinese memes respectively

Downloads

Next from ACL 2026

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Similar lecture

Exploring Capability Thresholds in Ultra-Lightweight LLM Judges for Nugget-Based Report Evaluation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2026

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Similar lecture

Exploring Capability Thresholds in Ultra-Lightweight LLM Judges for Nugget-Based Report Evaluation

Downloads