China

The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea.

EMNLP 2025

From KMMLU-Redux to Pro: A Professional Korean Benchmark Suite for LLM Evaluation

korean

benchmark

dataset

### Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Welcome to the 2025 meeting of the Nations of the Americas Chapter of the Association for Computational Linguistics! I am proud to help organize the first NAACL conference to carry the new name of our organization, one that emphasizes inclusion for all of the Americas. I am also pleased to welcome you to Albuquerque, New Mexico, a state whose unique blend of cultural influences will make for an excellent backdrop for NAACL 2025, especially with this year’s special theme on NLP in a Multicultural World. 
**[Continue reading...](https://drive.google.com/file/d/1jX-qGhqVSZZCIrAnJaz798pflu5Irrdn/view?usp=sharing)**

*- Colin Cherry, Google, NAACL 2025 General Chair* 

[![](https://assets.underline.io/markdown_image/1/image/b087f8a4dc5816d6e1a6514e59c59ac3.png)](https://drive.google.com/file/d/1T96GzPqObXrMln2BMByCSXTSizjTg69P/view?usp=sharing)

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

To access NAACL 2025 event page you are required to register. Please follow [**this link**](https://2025.naacl.org/registration/registration/) to register. Access will depend on your registration type.

Please register!

NAACL 2025

Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

poster

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community." 

*Message from the General Chair, Dirk Hovy*

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

The Impression section of a radiology report summarizes critical findings of a radiology report and thus plays a crucial role in communication between radiologists and physicians. Research on radiology report summarization mostly focuses on generating the Impression section by summarizing information from the Findings section, which typically details the radiologist's observations in the radiology images. Recent work start to explore how to incorporate radiology images as input to multimodal summarization models, with the assumption that it can improve generated summary quality, as it contains richer information. However, the real effectiveness of radiology images remains unclear. To answer this, we conduct a thorough analysis to understand whether current multimodal models can utilize radiology images in summarizing Findings section. Our analysis reveals that current multimodal models often fail to effectively utilize radiology images. For example, masking the image input leads to minimal or no performance drop. Expert annotation study shows that radiology images are unnecessary when they write the Impression section.

The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization

Large Language Models (LLMs) excel at reasoning but often struggle with multi-step tasks due to limited contextual understanding. While Retrieval-Augmented Generation (RAG) provides external context, it relies on static datasets and struggles to generalize across diverse queries. In this work, we propose a Generative Retrieval-Aligned Demonstrator (GRAD), a dynamic demonstration-based approach where an LLM model is trained to generate input-specific concise demonstrations. By tailoring demonstrations to each input, our method offers better contextual support than traditional RAG approaches. GRAD is grounded in a generative strategy allowing adaptive prompting without needing external retrieval. We demonstrate the superiority of GRAD under the budget constraints, where we limit both the number of tokens per demonstration and the output response budget. Trained solely on math dataset, GRAD consistently outperforms strong baselines ranging from math reasoning to advanced STEM questions on Qwen2.5-14B. This showcases the robust generalization of GRAD to out-of-distribution (OOD) STEM tasks, such as physics, chemistry and computer science. Since demonstrations can be generated by a different, smaller model, GRAD might reduce training cost while still maintaining similar accuracy. This work introduces a scalable demonstration generator model that is the first step towards a new few-shot learning paradigm in constrained, limited-resource settings.

GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoning

In scientific research, “limitations” refer to the shortcomings, constraints, or weaknesses within a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often a) underreport them in the papers’ text and b) use hedging strategies to satisfy editorial requirements at the cost of readers’ clarity and confidence. This underreporting behavior, along with an explosion in the number of publications, has created a pressing need to automatically extract/generate such limitations from scholarly papers. In that direction, we report a complete architecture for computational analysis of research limitations. Specifically, we a) create a dataset of limitations in ACL, NeurIPS, and PeerJ papers by extracting them from papers’ text and integrating them with external reviews; b) propose methods to automatically generate them using a novel Retrieval Augmented Generation (RAG) technique; c) create a fine-grained evaluation framework for generated limitations and provide a meta-evaluation for the proposed evaluation techniques.

BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text

Procedural text adaptation—such as modifying recipes or revising instructional guides—has traditionally relied on specialized models extensively fine‑tuned for specific domains. To address the scalability limitations of such approaches, recent research has increasingly turned to general‑purpose large language models (LLMs). However, existing prompting strategies for LLMs often yield superficial or erroneous adaptations due to alignment‑induced biases and the inherent complexity of procedural editing. To overcome these challenges, we propose the Over‑generation‑and‑Compaction (OC) prompting strategy, which first elicits an exhaustive set of procedural details to leverage the model’s latent knowledge, and subsequently compacts them into concise, coherent adaptations. We further introduce Recipe Consistency & Feasibility (RCF), a novel metric for systematically assessing procedural validity and practicality in cooking recipe adaptations. Experiments on public datasets demonstrate that OC significantly improves adaptation consistency and feasibility compared to baseline prompting methods, without the need for additional fine-tuning or curated training resources.

Over-Generation and Compaction: A Prompting Strategy for Procedural Text Adaptation with Large Language Models

The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation for building high-quality NLP resources in low-resource language/domain pairs.

TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

Large Language Models (LLMs) excel at generating synthetic data, but ensuring its quality and diversity remains challenging. We propose Genetic Prompt, a novel framework that innovatively combines genetic algorithms (GAs) with LLMs for synthetic data generation. Unlike previous GA approaches, Genetic Prompt implicitly represents semantic text attributes as gene sequences. Additionally, We implement an active learning scheme to optimize the GA process, eliminating traditional fitness evaluation and expanding offspring search space. Experiments on four relation extraction datasets show that Genetic Prompt significantly outperforms State-of-the-art baselines. Further more, analysis reveals that Genetic Prompt generates more diverse data with distributions closer to the gold dataset.

Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generation

Evaluating text generation capabilities of large language models (LLMs) is challenging, particularly for low-resource languages where methods for direct assessment are scarce. We propose MUG-Eval, a novel framework that evaluates LLMs' multilingual generation capabilities by transforming existing benchmarks into conversational tasks and measuring the LLMs' accuracies on those tasks. We specifically designed these conversational tasks to require effective communication in the target language. Then, we simply use task success rate as a proxy of successful conversation generation. Our approach offers two key advantages: it is independent of language-specific NLP tools or annotated datasets, which are limited for most languages, and it does not rely on LLMs-as-judges, whose evaluation quality degrades outside a few high-resource languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and low-resource categories, and we find that MUG-Eval correlates strongly with established benchmarks (r > 0.75) while enabling standardized comparisons across languages and models. Our framework provides a robust and resource-efficient solution for evaluating multilingual generation that can be extended to thousands of languages.

MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language

Long chain-of-thought (CoT) prompting helps Large Language Models (LLMs) solve difficult problems, but very long traces often slow or even degrade performance on fast, intuitive “System-1” tasks. We introduce Connector-Aware Compact CoT (CAC-CoT) --- a method that deliberately restricts reasoning to a small, fixed set of connector phrases, steering the model toward concise and well --- structured explanations. Despite its simplicity, our synthetic method with Gemini-2.0-Flash yields a high-quality training quality. CAC-CoT achieves approx 85\% on GSM8K and approx 40\% on GPQA (System-2) while retaining approx 90\% on S1-Bench (System-1). Its reasoning traces average approx 300 tokens(ART), about one-third the length of baseline traces, delivering higher efficiency without loss of accuracy.

CAC-CoT: Connector-Aware Compact Chain-of-Thought for Efficient Reasoning Data Synthesis Across Dual-System Cognitive Tasks

While LLMs have been extensively studied on general text generation tasks, there is less research on text rewriting, a task related to general text generation, and particularly on the behavior of models on this task. In this paper we analyze what changes LLMs make in a text rewriting setting. We focus specifically on argumentative texts and their improvement, a task named Argument Improvement (ArgImp). We present CLEAR: an evaluation pipeline consisting of 57 metrics mapped to four linguistic levels: lexical, syntactic, semantic and pragmatic. This pipeline is used to examine the qualities of LLM-rewritten arguments on a broad set of argumentation corpora and compare the behavior of different LLMs on this task and analyze the behavior of different LLMs on this task in terms of linguistic levels. By taking all four linguistic levels into consideration, we find that the models perform ArgImp by shortening the texts while simultaneously increasing average word length and merging sentences. Overall we note an increase in the persuasion and coherence dimensions.

CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models

The emergence of large language models (LLMs) has brought a new paradigm to automated essay scoring (AES), a long-standing and practical application of natural language processing in education. However, achieving human-level multi-perspective understanding and judgment remains a challenge. In this work, we propose Roundtable Essay Scoring (RES), a multi-agent evaluation framework designed to perform precise and human-aligned scoring under a zero-shot setting. RES constructs evaluator agents based on LLMs, each tailored to a specific prompt and topic context. Each agent independently generates a trait-based rubric and conducts a multi-perspective evaluation. Then, by simulating a roundtable-style discussion, RES consolidates individual evaluations through a dialectical reasoning process to produce a final holistic score that more closely aligns with human evaluation. By enabling collaboration and consensus among agents with diverse evaluation perspectives, RES outperforms prior zero-shot AES approaches. Experiments on the ASAP dataset using ChatGPT and Claude show that RES achieves up to a 34.86% improvement in average QWK over straightforward prompting (Vanilla) methods.

Downloads

Next from EMNLP 2025

The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization

Similar lecture

KMMLU: Measuring Massive Multitask Language Understanding in Korean