China

Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5% of the training data needed by the Random baseline to reach the target performance.

EMNLP 2025

FIRE: Flexible Integration of Data Quality Ratings for Effective Pretraining

scaling

data selection

pre-training

Truth and Responsibility
“Truth and Responsibility” is a call to reimagine anthropology to meet the demands of the present moment. The imperative to bear witness, take action, and be held accountable to the truths we write and circulate invites us to reflect on our responsibility in reckoning with disciplinary histories, harms, and possibilities. To whom are we giving evidence and toward what ends? For whom are we writing? To whom are we accountable, and in what ways? We are thrilled to announce the theme of the 2021 AAA Annual Meeting to be held in Baltimore, MD: Truth and Responsibility.

Many consider anthropology to be a holistic social science dedicated to cross-cultural understanding as a way to reduce bigotry and more fully comprehend “what it means to be human.” Others push against this readily circulated origin story, arguing that “human” remains a highly contested, political category—evident as the presumed set of rights and protections associated with the “human” continue to be stripped away violently in the communities in which some of us live and work. Which violent histories and epistemologies must anthropologists take responsibility for in telling truth(s) about the discipline’s development? How do the dual histories of settler colonialism and slavery continue to influence anthropological thought and practices? In these political times, how can anthropologists throughout the globe work to secure a capacious, progressive vision of the human? How might movements for anti-racism, decoloniality, queer liberation, and healing such as #BlackLivesMatter, #ProtectMaunaKea, and #MeToo push a future anthropology out of the ashes of the anthropological past? Finally, what are the limits and possibilities of the anthropological imagination?

Across subfields, we find truths in patterns of human behavior, language, evolution, and cultural worlds. Industry-positioned anthropologists do this by extending the boundaries of the discipline not as an administrative solution to job shortages but as a source of intervention and knowledge production. Increasingly, anthropologists who make use of the scientific method question how status and identity help to decide which facts come to represent truth. How might quantitative researchers use our methods to identify patterned truths, and what responsibility do we hold in challenging conventional wisdom about these patterns? What responsibility do we have to both our colleagues and larger publics, to push toward increased transparency and ethical grounding in our data collection, analysis, and presentation?

Pedagogy is a key medium for the communication of anthropological truths. Which (and whose) truths are foregrounded in our curricula? What are the possibilities for a liberatory pedagogy in a precarious, “post-truth” era? Conferences, museums, organizing spaces, and classrooms are sites where pedagogy and various forms of communication can help us interrogate the performativity of truth. How might anthropologists conceive of truth performances? Might visual, aural, and choreographic performances during AAA sessions offer anthropologists alternative genres for truth-telling and truth-making that provoke discussions about responsibility?

In accordance with the urgency of the year’s theme, the questions posed in this call for papers do not have easy answers. The hope is that those who submit proposals will take up these questions and engage in collective thinking and imagining about the truths we hold, the truths we challenge, and the responsibilities we bear in co-creating a free and more equitable world. All panel proposals should have a statement about how the panel has incorporated the goals of equity, diversity, and inclusion, and/or analysis of power.

Executive Session proposals have already been selected and speak directly to the theme “Truth and Responsibility.” To practice the critical reflexivity this call for papers invites, we asked that Executive Session submissions explicitly address the diversity of session participants, such as organizational/institutional affiliation, career stage, race, ethnicity, disability, sexuality, class, religion, national identity, and gender. Proposals for Executive Sessions that demonstrated a commitment to equity and inclusion and/or a clear analysis of power (i.e. anti-imperialist, anti-white supremacist, decolonial, anti-ableist, feminist, anti-transphobic, anti-xenophobic, etc) were given priority. We are excited to be in conversation with residents of the Baltimore region and look forward to the discussions, performances, and actions generated by the dynamic group of anthropologists attending the 2021 AAA Annual Meeting.

AAA Annual Meeting 2021

“Truth and Responsibility” is a call to reimagine anthropology to meet the demands of the present moment. The imperative to bear witness, take action, and be held accountable to the truths we write and circulate invites us to reflect on our responsibility in reckoning with disciplinary histories, harms, and possibilities. To whom are we giving evidence and toward what ends? For whom are we writing? To whom are we accountable, and in what ways? We are thrilled to announce the theme of the 2021 AAA Annual Meeting to be held in Baltimore, MD: Truth and Responsibility.

poster

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community."<br>

*Message from the General Chair, Dirk Hovy*





<br>

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

In recent years, developing compact and efficient large language models (LLMs) has emerged as a thriving area of research. However, traditional Supervised Fine-Tuning (SFT), which relies on singular ground truth labels, often fails to capture token-level dependencies and linguistic diversity. To address these limitations, we propose a logits-based fine-tuning framework that integrates the strengths of supervised learning and knowledge distillation. Our approach constructs enriched training targets by combining teacher logits with ground truth labels, preserving both correctness and linguistic diversity. This ensures more reliable and effective training. To validate our approach, we constructed a large-scale 1.2M logits dataset and trained a series of science-focused models. Experimental results demonstrate that our method achieves significant improvements over current SOTA, with accuracy gains of 18% on Mawps and 22.7% on TabMWP. Across nine widely used mathematical benchmarks, our method consistently outperforms prior SFT models, achieving an average improvement of 7.28%. All code and datasets will be open-sourced.

Logits-Based Finetuning

Compressing Large Language Models (LLMs) into task-specific Small Language Models (SLMs) encounters two significant challenges: safeguarding domain-specific knowledge privacy and managing limited resources. To tackle these challenges, we propose PPC-GPT, a novel unified framework that systematically addresses both privacy preservation and model compression in federated settings. PPC-GPT works on a server-client federated architecture, where the client sends differentially private (DP) perturbed task-specific data to the server's LLM. The LLM then generates synthetic data along with their corresponding rationales. This synthetic data is subsequently used for both LLM pruning and retraining processes. Our framework's key innovation lies in its holistic integration of privacy-preserving mechanisms, synthetic data generation, and task-specific compression techniques, creating unique benefits through component interaction. Our experiments across diverse text generation tasks demonstrate that PPC-GPT successfully achieves dual objectives: maintaining competitive performance comparable to full-sized LLMs while ensuring robust privacy protection through its federated architecture.

PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation

This work presents a novel trie (prefix-tree)-based parallel decoding method that addresses the memory inefficiency of batch-based beam search. By sharing a single KV cache across beams with common prefixes, our approach dramatically reduces memory usage and enables efficient decoding. We evaluated our method across three attention architectures--Multi-Head Attention (Phi-3.5-mini-instruct), Grouped Query Attention (Llama-3.1-8B-Instruct), and Sliding Window Attention (Mistral-Small-24B-Instruct-2501)--using CNN/DailyMail for abstractive summarization and HumanEval for code generation. Our experiments demonstrate substantial memory savings (4–8×) and notable decoding speed improvements (up to 2.4×), without compromising generation quality. These results highlight the method's suitability for memory-constrained environments and large-scale deployments.

Efficient Beam Search for Large Language Models Using Trie-Based Decoding

Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.

FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference

Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be Western-centric. This introduces a significant gap in fairly evaluating multilingual models' comprehension of factual information from diverse geographical locations. To address this, we introduce XNationQA to investigate the cultural literacy of multilingual LLMs. XNationQA encompasses questions on the geography, culture, and history of nine countries, containing a total of 49,280 questions in seven languages. We benchmark eight standard multilingual LLMs on XNationQA. Our analyses uncover a considerable discrepancy in the model's accessibility to culturally specific facts across languages. Notably, we often find that a model demonstrates greater knowledge of cultural information in English than in the dominant language of the respective culture. The models show better performance in Western languages, though this does not translate to their being more literate for Western countries, which is counter-intuitive. Furthermore, we observe that models have a very limited ability to transfer knowledge across languages, particularly evident in open-source models.

 Do You Know About My Nation? Investigating Multilingual Language Models’ Cultural Literacy Through Factual Knowledge

Fine-tuning pre-trained models on targeted datasets enhances task-specific performance but often comes at the expense of generalization. Model merging techniques, which integrate multiple fine-tuned models into a single multi-task model through task arithmetic, offer a promising solution. However, task interference remains a fundamental challenge, leading to performance degradation and suboptimal merged models. Existing approaches largely overlook the fundamental roles of neurons, their connectivity, and activation, resulting in a merging process and a merged model that does not consider how neurons relay and process information. In this work, we present the first study that relies on neuronal mechanisms for model merging. We decompose task-specific representations into two complementary neuronal subspaces that regulate neuron sensitivity and input adaptability. Leveraging this decomposition, we introduce NeuroMerging, a novel merging framework developed to mitigate task interference within neuronal subspaces, enabling training-free model fusion across diverse tasks. Through extensive experiments, we demonstrate that NeuroMerging achieves superior performance compared to existing methods on multi-task benchmarks across both natural language and vision domains. Our findings highlight the importance of aligning neuronal mechanisms in model merging, offering new insights into mitigating task interference and improving knowledge fusion. Code will be released upon acceptance.

To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging

Upgrading embedding models in production vector databases typically necessitates re-encoding the entire corpus and rebuilding the Approximate Nearest Neighbor (ANN) index, leading to significant operational disruption and computational cost. This paper presents Drift-Adapter, a lightweight, learnable transformation layer designed to bridge embedding spaces between model versions. By mapping new queries into the legacy embedding space, Drift-Adapter enables the continued use of the existing ANN index, effectively deferring full re-computation. We systematically evaluate three adapter parameterizations: Orthogonal Procrustes, Low-Rank Affine, and a compact Residual MLP, trained on a small sample of paired old/new embeddings. Experiments on MTEB text corpora and a CLIP image model upgrade (1M items) show that Drift-Adapter recovers 95–99% of the retrieval recall (Recall@10, MRR) of a full re-embedding, adding less than 10,μs query latency. Compared to operational strategies like full re-indexing or dual-index serving, Drift-Adapter dramatically reduces recompute costs (by over 100 times) and facilitates upgrades with near-zero operational interruption. We analyze robustness to varied model drift, training data size, scalability to billion-item systems, and the impact of design choices like diagonal scaling, demonstrating Drift-Adapter's viability as a pragmatic solution for agile model deployment.

Drift-Adapter: A Practical Approach to Near Zero-Downtime Embedding Model Upgrades in Vector Databases

Mixture-of-Experts (MoE) layers scale transformers by routing tokens to a sparse subset of feed-forward experts. Token-level routing, however, assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialisation. We introduce SliceMoE, an architecture that routes contiguous slices of a token's hidden vector. A d-dimensional embedding is partitioned into S slices, and for each slice, a lightweight shared router predicts the top-k experts. Experts operate on their assigned slices independently, and outputs are re-assembled, maintaining per-token FLOP efficiency. Because slices from different tokens interleave within an expert, utilisation is naturally smoother. We propose a slice-level capacity loss, cross-slice dropout, and efficient fused batched-GEMM kernels. Experiments on WikiText-103 language modelling, WMT En–De translation, and three text-classification datasets show SliceMoE attains up to 1.7x faster inference than dense baselines, 12–18% lower perplexity than parameter-matched token-MoE, and improved expert balance, with interpretable expertise over syntactic versus semantic sub-spaces.

SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling

Large language models (LLMs) tend to follow maliciously crafted instructions to generate deceptive responses, posing safety challenges. How deceptive instructions alter the internal representations of LLM compared to truthful ones remains poorly understood beyond output analysis. To bridge this gap, we investigate when and how these representations ``flip'', such as from truthful to deceptive, under deceptive versus truthful/neutral instructions. Analyzing the internal representations of Llama-3.1-8B-Instruct and Gemma-2-9B-Instruct on a factual verification task, we find the model's instructed True/False output is predictable via linear probes across all conditions based on the internal representation. Further, we use Sparse Autoencoders (SAEs) to show that the Deceptive instructions induce significant representational shifts compared to Truthful/Neutral representations (which are similar), concentrated in early-to-mid layers and detectable even on complex datasets. We also identify specific SAE features highly sensitive to deceptive instruction and use targeted visualizations to confirm distinct truthful/deceptive representational subspaces.

When Truthful Representations Flip Under Deceptive Instructions?

Conventional speculative decoding (SD) methods utilize a predefined length policy for proposing drafts, which implies the premise that the target model smoothly accepts the proposed draft tokens. However, reality deviates from this assumption: the oracle draft length varies significantly, and the fixed-length policy hardly satisfies such a requirement. Moreover, such discrepancy is further exacerbated in scenarios involving complex reasoning and long-form generation, particularly under test-time scaling for reasoning-specialized models. Through both theoretical and empirical estimation, we establish that the discrepancy between the draft and target models can be approximated by the draft model's prediction entropy: a high entropy indicates a low acceptance rate of draft tokens, and vice versa. Based on this insight, we propose SVIP: Self-Verification Length Policy for Long-Context Speculative Decoding, which is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy. Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP, achieving up to 17% speedup on MT-Bench at 8K context compared with fixed draft lengths, and 22% speedup for QwQ in long-form reasoning.

Downloads

Next from EMNLP 2025

Logits-Based Finetuning

Similar lecture

On Fire: Risk, Regeneration, Value