Austria

NLP research frequently grapples with multiple sources of variability—spanning runs, datasets, annotators, and more—yet conventional analysis methods often neglect these hierarchical structures, threatening the reproducibility of findings. To address this gap, we contribute a case study illustrating how linear mixed-effects models (LMMs) can rigorously capture systematic language-dependent differences (i.e., population-level effects) in a population of monolingual and multilingual language models. In the context of a bilingual hate speech detection task, we demonstrate that LMMs can uncover significant population-level effects—even under low-resource (small-N) experimental designs—while mitigating confounds and random noise. By setting out a transparent blueprint for repeated-measures experimentation, we encourage the NLP community to embrace variability as a feature, rather than a nuisance, in order to advance more robust, reproducible, and ultimately trustworthy results.

ACL 2025

Robust Estimation of Population-Level Effects in Repeated-Measures NLP Experimental Designs

linear mixed models

cross-lingual evaluation

statistical analysis

Congratulations to all the researchers who have qualified for the AMA Research Challenge virtual poster symposium and semifinals!

Welcome to the **largest national, multi-specialty research event** featuring research from nearly 1,000 medical students, residents, fellows, and international medical graduates across six topic areas - basic science, clinical and translational research, clinical vignettes, health systems science, medical education, and public health and health policy.

#### Access the Event
Use the buttons below or navigation menu to access the event. 

[![](https://assets.underline.io/markdown_image/1/image/d4076f2146939317a29eb9f69c53ad41.png)](https://underline.io/events/506/sessions?searchGroup=all)
[![](https://assets.underline.io/markdown_image/1/image/6bcee0b523a4874c4292ca0415d88dde.png)](https://underline.io/events/506/posters?searchGroup=all)

#### Event Details
The 2025 AMA Research Challenge includes the following symposium halls:
* **Semifinals hall:** Features research from 54 medical students, residents and international medical graduates that received top scores during first round scoring. 
* **Poster symposium hall:** Features research from nearly 1,000 medical students, residents and international medical graduates categorized by topics and subtopics.

**Here’s how to participate:**
* **View posters and presentations** on a variety of topics and specialties
* **Score posters in the semifinals** to help decide the top five finalists (AMA member exclusive opportunity)
* **Connect with presenters** and provide feedback to help advance their research
* **New this year: Watch live poster presentations** by topic area and engage directly with researchers

At the conclusion of this event, the AMA announced the top five finalists who advanced to the AMA Research Challenge final round, presented their research to an elite panel of judges, and competed to win a $10,000 grand prize presented by Laurel Road.

**Addison Shenk, a fourth-year medical student from Edward Via Virginia College of Osteopathic Medicine, was the winner of the 2025 AMA Research Challenge and was awarded the grand prize of $10,000, presented by Laurel Road.**

#### A message from Dr. Bobby Mukkamala, AMA President
<div style="position: relative; width: 100%; padding-bottom: 56.25%; height: 0; overflow: hidden;">
  <iframe 
    src="https://underline.io/embed/136894-ama-video?t=0"
    title="AMA Welcome Video"
    allowfullscreen
    style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: 0;">
  </iframe>
</div>


#### Recognized Institutions
In 2025, 1,400 abstracts were submitted across six topic areas representing 174 medical schools and more than 150 residency and health care institutions. [See if your institution is represented this year!](https://drive.google.com/file/d/1eFrTqCmCeobx9pr5drJK2hllvsIpFNOY/view?usp=drive_link)

#### Grand Prize Sponsor
AMA would like to thank Laurel Road, a brand of KeyBank, for sponsoring a $10,000 grand prize for the winner of the AMA Research Challenge. Laurel Road and KeyBank are not involved in winner selection process. The winner will be announced during the AMA Research Challenge final round. Payment of the grand prize will be directly from Laurel Road/KeyBank. See the [AMA Research Challenge Official Rules for terms and conditions](https://www.ama-assn.org/system/files/research-challenge-official-rules.pdf).

[![](https://assets.underline.io/uploads/markdown_image/1/image/611e331d4a0eba0920f240cfc04b4d77.png)](https://www.laurelroad.com/)

#### Code of Conduct
<html>
AMA has a code of conduct for its meetings – we ask all attendees to adhere to this code to participate in this year’s meeting. Please read and <a href="https://www.ama-assn.org/about/code-conduct" target="_blank">learn more here</a> about your conduct requirements.
<br><br>
	</html>
	

#### Semifinals Judges

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Need help? Contact us at <support@underline.io>.

Please log in to explore this event.

If you have used the AMA Research Challenge platform previously, please click the ‘Log-in’ button below. If you do not remember your password, please click “forgot password” and follow prompts to reset your password. After resetting your password, please refresh your webpage to access the platform. 

Register to explore research in a variety of topics. AMA members get exclusive access to score posters in the semifinals. 

**Please use [this link here](https://www.ama-assn.org/member-benefits/events/ama-research-challenge) to register for free. **

Registration Is Required

AMA Research Challenge 2025 

Join the AMA Research Challenge poster symposium and semifinals to explore top research posters and score posters in the semifinals (an AMA member exclusive opportunity).

poster

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair:<br>
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for  Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025.  ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am  deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.*<br>
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)<br><br><br>



To access this event page, you need to log in with the **email address you registered with**. <br>Access credentials will be sent to your email from Underline -  subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Code-switching is prevalent in multilingual communities but lacks adequate high-quality data for model development, especially for African languages. To address this, we present AfroCS-xs, a small human-validated synthetic code-switched dataset for four African languages (Afrikaans, Sesotho, Yoruba, isiZulu) and English within a specific domain—agriculture. Using large language models (LLMs), we generate code-switched sentences, including English translations, that are rigorously validated and corrected by native speakers. As a downstream evaluation task, we use this dataset to fine-tune different instruction-tuned LLMs for code-switched translation and compare their performance against machine translation (MT) models. Our results demonstrate that LLMs consistently improve in translation accuracy when fine-tuned on the high-quality AfroCS-xs dataset, highlighting that substantial gains can still be made with a low volume of data. We also observe improvements on natural code-switched and out-of-domain (personal finance) test sets. Overall, regardless of data size and prior exposure to a language, LLMs benefit from higher quality training data when translating code-switched texts in under-represented languages.

AfroCS-xs: Creating a Compact, High-Quality, Human-Validated Code-Switched Dataset for African Languages

This paper focuses on efficient Large Language Model data compression. Considering the linear context growth of self-evaluating and divide-and-conquer LLM modeling methods, techniques are needed to manage the size of shared context. Existing approaches compress data through prompt tuning, using detailed instructions to guide the output. However, this method may be suboptimal: (i) defining principles may restrict an LLM's inherent ability to compress data; (ii) longer prompts increase the overhead needed to process data.

To address these issues, we built upon the framework proposed by LLMLingua2, which formulates data compression as a token classification problem, and trains knowledge distilled models on data generated using compression prompts. We observed their model's output, designed new prompts targeted at areas of improvement, and evaluated on downstream tasks, such as summarization, question answering and mathematics. We then test our best prompting method on the summarization task of MeetingBank, 3% the size of LLMLingua2's prompt, while achieving a 61% size reduction of distilled data and higher model evaluation result than LLMLingua2's prompting method on all eight different metrics, at a low resource level of 1000 training pairs.

Overcoming Self-Imposed Limits: Five Words to Break an LLM's Context Compression Barrier

Direct speech translation (ST) has garnered increasing attention nowadays, yet the accurate translation of terminology within utterances remains a great challenge. In this regard, current studies mainly concentrate on leveraging various translation knowledge into ST models. However, these methods often struggle with interference from irrelevant noise and can not fully utilize the translation knowledge. To address these issues, in this paper, we propose a novel Locate-and-Focus method for terminology translation. It first effectively locates the speech clips containing terminologies within the utterance to construct translation knowledge, minimizing irrelevant information for the ST model. Subsequently, it associates the translation knowledge with the utterance and hypothesis from both audio and textual modalities, allowing the ST model to better focus on translation knowledge during translation. Experimental results across various datasets demonstrate that our method effectively locates terminologies within utterances and enhances the success rate of terminology translation, while maintaining robust general translation performance.

Locate-and-Focus: Enhancing Terminology Translation in Speech Language Models

The recent progress in Vision-Language Models (VLMs) has broadened the scope of multimodal applications. However, evaluations often remain limited to functional tasks, neglecting abstract dimensions such as personality traits and human values. To address this gap, we introduce Value-Spectrum, a novel Visual Question Answering (VQA) benchmark aimed at assessing VLMs based on Schwartz's value dimensions that capture core human values guiding people’s preferences and actions. We design a VLM agent pipeline to simulate video browsing and construct a vector database comprising over 50,000 short videos from TikTok, YouTube Shorts, and Instagram Reels. These videos span multiple months and cover diverse topics, including family, health, hobbies, society, technology, etc. Benchmarking on Value-Spectrum highlights notable variations in how VLMs handle value-oriented content. Beyond identifying VLMs' intrinsic preferences, we also explore the ability of VLM agents to adopt specific personas when explicitly prompted, revealing insights into the adaptability of the model in role-playing scenarios. These findings highlight the potential of Value-Spectrum as a comprehensive evaluation set for tracking VLM preferences in value-based tasks and abilities to simulate diverse personas. The complete code and data are available at https://github.com/Jeremyyny/Value-Spectrum.

Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts

One of the research focuses of large language models (LLMs) is the ability to generate action plans. Recent studies have revealed that the performance of LLMs can be significantly improved by integrating external tools. Based on this, we propose a benchmark framework called PlanningArena, which aims to simulate real application scenarios and provide a series of apps and API tools that may be involved in the actual planning process. This framework adopts a modular task structure and combines user portrait analysis to evaluate the ability of LLMs in correctly selecting tools, logical reasoning in complex scenarios, and parsing user information. In addition, we deeply diagnose the task execution effect of LLMs from both macro and micro levels. The experimental results show that even the most outstanding GPT-4o and DeepSeekV3 models only achieved a total score of 56.5% and 41.9% in PlanningArena, respectively, indicating that current LLMs still face challenges in logical reasoning, context memory, and tool calling when dealing with different structures, scenarios, and their complexity. Through this benchmark, we further explore the path to optimize LLMs to perform planning tasks.

PlanningArena: A Modular Benchmark for Multidimensional Evaluation of Planning and Tool Learning

Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits.

Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction

We present FlagEvalMM, an open-source evaluation framework designed to comprehensively assess multimodal models across a diverse range of vision-language understanding and generation tasks, such as visual question answering, text-to-image/video generation, and image-text retrieval. We decouple model inference from evaluation through an independent evaluation service, thus enabling flexible resource allocation and seamless integration of new tasks and models. Moreover, FlagEvalMM utilizes advanced inference acceleration tools (e.g., vLLM, SGLang) and asynchronous data loading to significantly enhance evaluation efficiency. Extensive experiments show that FlagEvalMM offers accurate and efficient insights into model strengths and limitations, making it a valuable tool for advancing multimodal research. The framework is publicly accessible at https://github.com/flageval-baai/FlagEvalMM, with a demonstration video available at https://youtu.be/L7EtacjoM0k.

FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation

Pretrained language models have significantly advanced the state of the art in generating distributed representations of text. However, they do not account for the wide variety of available expert-generated language resources and lexicons that explicitly encode linguistic/domain knowledge. Such lexicons can be paired with learned embeddings to further enhance NLP prediction and linguistic inquiry. In this work we present Textagon, a Python package for generating parallel representations for text based on predefined lexicons and selecting representations that provide the most information. We discuss the motivation behind the software, its implementation, as well as two case studies for its use to demonstrate operational utility.

Textagon: Boosting Language Models with Theory-guided Parallel Representations

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open source, with the aim of expanding NLP research opportunities and supporting the development of industrial solutions for the Russian language.

GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

Language agents powered by large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and executing complex tasks. However, developing robust agents presents significant challenges: substantial engineering overhead, lack of standardized components, and insufficient evaluation frameworks for fair comparison. We introduce Agent Graph-based Orchestration for Reasoning and Assessment (AGORA), a flexible and extensible framework that addresses these challenges through three key contributions: (1) a modular architecture with a graph-based workflow engine, efficient memory management, and clean component abstraction; (2) a comprehensive suite of reusable agent algorithms implementing state-of-the-art reasoning approaches; and (3) a rigorous evaluation framework enabling systematic comparison across multiple dimensions. Through extensive experiments on mathematical reasoning and multimodal tasks, we evaluate various agent algorithms across different LLMs, revealing important insights about their relative strengths and applicability. Our results demonstrate that while sophisticated reasoning approaches can enhance agent capabilities, simpler methods like Chain-of-Thought often exhibit robust performance with significantly lower computational overhead. AGORA not only simplifies language agent development but also establishes a foundation for reproducible agent research through standardized evaluation protocols.
We made a demo video at: https://www.youtube.com/watch?v=WRH-F1zegKI. The comparison agent of algorithms is also available at https://huggingface.co/spaces/omlab/open-agent-leaderboard. Source code of AGORA can be found at https://github.com/om-ai-lab/OmAgent.

Downloads

Next from ACL 2025

AfroCS-xs: Creating a Compact, High-Quality, Human-Validated Code-Switched Dataset for African Languages

Similar lecture

Approximating SIR-type structured-population models using data-driven homogeneous-population analogs