Singapore

&#39;LLM-as-a-judge,&#39;&#39; which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks.
However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance.
This study investigates the cause of this bias.
Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests alignment reduces output diversity, we hypothesize that numerical bias arises from alignment.
To test this, we compare outputs from pre- and post-alignment LLMs, and observe that alignment indeed increases numerical bias.
We also explore mitigation strategies for post-alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment.
Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic.
Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.

AAAI 2026

Exploring the Effects of Alignment on Numerical Bias in Large Language Models

nlp: (large) language models

robustness & trustworthiness

peai: safety

fairness & equity

peai: bias

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

ACL 2025

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

'LLM-as-a-judge,'' which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks.
However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance.
This study investigates the cause of this bias.
Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests alignment reduces output diversity, we hypothesize that numerical bias arises from alignment.
To test this, we compare outputs from pre- and post-alignment LLMs, and observe that alignment indeed increases numerical bias.
We also explore mitigation strategies for post-alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment.
Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic.
Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.

workshop paper

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

Adaptations are one of the most prominent methods that enable the utilization of large backbone models for those who have limited computational resources. Yet, even in their last resort of using adaptations, many face a dead end in training when adaptation data are insufficient. Since the acquisition of adaptation data is arduous, the situation frequently appears, daunting without a particular solution. Often, even with extra training or random data augmentation, the performance of adaptations does not improve. To address this, we organize a solid probabilistic background and sampling method that focuses on the enormous amount of backbone data used for pre-training. We name the method, Backbone Augmented Training (BAT). First, we formulate and prove two key probabilistic propositions: one establishes the validity of BAT, while the other identifies a condition under which BAT benefits adaptation. Furthermore, we introduce an original data sampling scheme that satisfies these propositions and present ALgorithm for BAT (ALBAT) to efficiently implement this approach. ALBAT effectively enhances adaptation training even when other methods meet performance stalemate due to adaptation data scarcity.

BAT: Backbone Augmented Training for Adaptations

This paper investigates how the degree of group fairness changes when the degree of individual fairness is actively controlled. As a metric quantifying individual fairness, we consider generalized entropy (GE) recently introduced into machine learning community. To control the degree of individual fairness, we design a classification algorithm satisfying a given degree of individual fairness through an empirical risk minimization (ERM) with a fairness constraint specified in terms of GE. We show the PAC learnability of the fair ERM problem by proving that the true fairness degree does not deviate much from an empirical one with high probability for finite VC dimension if the sample size is big enough. Our experiments show that strengthening individual fairness degree does not always lead to enhancement of group fairness.

Impacts of Individual Fairness on Group Fairness from the Perspective of Generalized Entropy

Semi-supervised object detection (SSOD) has achieved notable progress through advances in self-training frameworks. However, most existing approaches still rely solely on classification confidence to select pseudo-labels, overlooking the localization uncertainty. This results in pseudo-labels that lack sufficient localization information, since classification confidence alone does not adequately represent the accuracy of spatial localization. In this paper, we address this limitation from two complementary perspectives: edge-level reliability estimation and instance-level pseudo-label selection. From the former perspective, we quantify localization confidence using a simple statistic — the maximum adjacent probability sum derived from the Distribution Focal Loss — and aggregate edge-level scores into a box-level reliability via the minimum over edges. This suppresses boxes with at least one uncertain boundary, improving spatial precision with negligible computational overhead. From the latter perspective, we integrate this localization-confidence-aware criterion into Efficient Teacher, a YOLO-oriented semi-supervised object detection framework, to adaptively filter pseudo-labels during self-training. Experiments on the KITTI dataset using YOLOv8-m demonstrate consistent improvements over baselines and improved training stability. Further analysis reveals that our approach achieves higher recall of pseudo-labels while maintaining precision, validating the effectiveness of localization confidence modeling in SSOD.

Localization-Confidence-Aware Pseudo-Label Selection for YOLO-based Semi-Supervised Object Detection

Eye movements during reading provide valuable data for studying cognitive processes and text properties. While eye-tracking data can enhance language models, its application is limited by data scarcity, particularly for low-resource languages. Although synthetic scanpath generation offers a promising solution, current models are largely confined to English and Chinese. This work explores multilingual model for scanpath generation, demonstrating that a single model trained on 13 typologically diverse languages outperforms monolingual counterparts across most metrics. By leveraging shared linguistic patterns, our approach mitigates data scarcity for individual languages and facilitates synthetic gaze data creation for broader multilingual NLP applications.

Cross-Lingual Transfer Learning for Enhanced Synthetic Scanpath Prediction in Reading

Scene Text Editing (STE) is the task of modifying text content in an image while preserving its visual style, such as font, color, and background. While recent diffusion-based approaches have shown improvements in visual quality, key limitations remain: lack of support for low-resource languages, domain gap between synthetic and real data, and the absence of appropriate metrics for evaluating text style preservation. To address these challenges, we propose STELLAR (Scene Text Editor for Low-resource LAnguages and Real-world data). STELLAR enables reliable multilingual editing through a language‑adaptive glyph encoder and a multi‑stage training strategy that first pre‑trains on synthetic data and then fine‑tunes on real images. We also construct a new dataset, STIPLAR(Scene Text Image Pairs of Low-resource lAnguages and Real-world data), for training and evaluation. Furthermore, we propose Text Appearance Similarity (TAS), a novel metric that assesses style preservation by independently measuring font, color, and background similarity, enabling robust evaluation even without ground truth. Experimental results demonstrate that STELLAR outperforms state-of-the-art models in visual consistency and recognition accuracy, achieving an average TAS improvement of 2.2% across languages over the baselines.

STELLAR: Scene Text Editor for Low-Resource Languages and Real-World Data

View selection is critical in active 3D neural reconstruction as it impacts the contents of training set and resulting final output quality. Recent view selection strategies emphasize the visibility when evaluating model uncertainty in active 3D reconstruction.
However, existing approaches estimate visibility only after the model fully converges, which has confined their application primarily to non-continuous active learning settings. This paper proposes Surface-Based Visibility field (SBV) that successfully estimates the visibility-guided uncertainty in continuous active 3D neural reconstruction. During learning neural implicit surfaces, our model learns rendering uncertainties and infers surface confidence values derived from signed distance functions. It then updates surface confidences using a voxel grid, robustly deducing the surface-based visibility for uncertainties.
This approach captures uncertainties across all regions, whether well-defined surfaces or ambiguous areas, ensuring accurate visibility measurement in continuous active learning. 
Experiments on benchmark datasets—Tanks and Temples, BlendedMVS, Blender, DTU—and the newly proposed imbalanced viewpoint dataset (ImBView) show that view selection based on SBV-guided uncertainty improves performance by up to 11.6% over existing methods, highlighting its effectiveness in challenging reconstruction scenarios.

Surface-Based Visibility-Guided Uncertainty for Continuous 3D Active Neural Reconstruction

Large Language Models (LLMs) can be conditioned with explicit personality prompts, yet their behavioral realization often varies depending on context. This study examines how identical personality prompts lead to distinct linguistic, behavioral, and emotional outcomes across four conversational settings—ice-breaking, negotiation, group decision, and empathy tasks. Results show that contextual cues systematically influence both personality expression and emotional tone, suggesting that the same traits are expressed differently depending on social and affective demands. This raises an important question for LLM-based dialogue agents: whether such variations reflect inconsistency or context-sensitive adaptation akin to human behavior. Viewed through the lens of Whole Trait Theory, these findings highlight that LLMs exhibit context-sensitive rather than fixed personality expression, adapting flexibly to social interaction goals and affective conditions.

Personality Expression Across Contexts: Linguistic and Behavioral Variation in LLM Agents

Personality traits are increasingly incorporated into AI agents in applications involving human interaction. However, it remains uncertain whether LLMs truly manifest the intended traits when prompted. In this paper, we evaluate six LLMs: GPT-4o Mini, Llama 3.2, Mistral NeMo, Gemini 2.0 Flash Lite, Gemma 2, and Claude 3 Haiku by prompting high and low configurations of the Big Five personality traits and as- sessing both overall trait alignment and item-level inconsis- tencies through asking the prompted LLM to take a person- ality survey. Our analysis reveals pronounced differences in how these LLMs express prompted personalities, with Llama 3.2, Mistral NeMo, and Claude 3 Haiku struggling to reflect specific trait items in low configurations, and GPT-4o Mini, Gemma 2, and Gemini 2.0 Flash Lite exhibiting strong per- sonality alignment under high-configuration settings. These findings show a personality misalignment, where LLMs do not necessarily express the intended traits as expected after personality prompting.

Evaluating LLM Alignment under Big Five Personality Prompting

In online chat platforms, multiple dialogues are often entangled in a complex manner. Disentangling them into coherent dialogues is essential for understanding conversations. Supervised models have long dominated this task. The application of large language models (LLMs) to this problem remains underexplored; however, preliminary studies have shown their performance to be substantially inferior to that of machine-learning methods. This raises fundamental questions: Do LLMs inherently lack the capability for dialogue disentanglement, and what approaches are necessary to enhance their performance? We answer these questions by introducing two novel methods: dialogue-level assignment (DLA), which tasks LLMs with assigning an utterance to a dialogue, and subsequent context (SC), which provides subsequent context as auxiliary evidence. Experiments on the benchmark IRC dataset show that our method, equipped with DLA+SC, achieves new state-of-the-art results across all metrics. These results prove that LLMs possess a strong capability for the studied task. Furthermore, our ablation study on open-source models investigates the effectiveness of DLA and SC, revealing that it is model- dependent. Nevertheless, it also demonstrates that both methods are key factors in improving performance. This research marks a paradigm shift for dialogue disentanglement, transitioning from machine-learning approaches to applying LLMs.

Rethinking Dialogue Disentanglement for LLMs via Dialogue-Level Assignment and Subsequent Context

Large Language Models (LLMs) have demonstrated remarkable success in conversational systems by generating human-like responses. However, they can fall short, especially when required to account for personalization or specific knowledge. In real-life settings, it is impractical to rely on users to detect these errors and request a new response. One way to address this problem is to refine the response before returning it to the user. While existing approaches focus on refining responses within a single LLM, this method struggles to consider diverse aspects needed for effective conversations. In this work, we propose refining responses through a multi-agent framework, where each agent is assigned a specific role for each aspect. We focus on three key aspects crucial to conversational quality: factuality, personalization, and coherence. Each agent is responsible for reviewing and refining one of these aspects, and their feedback is then merged to improve the overall response. To enhance collaboration among them, we introduce a dynamic communication strategy. Instead of following a fixed sequence of agents, our approach adaptively selects and coordinates the most relevant agents based on the specific requirements of each query. We validate our framework on challenging conversational datasets, demonstrating that ours significantly outperforms relevant baselines, particularly in tasks involving knowledge or user’s persona, or both.

Downloads

Next from AAAI 2026

BAT: Backbone Augmented Training for Adaptations

Similar lecture

McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2026

BAT: Backbone Augmented Training for Adaptations

Similar lecture

McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models

Downloads