United States

NAACL 2025

Closing Ceremony

[![](https://assets.underline.io/uploads/markdown_image/1/image/9a88ae95fbde06e80f4bcf6d4f7d133a.png)](https://certificateaixia.paperform.co/)
The certificate will be provided by December 17. 
In case you need it urgently please contact: francesca.gasparini@unimib.it. 

 

The 20th International Conference of the Italian Association for Artificial Intelligence (**AIxIA 2021**) is organized by AIxIA (Associazione Italiana per l’Intelligenza Artificiale), which is a non-profit scientific society founded in 1988 and devoted to the promotion of Artificial Intelligence. The society aims to increase the public awareness of AI, encourage the teaching of it and promote research in the field.

 
 



**General Chair**

Stefania Bandini, University of Milano-Bicocca, Italyand RCAST - The University of Tokyo, Japan


**Program Chairs**

Viviana Mascardi, University of Genova, Italy; Matteo Palmonari, University of Milano-Bicocca, Italy; Giuseppe Vizzari, University of Milano-Bicocca, Italy

**Organizing Committee**

Francesca Gasparini, University of Milano-Bicocca, Italy (**Chair**); Stefano Ferilli, University of Bari, Italy; Marta Giltri, University of Milano-Bicocca, Italy; Daniela Briola, University of Insubria, Italy


**Publicity and Communication Committee**

Elisabetta Fersini, University of Milano-Bicocca, Italy (**Chair**); Angelo Ferrando, University of Genova, Italy; Francesca Alessandra Lisi, University of Bari, Italy; Giovanni Stilo, University of L'Aquila, Italy; Marco Viviani, University of Milano-Bicocca, Italy


**Workshop Chairs**

Gabriella Cortellessa, ISTC-CNR, Rome, Italy; Silvia Rossi, University of Naples Federico II, Italy; Fabio Stella, University of Milano-Bicocca, Italy


**PC Members**

Fulvio Mastrogiovanni, University of Genova, Italy; Roberto Basili, University of Rome Tor Vergata, Italy; Nicola Gatti, Polytechnic University of Milan, Italy; Stefano Bistarelli, University of Perugia, Italy; Piero Poccianti, Consorzio Operativo Gruppo MPS, Italy; Francesco Scarcello, University of Calabria, Italy; Flavio S. Correa Da Silva, University of São Paulo, Brazil; Elena Cabrio, Université Côte d'Azur, CNRS, Inria, I3S, France; Antonio Lieto, University of Turin, Italy; Diego Calvanese, Free University of Bozen-Bolzano, Italy; Alessio Micheli, University of Pisa, Italy; Francesca A. Lisi University of Bari Aldo Moro, Italy; Mauro Vallati, University of Huddersfield, United Kingdom; Bernardo Magnini, Fondazione Bruno Kessler (FBK), Italy; Luigia C. Aiello, Sapienza University of Rome, Italy; Rafael H. Bordini, Pontifical Catholic University of Rio Grande do Sul, Brazil; Chiara Ghidini, Fondazione Bruno Kessler (FBK), Italy; Federico Bergenti, University of Parma, Italy; Francesco Ricca, University of Calabria, Italy; Andrea Burattin, Technical University of Denmark, Denmark; Angelo Oddi, ISTC-CNR, Italian National Research Council, Italy; Farshad Badie, Aalborg University, Denmark; Emanuel Sallinger, University of Oxford, United Kingdom; Marin Lujak, University Rey Juan Carlos, Spain; Giancarlo Fortino, University of Calabria, Italy; Salvatore Gaglio, University of Palermo, Italy; Johannes K. Fichte, TU Dresden, Germany; Matteo Baldoni, University of Turin, Italy; Fabrizio Riguzzi, Univeristy of Ferrara, Italy; Ivana Dusparic, Trinity College Dublin, Ireland; Serena Villata, CNRS - Laboratoire d'Informatique, Signaux et Systèmes de Sophia-Antipolis, France; Paolo Rosso, Technical University of Valencia, Spain; Ana L. C. Bazzan, Federal University of Rio Grande do Sul, Brazil; Gianluigi Greco, University of Calabria, Italy; Stefano Cagnoni, University of Parma, Italy; Gian Luca Pozzato, University of Turin, Italy; Salvatore Ruggieri, University of Turin, Italy; Claudio Di Ciccio, Sapienza University of Rome, Italy; Eloisa Vargiu, Cetaqua, Water Technology Center, Spain; Marco Maratea, University of Genova, Italy; Andrea Roli, University of Bologna, Italy; Viviana Patti, University of Turin, Italy; Stefania Costantini, University of L'Aquila, Italy; Antonius Weinzierl, Vienna University of Technology, Austria; Silvia Rossi, University of Naples Federico II, Italy; Giovanni Semeraro, University of Bari, Italy; Luciano Serafini, Fondazione Bruno Kessler (FBK), Italy; Laura Giordano, University of Eastern Piedmont, Italy; Ernesto Jimenez-Ruiz, City, University of London, United Kingdom; Marco Gavanelli, University of Ferrara, Italy; Federico Chesani, University of Bologna, Italy

Please register with AIxIA to be able to enter the conference on Underline.io. If you have already registered, please login to Underline.io to be able to see full conference content. In case you do not have an Underline account, please create one. Thank you

Please login to your Underline account

AIxIA 2021

The 20th International Conference of the Italian Association for Artificial Intelligence (AIxIA 2021) is organized by AIxIA (Associazione Italiana per l’Intelligenza Artificiale), which is a non-profit scientific society founded in 1988 and devoted to the promotion of Artificial Intelligence. The society aims to increase the public awareness of AI, encourage the teaching of it and promote research in the field.

### Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Welcome to the 2025 meeting of the Nations of the Americas Chapter of the Association for Computational Linguistics! I am proud to help organize the first NAACL conference to carry the new name of our organization, one that emphasizes inclusion for all of the Americas. I am also pleased to welcome you to Albuquerque, New Mexico, a state whose unique blend of cultural influences will make for an excellent backdrop for NAACL 2025, especially with this year’s special theme on NLP in a Multicultural World. 
**[Continue reading...](https://drive.google.com/file/d/1jX-qGhqVSZZCIrAnJaz798pflu5Irrdn/view?usp=sharing)**

*- Colin Cherry, Google, NAACL 2025 General Chair* 

[![](https://assets.underline.io/markdown_image/1/image/b087f8a4dc5816d6e1a6514e59c59ac3.png)](https://drive.google.com/file/d/1T96GzPqObXrMln2BMByCSXTSizjTg69P/view?usp=sharing)

You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

To access NAACL 2025 event page you are required to register. Please follow [**this link**](https://2025.naacl.org/registration/registration/) to register. Access will depend on your registration type.

Please register!

Welcome to 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

Large Vision-Language Models (VLMs) have demonstrated strong capabilities in tasks requiring a fine-grained understanding of literal meaning in images and text, such as visual question-answering or visual entailment. However, there has been little exploration of the capabilities of these models when presented with images and captions containing figurative meaning, such as metaphors or humor. To close this gap, we propose a new task framing the figurative meaning understanding problem as an explainable visual entailment task, where the model has to predict whether the image (premise) entails a caption (hypothesis) and justify the predicted label with a textual explanation. The figurative phenomena can be present in the image, in the caption, or both. Using a human-AI collaboration approach, we build the accompanying expert-verified dataset V-FLUTE, containing 6,027 {image, caption, label, explanation} instances spanning five diverse figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. Through automatic evaluation, we find that VLMs struggle to generalize from literal to figurative meaning, particularly when it is present in images. Further, we identify common types of errors in VLM reasoning (hallucination and incomplete or unsound reasoning) across classes of models via human evaluation.

Understanding Figurative Meaning through Explainable Visual Entailment

In this paper, we investigate whether current state-of-the-art large language models (LLMs) are effective as AI tutors and whether they demonstrate pedagogical abilities necessary for good AI tutoring in educational dialogues. Previous efforts towards evaluation have been limited to subjective protocols and benchmarks. To bridge this gap, we propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles, which is designed to assess the pedagogical value of LLM-powered AI tutor responses grounded in student mistakes or confusions in the mathematical domain. We release MRBench – a new evaluation benchmark containing 192 conversations and 1,596 responses from seven state-of-the-art LLM-based and human tutors, providing gold annotations for eight pedagogical dimensions. We assess reliability of the popular Prometheus2 and Llama-3.1-8B LLMs as evaluators and analyze each tutor’s pedagogical abilities, highlighting which LLMs are good tutors and which ones are more suitable as question-answering systems. We believe that the presented taxonomy, benchmark, and human-annotated labels will streamline the evaluation process and help track the progress in AI tutors’ development.

Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors

Users can divulge sensitive information to proprietary LLM providers, raising significant privacy concerns. While open-source models, hosted locally on the user's machine, alleviate some concerns, models that users can host locally are often less capable than proprietary frontier models. Toward preserving user privacy while retaining the best quality, we propose Privacy-Conscious Delegation, a novel task for chaining API-based and local models. We utilize recent public collections of user-LLM interactions to construct a natural benchmark called PUPA, which contains personally identifiable information (PII). To study potential approaches, we devise PAPILLON, a multi-stage LLM pipeline that uses prompt optimization to address a simpler version of our task. Our best pipeline maintains high response quality for 85.5% of user queries while restricting privacy leakage to only 7.5%. We still leave a large margin to the generation quality of proprietary LLMs for future work.

PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles

Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models

Retrieval-Augmented Generation (RAG) systems have shown promise in enhancing the performance of Large Language Models (LLMs). However, these systems face challenges in effectively integrating external knowledge with the LLM's internal knowledge, often leading to issues with misleading or unhelpful information. This work aims to provide a systematic study on knowledge checking in RAG systems. We conduct a comprehensive analysis of LLM representation behaviors and demonstrate the significance of using representations in knowledge checking. Motivated by the findings, we further develop representation-based classifiers for knowledge filtering. We show substantial improvements in RAG performance, even when dealing with noisy knowledge databases. Our study provides new insights into leveraging LLM representations for enhancing the reliability and effectiveness of RAG systems.

Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective

Crowdsourcing has been increasingly utilized to gather subjective assessment, such as evaluating the toxicity of texts. Since there does not exist a single “ground truth” answer for subjective annotations, obtaining annotations to accurately reflect the opinions of different subgroups becomes a key objective for these subjective assessment tasks. Traditionally, this objective is accomplished by directly soliciting a large number of annotations from each subgroup, which can be costly especially when annotators of certain subgroups are hard to access. In this paper, using toxicity evaluation as an example, we explore the feasibility of using perspective taking—that is, asking annotators to take the point of views of a certain subgroup and estimate opinions within that subgroup—as a way to achieve this objective cost-efficiently. Our results show that compared to the baseline approach of directly soliciting annotations from the target subgroup, perspective taking could lead to better estimates of the subgroup-level opinion when annotations from the target subgroup is costly while the budget is limited. Moreover, prompting annotators to take the perspectives of contrasting subgroups simultaneously can further improve the quality of the estimates. Finally, we find that aggregating multiple perspective-taking annotations while soliciting a small number of annotations directly from the target subgroup for calibration leads to the highest-quality estimates under limited budget.

Exploring the Cost-Effectiveness of Perspective Taking in Crowdsourcing Subjective Assessment: A Case Study of Toxicity Detection

Debating over conflicting issues is a necessary first step towards resolving conflicts. However, intrinsic perspectives of an arguer are difficult to overcome by persuasive argumentation skills. Proceeding from a debate to a deliberative process, where we can identify actionable options for resolving a conflict requires a deeper analysis of arguments and the perspectives they are grounded in - as it is only from there that one can derive mutually agreeable resolution steps. In this work we develop a framework for a deliberative analysis of arguments in a computational argumentation setup. We conduct a fine-grained analysis of perspectivized stances expressed in the arguments of different arguers or stakeholders on a given issue, aiming not only to identify their opposing views, but also shared perspectives arising from their attitudes, values or needs. We formalize this analysis in Perspectivized Stance Vectors that characterize the individual perspectivized stances of all arguers on a given issue. We construct these vectors by determining issue- and argument-specific concepts, and predict an arguer's stance relative to each of them. The vectors allow us to measure a modulated (dis)agreement between arguers, structured by perspectives, which allows us to identify actionable points for conflict resolution, as a first step towards deliberation.

From Argumentation to Deliberation: Perspectivized Stance Vectors for Fine-grained (Dis)agreement Analysis

Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval. The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard.

The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design

LLMs have demonstrated impressive performance in answering medical questions, such as achieving passing scores on medical licensing examinations. However, medical board exams or general clinical questions do not capture the complexity of realistic clinical cases. Moreover, the lack of reference explanations means we cannot easily evaluate the reasoning of model decisions, a crucial component of supporting doctors in making complex medical decisions. To address these challenges, we construct two new datasets: JAMA Clinical Challenge and Medbullets. (Footnote: Datasets and code are available at <https://github.com/HanjieChen/ChallengeClinicalQA>.) JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises simulated clinical questions. Both datasets are structured as multiple-choice question-answering tasks, accompanied by expert-written explanations. We evaluate seven LLMs on the two datasets using various prompts. Experiments demonstrate that our datasets are harder than previous benchmarks. In-depth automatic and human evaluations of model-generated explanations provide insights into the promise and deficiency of LLMs for explainable medical QA.

Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, often assessed through multiple-choice questions (MCQs) that include an image, a question, and several options. However, many benchmarks used for such evaluations suffer from systematic biases. Remarkably, Large Language Models (LLMs) without any visual perception capabilities achieve non-trivial performance, undermining the credibility of these evaluations. To address this issue while maintaining the efficiency of MCQ evaluations, we propose MMEVALPRO, a benchmark designed to avoid Type-I errors through a trilogy evaluation pipeline and more rigorous metrics. For each original question from existing benchmarks, human annotators augment it by creating one perception question and one knowledge anchor question through a meticulous annotation process. MMEVALPRO comprises 2,138 question triplets, totaling 6,414 distinct questions. Two-thirds of these questions are manually labeled by human experts, while the rest are sourced from existing benchmarks (MMMU, ScienceQA, and MathVista). Compared with the existing benchmarks, our experiments with the latest LLMs and LMMs demonstrate that MMEVALPRO is **more challenging** (the best LMM lags behind human performance by 31.73%, compared to an average gap of 8.03% in previous benchmarks) and **more trustworthy** (the best LLM trails the best LMM by 23.09%, whereas the gap for previous benchmarks is just 14.64%). Our in-depth analysis explains the reason for the large performance gap and justifies the trustworthiness of evaluation, underscoring its significant potential for advancing future research.

Downloads

Next from NAACL 2025

Understanding Figurative Meaning through Explainable Visual Entailment

Similar lecture

Closing

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAACL 2025

Understanding Figurative Meaning through Explainable Visual Entailment

Similar lecture

Closing

Downloads