Austria

Search engines increasingly use large language models (LLMs) to generate direct answers, while AI chatbots retrieve updated information from the
Internet. As information curators for billions of users, LLMs must evaluate the accuracy and reliability of sources. This study audits nine LLMs
from OpenAI, Google, and Meta to assess their ability to evaluate the credibility and quality of the top 20 most popular Bangladeshi news outlets.
While LLMs rate most tested outlets, larger models more often refuse to rate sources due to insufficient information, while smaller models are
prone to hallucinations. When ratings are provided, LLMs show strong internal consistency with an average correlation coefficient (𝜌) of 0.72, but
their alignment with human expert evaluations is moderate, with an average 𝜌 of 0.45. We introduce a dataset of expert opinions (journalism and
media studies students) on the credibility and political bias of Bangladeshi news outlets to evaluate LLMs’ political bias and credibility assessments.
Our analysis reveals that LLMs in default configurations favor the Bangladesh Awami League-affiliated sources in credibility ratings. Assigning
partisan identities to LLMs further amplifies politically congruent biases in their assessments. These findings highlight the need to address political
bias and improve credibility evaluations as LLMs increasingly shape how news and political information are curated worldwide.

ACL 2025

Evaluating Credibility and Political Bias in LLMs for News Outlets in Bangladesh

bangladesh

news outlets

large language models (llms)

credibility

political bias

#### *Message from the General Chair, Aline Villavicencio*
I’m delighted and honoured to welcome you to the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), taking place in the beautiful city of Rabat, in Morocco, in March 24-29, 2026. EACL is the flagship European conference of the Association and EACL 2026 proudly continues our field’s tradition of excellence in scholarship, innovation, and inclusivity. I am deeply grateful to the many volunteers whose dedication, generosity, and tireless efforts have made this conference possible.
For the first time EACL is being hosted in the African continent. This is an important milestone for our community, and we are grateful to our Moroccan hosts for enabling this historic moment by bringing this edition of EACL to Rabat. We are also delighted that the Second Arabic NLP School is co-located with EACL. We hope attendees enjoy this wonderful opportunity to strengthen ties with the Computational Linguistics communities across the African continent. *[Read full message](https://drive.google.com/file/d/14NlmHvwM6fPJuMmOvVh7K0vtQbEyv3SZ/view?usp=sharing)* 

<html><button style="display: inline-flex; align-items: center; justify-content: center; white-space: nowrap; border-radius: 9999px; font-weight: bold; background: #7c3aed; color: white; font-family: 'Space Grotesk', sans-serif; height: 40px; font-size: 16px; padding: 0 20px; border: none; cursor: pointer" onclick="window.open('https://underline.io/events/522/reception','_blank')">Go to Workshops and Tutorials Program</button></html>

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to EACL 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://2026.eacl.org/registration/) first.

**Online Registration Form**: https://acl.swoogo.com/eacl2026

Registration Required

EACL 2026

Welcome to the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL). EACL 2026 will be held in Rabat, Morocco, from March 24–29, 2026. 

workshop paper

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Chinese idioms (成语, Chengyu) are concise four-character expressions steeped in history and culture, whose literal translations often fail to capture their full meaning. This complexity makes them challenging for language models to interpret and use correctly. Existing benchmarks focus on narrow tasks—multiple-choice cloze tests, isolated translation, or simple paraphrasing. We introduce CHENGYU-BENCH, a comprehensive benchmark featuring three tasks: (1) Evaluative Connotation, classifying idioms as positive or negative; (2) Appropriateness, detecting incorrect idiom usage in context; and (3) Open Cloze, filling blanks in longer passages without options. CHENGYU-BENCH comprises 2,937 human-verified examples covering 1,765 common idioms sourced from diverse corpora. We evaluate leading LLMs and find they achieve over 95\% accuracy on Evaluative Connotation, but only ~85% on Appropriateness and ~40% top-1 accuracy in Open Cloze. Error analysis reveals that most mistakes arise from fundamental misunderstandings of idiom meanings. CHENGYU-BENCH demonstrates that while LLMs can reliably gauge idiom sentiment, they still struggle to grasp the cultural and contextual nuances essential for proper usage. The benchmark and code will be released upon paper acceptance.

CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use

While large language models (LLMs) have demonstrated impressive capabilities across diverse tasks, their ability to generate valid graph structures remains underexplored. We evaluate fifteen state-of-the-art LLMs on five specialized graph generation tasks spanning delivery networks, social networks, quantum circuits, gene-disease networks, and transportation systems. We also test the LLMs using 3 different prompt types: direct, iterative feedback, and program-augmented. Models supported with explicit reasoning modules (o3-mini-high, o1, Claude 3.7 Sonnet, DeepSeek-R1) solve more than twice as many tasks as their general-purpose peers, independent of parameter count. Error forensics reveals two recurring failure modes: Llama-family models often violate basic structural constraints, whereas Claude models respect topology but mismanage higher-order logical rules. Allowing models to iteratively refine their answers yields uneven gains, underscoring fundamental differences in error-correction capacity. This work demonstrates that graph competence stems from specialized architectural design rather than scale, establishing a framework for developing truly graph-savvy language models. Results and verification scripts available at https://github.com/anonymized-for-the-blind-review.

Are LLMs Truly Graph-Savvy? A Comprehensive Evaluation of Graph Generation

This paper quantifies the "embodiment gap" between disembodied language models and embodied agricultural knowledge communication through mixed-methods analysis with 78 farmers. Our key contributions include: (1) the Embodied Knowledge Representation Framework (EKRF), a novel computational architecture with specialized lexical mapping that incorporates embodied linguistic patterns from five identified domains of agricultural expertise; (2) the Embodied Prompt Engineering Protocol (EPEP), which reduced the embodiment gap by 47.3\% through systematic linguistic scaffolding techniques; and (3) the Embodied Knowledge Representation Index (EKRI), a new metric for evaluating embodied knowledge representation in language models. Implementation results show substantial improvements across agricultural domains, with particularly strong gains in tool usage discourse (58.7\%) and soil assessment terminology (67\% reduction in embodiment gap). This research advances both theoretical understanding of embodied cognition in AI and practical methodologies to enhance LLM performance in domains requiring embodied expertise.

Bridging the Embodiment Gap in Agricultural Knowledge Representation for Language Models

Effective communication of health information requires adapting complexity to match the target audience's literacy level. However, manually simplifying medical content is both time-consuming and difficult to scale. To address this challenge, we developed a new framework for automatically generating health answers at multiple complexity levels.

We began by collecting 166 linguistic features to quantify text complexity, including traditional readability metrics (e.g., Flesch-Kincaid, SMOG), medical terminology usage (e.g., UMLS coverage, medical entity recognition), syntactic complexity, semantic coherence, and LLM-based measures (e.g., masked language modeling, LLM-as-a-judge). Applying these features to a custom dataset of parallel health texts and external medical benchmarks, we used feature selection to identify 13 key metrics that reliably distinguish between simple and complex text pairs. From these, we derived a complexity scoring formula by combining the metrics with weights learned from a logistic regression model.

Using this formula, we created a large multi-level dataset of health question-answer pairs, ranging from elementary patient-friendly explanations to advanced technical summaries.
The initial QA pairs came from established datasets including LiveQA, MedicationQA, and MEDIQA-AnS. We then used LLaMA-based language models with carefully engineered prompts to transform the original answers into five different versions ordered by complexity.
Finally, we fine-tuned a large language model on this dataset, incorporating special tokens to control the complexity of the generated text. The resulting model can generate health answers at fine-grained complexity levels, allowing users to select the desired level of detail and technicality.

A Framework for Fine-Grained Complexity Control in Health Answer Generation

This paper represents an implementation of an approach rather similar to that of (Zhu et al., 2024), but for the Russian-language data. We introduce a RusConText Benchmark for evaluating short context understanding in Russian, comprising four distinct yet interrelated tasks: ellipsis resolution, coreference resolution, and idiom interpretation, and discourse understanding. Each task targets a specific aspect of linguistic processing, challenging large language model to recover omitted information, resolve referential dependencies, interpret idioms and discourse. RusConText Benchmark serves as an additional resource beyond standard benchmarks, designed to assess model performance from a specific perspective. In addition, we present the results of scoring 4 models on our benchmark.

RusConText Benchmark: A Russian Language Evaluation Benchmark for Understanding Context

Neural topic modeling is a scalable automated technique for text data mining. In various downstream tasks of topic modeling, it is prefered that the discovered topics well align with labels. However, due to the lack of guidance from labels, unsupervised neural topic models are less powerful in this situation. Existing supervised neural topic models often adopt a label-free prior to generate the latent document-topic distributions and use them to predict the labels and thus achieve label-topic alignment indirectly. Such a mechanism faces the following issues: 1) The label-free prior leads to topics blending the latent patterns of multiple labels; 2) One is unable to intuitively identify the explicit relationships between labels and the discovered topics. To tackle these problems, we develop a novel supervised neural topic model which utilizes a chain-structured graphical model that emphasizes direct constraints of labels on topics. Soft indicators are introduced to explicitly construct the label-topic relationships. To obtain well-organized label-topic relationships, we formalize an entropy-regularized optimal transport problem on the embedding space and model them as the transport plan. Moreover, our proposed method can be seamlessly integrated with most existing unsupervised neural topic models in a flexible manner. Experiment results on multiple datasets demonstrate that our model can greatly enhance the alignment between labels and topics while maintaining good topic quality.

Supervised Neural Topic Modeling with Label Alignment

Amidst the rapid advancement of artificial intelligence, research on large vision-language models (LVLMs) has emerged as a pivotal area. However, understanding their internal mechanisms remains challenging due to the limitations of existing interpretability methods, especially regarding faithfulness and plausibility. To address this, we first construct a human response interpretability dataset that evaluates the plausibility of model explanations by comparing the attention regions between the model and humans when answering the same questions. We then propose a patchwise cooperative game-based interpretability method for LVLMs, which employs Shapley values to quantify the impact of individual image patches on generation likelihood and enhances computational efficiency through a single input approximation approach. Experimental results demonstrate our method's faithfulness, plausibility, and robustness. Our method provides researchers with deeper insights into model behavior, allowing for an examination of the specific image regions each layer relies on during response generation, ultimately enhancing model reliability. Our codes are available at https://github.com/ZY123-GOOD/Patchwise_Cooperative.

Patchwise Cooperative Game-based Interpretability Method for Large Vision-Language Models

We propose the Data Contamination Quiz (DCQ), a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions, devising a quiz format wherein three perturbed versions of each instance, subsampled from a specific dataset partition, are created. These changes only include word-level perturbations. The generated perturbations, along with the original dataset instance, form the options in the DCQ, with an extra option accommodating the selection of none of the provided options. Given that the only distinguishing signal among the options is the exact wording with respect to the original dataset instance, an LLM, when tasked with identifying the original dataset instance, gravitates towards selecting the original one if it has been exposed to it. While accounting for positional biases in LLMs, the quiz performance reveals the contamination level for the tested model with the dataset partition to which the quiz pertains. Applied to various datasets and LLMs, under controlled and uncontrolled contamination, our findings—while fully lacking access to training data and model parameters—suggest that DCQ achieves state-of-the-art results and uncovers greater contamination/memorization levels compared to existing methods. Also, it proficiently bypasses more safety filters, especially those set to avoid generating copyrighted content.

Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models

Adversarial examples pose a significant challenge to deep neural networks across both image and text domains, with the intent to degrade model performance through carefully altered inputs. Adversarial texts, however, are distinct from adversarial images due to their requirement for semantic similarity and the discrete nature of the textual contents. This study delves into the concept of human suspiciousness, a quality distinct from the traditional focus on imperceptibility found in image-based adversarial examples, where adversarial changes are often desired to be indistinguishable to the human eye even when placed side by side with originals. Although this is generally not possible with text, textual adversarial content must still often remain undetected or non-suspicious to human readers. Even when the text’s purpose is to deceive NLP systems or bypass filters, the text is often expected to be natural to read. In this research, we expand the study of human suspiciousness by analyzing how individuals perceive adversarial texts. We gather and publish a novel dataset of Likert-scale human evaluations on the suspiciousness of adversarial sentences, crafted by four widely used adversarial attack methods and assess their correlation with the human ability to detect machine-generated alterations. Additionally, we develop a regression-based model to predict levels of suspiciousness and establish a baseline for future research in reducing the suspiciousness in adversarial text generation. We also demonstrate how the regressor-generated suspicious scores can be incorporated into adversarial generation methods to produce texts that are less likely to be perceived as computer-generated.

Graded Suspiciousness of Adversarial Texts to Humans

We present a corpus of 8400 Dutch sentence pairs, intended primarily for the grammatical evaluation of language models. Each pair consists of a grammatical sentence and a minimally different ungrammatical sentence. The corpus covers 84 paradigms, classified into 22 syntactic phenomena. Ten sentence pairs of each paradigm were created by hand, while the remaining ninety were generated semi-automatically and manually validated afterwards. Nine of the ten hand-crafted sentences of each paradigm are rated for acceptability by at least 30 participants each, and for the same 9 sentences reading times are recorded per word, through self-paced reading.Â  Here, we report on the construction of the dataset, the measured acceptability ratings and reading times, as well as the extent to which a variety of language models can be used to predict both the ground-truth grammaticality and human acceptability ratings.

Downloads

Next from ACL 2025

CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use

Similar lecture

News Credibility Assessment by LLMs and Humans: Implications for Political Bias

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2025

CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use

Similar lecture

News Credibility Assessment by LLMs and Humans: Implications for Political Bias

Downloads