United States

Large language models (LLMs) can help elucidate hate, violence, and other toxicity. However, labeling harmful events is challenging due to the subjectivity of labels such as toxicity&#39;&#39; and hate.&#39;&#39; Motivated by the rise of antisemitism, this paper studies the capability of LLMs to discover reports of antisemitic events. We pilot the task of hateful event classification on the AMCHA Corpus---a continuously updated dataset with expert-labeled instances of fine-grained types of antisemitism---and show that incorporating domain knowledge from fine-grained taxonomies is needed to make LLMs more effective. Our experiments find that providing precise definitions from a taxonomy can steer GPT-4 and Llama-3 to somewhat improve on tagging antisemitic event descriptions, with GPT-4 achieving up to a 14% increase in mean weighted F1. However, LLMs are still far from perfect at understanding antisemitic events, suggesting avenues for future work on LLM alignment and precise definition of antisemitism.

EMNLP 2024

On the Importance of Nuanced Taxonomies for LLM-Based Understanding of Harmful Events: A Case Study on Antisemitism

hateful event understanding

antisemitism

toxicity detection

large language models

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community." 

*Message from the General Chair, Dirk Hovy*

EMNLP 2025

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

Large language models (LLMs) can help elucidate hate, violence, and other toxicity. However, labeling harmful events is challenging due to the subjectivity of labels such as toxicity'' and hate.'' Motivated by the rise of antisemitism, this paper studies the capability of LLMs to discover reports of antisemitic events. We pilot the task of hateful event classification on the AMCHA Corpus---a continuously updated dataset with expert-labeled instances of fine-grained types of antisemitism---and show that incorporating domain knowledge from fine-grained taxonomies is needed to make LLMs more effective. Our experiments find that providing precise definitions from a taxonomy can steer GPT-4 and Llama-3 to somewhat improve on tagging antisemitic event descriptions, with GPT-4 achieving up to a 14% increase in mean weighted F1. However, LLMs are still far from perfect at understanding antisemitic events, suggesting avenues for future work on LLM alignment and precise definition of antisemitism.

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

Please register!

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

Social movements use social media to draw attention to their cause, disseminate information, coordinate offline action, and build collective identity (Harlow, 2012; Jost et al., 2018). Social media further provides a space for activists, bystanders and opponents to collectively construct and contest narratives of their movement via collective action framing (Snow et al., 1988). Focusing on Twitter (X) messages from social movements surrounding three issues (guns, immigration, and LGBTQ rights), we create a codebook, annotated dataset, and computational models to detect diagnostic (problem identification and attribution), prognostic (proposed solutions and tactics) and motivational (calls to action) framing strategies. We conduct an in-depth unsupervised linguistic analysis of each strategy and uncover cross-movement similarities in associations between framing and linguistic features such as personal pronouns. We then analyze frame variation across a broad set of sociocultural and interactional contexts.

Framing Social Movements on Social Media: Unpacking Diagnostic, Prognostic, and Motivational Strategies

Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.

Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models

News headline generation has seen significant advancements in resource-rich languages like the English language, leveraging sophisticated natural language processing (NLP) techniques. However, similar progress has not been observed for low-resource Indian languages, particularly Telugu. We focus on implementing news headline generation given the news article using an abstractive summarization approach, which enables the generation of contextually rich headlines by interpreting and rephrasing content using deep learning techniques. We create a dataset of articles and headlines scraped from the Telugu news website `Sakshi'. We use pre-trained language models such as mBART50, MT5, and IndicBART for fine-tuning using our dataset. Our findings show the effectiveness of fine-tuning pre-trained models for news headline-generation tasks in Telugu.

News Headline Generation in Telugu

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have con ducted a survey on the use of automatic metrics, focusing particularly on natural language gen eration (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, in cluding inappropriate metric usage, lack of im plementation details and missing correlations with human judgements. We conclude with rec ommendations that we believe authors should follow to enable more rigour within the field.

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedical knowledge base, ignoring the inter-concept interactions and a concept's local neighborhood in a knowledge base graph. In this paper, we introduce Biomedical Entity Representation with a Graph-Augmented Multi-Objective Transformer (BERGAMOT), which adopts the power of pre-trained language models (LMs) and graph neural networks to capture both inter-concept and intra-concept interactions from the multilingual UMLS graph. We apply contrastive loss on textual and graph representations to make them less sensitive to surface forms and enable intermodal knowledge exchange between two uni-modal encoders. BERGAMOT achieves state-of-the-art results in zero-shot entity linking without task-specific supervision on three monolingual datasets and Mantra multilingual benchmark.

Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer

Automated Emotion Recognition in language is a challenging task that has attracted considerable attention especially in recent years. We present a summary of our findings and observations based on a thorough analysis of recent resources and related practices.

Emotion-Annotated Data in NLP: Perspectives on Recent Resources and Practices

The textual content that is fed to Large Language Models (LLMs) can sometimes be noisy con- taining misspelled words. We investigate the effect of such mistakes on the performance of 9 language models, with parameters ranging from 0.2B to 13B, in 3 different NLP tasks, namely Natural Language Inference (NLI), Name Entity Recognition (NER), and Intent Classification (IC). We perform our experiments on 6 different languages (English, German, French, Spanish, Hindi, and Turkish) and build a dictionary of real-world noise for them using Wikipedia edit history. We show that the performance gap of the studied models on the clean and noisy test data averaged across all the datasets and languages ranges from 2.3 to 4.3 absolute percentage points. In addition, mT5 models, in general, show more robustness compared to BLOOM, Falcon, and BERT-like models. In particular, mT5 (13B), was the most robust on average overall, across the 3 tasks, and in 4 out of the 6 languages.

Exploring Robustness of Multilingual LLMs on Real-World Noisy Data

Large language models (LLMs) have significantly improved their reasoning and decision-making capabilities, as seen in methods like ReAct. However, despite its effectiveness in tackling complex tasks, ReAct faces two main challenges: losing focus on the original question and becoming stuck in action loops. To address these issues, we introduce Focused ReAct, an enhanced version of the ReAct paradigm that incorporates reiteration and early stop mechanisms. These improvements help the model stay focused on the original query and avoid repetitive behaviors. Experimental results show accuracy gains of 18% to 530% and a runtime reduction of up to 34% compared to the original ReAct method.

Focused ReAct: Improving ReAct through Reiterate and Early Stop

We investigate the knowledge of object af- fordances in pre-trained language models (LMs) and pre-trained Vision-Language mod- els (VLMs). A growing body of literature shows that PTLMs fail inconsistently and non- intuitively, demonstrating a lack of reasoning and grounding. To take a first step toward quan- tifying the effect of grounding (or lack thereof), we curate a novel and comprehensive dataset of object affordances – TEXT2AFFORD, charac- terized by 15 affordance classes. Unlike affor- dance datasets collected in vision and language domains, we annotate in-the-wild sentences with objects and affordances. Experimental results reveal that PTLMs exhibit limited rea- soning abilities when it comes to uncommon object affordances. We also observe that pre- trained VLMs do not necessarily capture ob- ject affordances effectively. Through few-shot fine-tuning, we demonstrate improvement in affordance knowledge in PTLMs and VLMs. Our research contributes a novel dataset for lan- guage grounding tasks, and prese

Text2Afford: Probing Object Affordance Prediction abilities of Language Models solely from Text

To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zeroshot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.

Next from EMNLP 2024

Framing Social Movements on Social Media: Unpacking Diagnostic, Prognostic, and Motivational Strategies

Similar lecture

Evaluating Large Language Models for Detecting Antisemitism