Malta

Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as conversational
agents. These models, such as GPT-4, can not only master language but also solve
complex tasks in areas like mathematics, coding, medicine, and law. However, there
is still a lack of comprehensive understanding regarding their full potential,
primarily due to the black-box nature of many models and lack of holistic evaluation.
To address these challenges, we present InstructEval, a more comprehensive evaluation
suite designed specifically for instruction-tuned large language models. Unlike
previous works, our evaluation involves a rigorous assessment of models based
on problem-solving, writing ability, and alignment to human values. We take a
holistic approach to analyze various factors affecting model performance, including
the pretraining foundation, instruction-tuning data, and training methods. Our
findings reveal that the quality of instruction data is a crucial factor in scaling
model performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment.

EACL 2024

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

evaluation; instruction tuning; large language models

## Welcome to 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics!
This year, the conference is in Mexico City. NAACL was actually already planned for Mexico City in 2021, but due to the pandemic the entire conference was moved online. This year, finally, we get to go! So it is my sincere pleasure to welcome you to Mexico City, whether in person or virtually. Having the conference in Mexico City is a good opportunity to emphasize that NAACL is our flagship conference for ACL members not only in North America but also in Central and South America, even though NAACL has been bearing “North” in its name. At this year’s conference, we have a theme to match, with a theme track on the Languages of Latin America to showcase the linguistic diversity of the region.
 
The opportunity to present at NAACL should not depend on a researcher’s travel budget, or their family status. This is why it is so important to make virtual participation at NAACL as good an experience as possible – but we want to also provide a good experience for in-person participants. As a community, we are still working out the best way to do that. This year at NAACL, we are trying out a big virtual poster session ahead of the conference, with the hope that this will make make for a lively and interactive experience. At the same time, we are reducing virtual oral presentations, which seem to be particularly tricky to make to work well. A big thanks to the NAACL program chairs and to Luciana Benotti for all their ideas and work to improve the virtual experience. And participants, virtual as well as in-person: Please let us know what worked for you and what didn’t, so we can continue to improve hybrid conferences. 
 
I have been lucky to work with many amazing people. Without their insight, dedication and patience, and without the many hours of work they put in, NAACL would not have been possible. A huge thank you to the program chairs Helena Gomez, Kevin Duh, and Steve Bethard – you are the best! 
 
Finally, I would like to thank all authors, invited speakers and panelists, area chairs and reviewers, the volunteers organizing and chairing sessions, and all attendees, in-person and virtual. Thank you for helping us make NAACL 2024 come to life. 
 
Welcome and hope you all enjoy the conference! 
Katrin Erk 
The University of Texas at Austin 
NAACL-2024 General Chair 
*You can read the full Welcome message in the [Conference Handbook (downloadable)](https://drive.google.com/file/d/1H1NvW0VASQjkSCYgw3mr-3l4yYDSxz6B/view?usp=sharing)*

To access the event page you need to register [**here.**](http://acl.swoogo.com/naacl2024) 
Your access to the event page is limited based on your registration type. If you registered for workshops only, you will gain access to full workshops content on the day of the workshops program.

Please register!

NAACL 2024

2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

workshop paper

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs'' sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

Can Large Language Models Reason About Goal-Oriented Tasks?

No two authors write alike. Personal flourishes invoked in written narratives,
from lexicon to rhetorical devices, imply a particular author—what literary theorists
label the implied or virtual author; distinct from the real author or narrator
of a text. Early large language models trained on unfiltered training sets drawn
from a variety of discordant sources yielded incoherent personalities, problematic
for conversational tasks but proving useful for sampling literature from multiple
perspectives. Successes in alignment research in recent years have allowed researchers
to impose subjectively consistent personae on language models via instruction
tuning and reinforcement learning from human feedback (RLHF), but whether aligned
models retain the ability to model an arbitrary virtual author has received little
scrutiny. By studying 4,374 stories sampled from three OpenAI language models,
we show successive versions of GPT-3 suffer from increasing degrees of “mode collapse”
whereby overfitting the model during alignment constrains it from generalizing
over authorship: models suffering from mode collapse become unable to assume a
multiplicity of perspectives. Our method and results are significant for researchers
seeking to employ language models in sociological simulations.

Detecting Mode Collapse in Language Models via Narration

The extraction of structured data from websites is critical for numerous Artificial Intelligence applications, but modern web design increasingly stores information visually in images rather than in text. This shift calls into question the optimal technique, as language-only models fail without textual cues while new multimodal models like GPT-4 promise image understanding abilities. We conduct the first rigorous comparison between text-based and vision-based models for extracting event metadata harvested from comic convention websites. Surprisingly, our results between GPT-4 Vision and GPT-4 Text uncover a significant accuracy advantage for vision-based methods in an applies-to-apples setting, indicating that vision models may be outpacing language-alone techniques in the task of information extraction from websites. We release our dataset and provide a qualitative analysis to guide further research in multi-modal models for web information extraction.

The Future of Web Data Mining: Insights from Multimodal and Code-based Extraction Methods

Freedom of Information Act (FOIA) legislation grants citizens the right to request information from various levels of the government, and aims to promote the transparency of governmental agencies. However, the processing of these requests is often met with delays, due to the inherent complexity of gathering the required documents. To obtain accurate estimates of the processing times of requests, and to identify bottlenecks in the process, this research proposes a pipeline to automatically extract these timelines from decision letters of Dutch FOIA requests. These decision letters are responses to requests, and contain an overview of the process, including when the request was received, and possible communication between the requester and the relevant agency. The proposed pipeline can extract dates with an accuracy of .94, extract event phrases with a mean ROUGE- L F1 score of .80 and can classify events with a macro F1 score of .79. Out of the 50 decision letters used for testing (each letter containing one timeline), the model correctly classified 10 of the timelines completely correct, with an average of 3.1 mistakes per decision letter.

Timeline Extraction from Decision Letters Using ChatGPT

We describe a new weakly supervised method for sentence-level event detection, based exclusively on linear prototype patterns like "people got sick" or "a roadside bomb killed people". We propose a new BERT based algorithm for approximate pattern matching to identify event phrases, semantically similar to these prototypes. To the best of our knowledge, a similar approach has not been used in the context of event detection. We experimented with two event corpora in the area of disease outbreaks and terrorism and we achieved promising results in sentence level event identification: 0.78 F1 score for new disease cases detection and 0.68 F1 in detecting terrorist attacks. Results were in line with some state-of-the-art systems.

Leveraging Approximate Pattern Matching with BERT for Event Detection

There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text. The survey presented in this paper aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest ‚Äì both from the NLP and political science communities ‚Äì offering a unified view of the work done across different disciplines.

Socio-political Events of Conflict and Unrest: A Survey of Available Datasets

ChatGPT, developed by OpenAI, has made a significant impact on the world, mainly on how people interact with technology. In this study, we evaluate ChatGPT's ability to detect hate speech in Turkish tweets and measure its strength using zero- and few-shot paradigms and compare the results to the supervised fine-tuning BERT model. On evaluations with the SIU2023-NST dataset, ChatGPT achieved 65.81 accuracy in detecting hate speech for the few-shot setting, while BERT with supervised fine-tuning achieved 82.22 accuracy. This results supports previous findings that show that, despite its much smaller size, BERT is more suitable for natural language classifications tasks such as hate speech detection.

Evaluating ChatGPT's Ability to Detect Hate Speech in Turkish Tweets

This paper introduces a zero-shot hate detection experiment using a multimodal large model. Although the implemented model comprises an unsupervised method, results demonstrate that its performance is comparable to previous supervised methods. Furthemore, this study proposed experiments with various prompts and demonstrated that simpler prompts, as opposed to the commonly used detailed prompts in large language models, led to better performance for multimodal hate speech event detection tasks. While supervised methods offer high performance, they require significant computational resources for training, and the approach proposed here can mitigate this issue. The code is publicly available at https://github.com/yamagishi0824/zeroshot-hate-detect.

YYama@Multimodal Hate Speech Event Detection 2024: Simpler Prompts, Better Results - Enhancing Zero-shot Detection with a Large Multimodal Model

This paper describes the system that participated in the Climate Activism Stance and Hate Event Detection shared task organized at The 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024). The system tackles the important task of hate speech detection by combining large language model predictions with manually designed features, while trying to explain where the LLM approach fails to predict the correct results.

RACAI at ClimateActivism 2024: Improving Detection of Hate Speech by Extending LLM Predictions with Handcrafted Features

In the context of the proliferation of multimodal hate speech related to the Russia-Ukraine conflict, we introduce a unified multimodal fusion system for detecting hate speech and its targets in text-embedded images. Our approach leverages the Twitter-based RoBERTa and Swin Transformer V2 models to encode textual and visual modalities, and employs the Multilayer Perceptron (MLP) fusion mechanism for classification. Our system achieved macro F1 scores of 87.27 for hate speech detection and 80.05 for hate speech target detection in the Multimodal Hate Speech Event Detection Challenge 2024, securing the 1st rank in both subtasks. We open-source the trained models at https://huggingface.co/Yestin-Wang

Downloads

Next from EACL 2024

Can Large Language Models Reason About Goal-Oriented Tasks?

Similar lecture

InstructEval: Systematic Evaluation of Instruction Selection Methods

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EACL 2024

Can Large Language Models Reason About Goal-Oriented Tasks?

Similar lecture

InstructEval: Systematic Evaluation of Instruction Selection Methods

Downloads