Malta

The automatic classification of stage directions is a little explored topic in computational drama analysis, in spite of their relevance for plays&#39; structural and stylistic analysis. With a view to start assessing good practices for the automatic annotation of this textual element, we developed a 13-class stage direction typology, based on annotations in the FreDraCor corpus (French-language plays), but abstracting away from their huge variability while still providing classes useful for literary research. We fine-tuned transformers-based models to classify against the typology, gradually decreasing the corpus size used for fine tuning, to compare model efficiency with reduced training data. A result comparison speaks in favour of distilled monolingual models for this task, and, unlike earlier research on German, shows no negative effects of model case-sensitivity. The results have practical relevance for computational literary studies, as comparing classification results with complementary stage direction typologies, limiting the amount of manual annotation needed to apply them, would be helpful towards a systematic study of this important textual element.

EACL 2024

Stage Direction Classification in French Theater: Transfer Learning Experiments

computational drama analysis

stage directions

computational literary studies

machine learning

transfer learning

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

The automatic classification of stage directions is a little explored topic in computational drama analysis, in spite of their relevance for plays' structural and stylistic analysis. With a view to start assessing good practices for the automatic annotation of this textual element, we developed a 13-class stage direction typology, based on annotations in the FreDraCor corpus (French-language plays), but abstracting away from their huge variability while still providing classes useful for literary research. We fine-tuned transformers-based models to classify against the typology, gradually decreasing the corpus size used for fine tuning, to compare model efficiency with reduced training data. A result comparison speaks in favour of distilled monolingual models for this task, and, unlike earlier research on German, shows no negative effects of model case-sensitivity. The results have practical relevance for computational literary studies, as comparing classification results with complementary stage direction typologies, limiting the amount of manual annotation needed to apply them, would be helpful towards a systematic study of this important textual element.

workshop paper

Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as conversational
agents. These models, such as GPT-4, can not only master language but also solve
complex tasks in areas like mathematics, coding, medicine, and law. However, there
is still a lack of comprehensive understanding regarding their full potential,
primarily due to the black-box nature of many models and lack of holistic evaluation.
To address these challenges, we present InstructEval, a more comprehensive evaluation
suite designed specifically for instruction-tuned large language models. Unlike
previous works, our evaluation involves a rigorous assessment of models based
on problem-solving, writing ability, and alignment to human values. We take a
holistic approach to analyze various factors affecting model performance, including
the pretraining foundation, instruction-tuning data, and training methods. Our
findings reveal that the quality of instruction data is a crucial factor in scaling
model performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment.

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Most adults can complete a sequence of steps to achieve a certain goal,
such as making a sandwich or repairing a bicycle tire. In completing these goal-oriented
tasks, or simply tasks in this paper, one must use sequential reasoning to understand
the relationship between the sequence of steps and the goal. LLMs have shown
impressive capabilities across various natural language understanding tasks. However,
prior work has mainly

focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); how well
LLMs can perform on more complex reasoning tasks like sequential reasoning is
not clear. In this paper, we address this gap and conduct a comprehensive evaluation
of how well LLMs are able to conduct this reasoning for tasks and how they scale
w.r.t multiple dimensions(e.g. adaptive prompting strategies, number of in-context
examples, varying complexity of the sequential task). Our findings reveal that
while Chain of Thought (CoT) prompting can significantly enhance LLMs'' sequential
reasoning in certain scenarios, it can also be detrimental in others, whereas
Tree of Thoughts (ToT) reasoning is less effective for this type of task. Additionally,
we discover that an increase in model size or in-context examples does not consistently
lead to improved performance.

Can Large Language Models Reason About Goal-Oriented Tasks?

No two authors write alike. Personal flourishes invoked in written narratives,
from lexicon to rhetorical devices, imply a particular author—what literary theorists
label the implied or virtual author; distinct from the real author or narrator
of a text. Early large language models trained on unfiltered training sets drawn
from a variety of discordant sources yielded incoherent personalities, problematic
for conversational tasks but proving useful for sampling literature from multiple
perspectives. Successes in alignment research in recent years have allowed researchers
to impose subjectively consistent personae on language models via instruction
tuning and reinforcement learning from human feedback (RLHF), but whether aligned
models retain the ability to model an arbitrary virtual author has received little
scrutiny. By studying 4,374 stories sampled from three OpenAI language models,
we show successive versions of GPT-3 suffer from increasing degrees of “mode collapse”
whereby overfitting the model during alignment constrains it from generalizing
over authorship: models suffering from mode collapse become unable to assume a
multiplicity of perspectives. Our method and results are significant for researchers
seeking to employ language models in sociological simulations.

Detecting Mode Collapse in Language Models via Narration

The extraction of structured data from websites is critical for numerous Artificial Intelligence applications, but modern web design increasingly stores information visually in images rather than in text. This shift calls into question the optimal technique, as language-only models fail without textual cues while new multimodal models like GPT-4 promise image understanding abilities. We conduct the first rigorous comparison between text-based and vision-based models for extracting event metadata harvested from comic convention websites. Surprisingly, our results between GPT-4 Vision and GPT-4 Text uncover a significant accuracy advantage for vision-based methods in an applies-to-apples setting, indicating that vision models may be outpacing language-alone techniques in the task of information extraction from websites. We release our dataset and provide a qualitative analysis to guide further research in multi-modal models for web information extraction.

The Future of Web Data Mining: Insights from Multimodal and Code-based Extraction Methods

Freedom of Information Act (FOIA) legislation grants citizens the right to request information from various levels of the government, and aims to promote the transparency of governmental agencies. However, the processing of these requests is often met with delays, due to the inherent complexity of gathering the required documents. To obtain accurate estimates of the processing times of requests, and to identify bottlenecks in the process, this research proposes a pipeline to automatically extract these timelines from decision letters of Dutch FOIA requests. These decision letters are responses to requests, and contain an overview of the process, including when the request was received, and possible communication between the requester and the relevant agency. The proposed pipeline can extract dates with an accuracy of .94, extract event phrases with a mean ROUGE- L F1 score of .80 and can classify events with a macro F1 score of .79. Out of the 50 decision letters used for testing (each letter containing one timeline), the model correctly classified 10 of the timelines completely correct, with an average of 3.1 mistakes per decision letter.

Timeline Extraction from Decision Letters Using ChatGPT

We describe a new weakly supervised method for sentence-level event detection, based exclusively on linear prototype patterns like "people got sick" or "a roadside bomb killed people". We propose a new BERT based algorithm for approximate pattern matching to identify event phrases, semantically similar to these prototypes. To the best of our knowledge, a similar approach has not been used in the context of event detection. We experimented with two event corpora in the area of disease outbreaks and terrorism and we achieved promising results in sentence level event identification: 0.78 F1 score for new disease cases detection and 0.68 F1 in detecting terrorist attacks. Results were in line with some state-of-the-art systems.

Leveraging Approximate Pattern Matching with BERT for Event Detection

There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text. The survey presented in this paper aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest ‚Äì both from the NLP and political science communities ‚Äì offering a unified view of the work done across different disciplines.

Socio-political Events of Conflict and Unrest: A Survey of Available Datasets

ChatGPT, developed by OpenAI, has made a significant impact on the world, mainly on how people interact with technology. In this study, we evaluate ChatGPT's ability to detect hate speech in Turkish tweets and measure its strength using zero- and few-shot paradigms and compare the results to the supervised fine-tuning BERT model. On evaluations with the SIU2023-NST dataset, ChatGPT achieved 65.81 accuracy in detecting hate speech for the few-shot setting, while BERT with supervised fine-tuning achieved 82.22 accuracy. This results supports previous findings that show that, despite its much smaller size, BERT is more suitable for natural language classifications tasks such as hate speech detection.

Evaluating ChatGPT's Ability to Detect Hate Speech in Turkish Tweets

This paper introduces a zero-shot hate detection experiment using a multimodal large model. Although the implemented model comprises an unsupervised method, results demonstrate that its performance is comparable to previous supervised methods. Furthemore, this study proposed experiments with various prompts and demonstrated that simpler prompts, as opposed to the commonly used detailed prompts in large language models, led to better performance for multimodal hate speech event detection tasks. While supervised methods offer high performance, they require significant computational resources for training, and the approach proposed here can mitigate this issue. The code is publicly available at https://github.com/yamagishi0824/zeroshot-hate-detect.

YYama@Multimodal Hate Speech Event Detection 2024: Simpler Prompts, Better Results - Enhancing Zero-shot Detection with a Large Multimodal Model

This paper describes the system that participated in the Climate Activism Stance and Hate Event Detection shared task organized at The 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024). The system tackles the important task of hate speech detection by combining large language model predictions with manually designed features, while trying to explain where the LLM approach fails to predict the correct results.

Downloads

Next from EACL 2024

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Similar lecture

Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EACL 2024

InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Similar lecture

Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama

Downloads