Singapore

This paper examines the main challenges in Amharic hate speech classification tasks. The potential sources of errors in Amharic hate speech classification are explored in detail. Out of a total of more than 18 million tweets in our Twitter corpus, we annotate over 15k tweets with two independent Amharic native-speaker annotators. A Cohen&#39;s kappa score of 48% for inter-annotator agreement is achieved. An adjudicator is employed to decide on disputed annotations and determine the final gold labels. Logistic regression, Linear SVM, and the two fine-tuned contextual embedding transformer models: AmFLAIR and AmRoBERTa are employed. Among all the models, AmFLAIR achieves the best performance result with an F1-score of 72%.

EMNLP 2023

Error Analysis in Amharic Hate Speech Detection Task

error analysis

hate speech

annotation

This paper examines the main challenges in Amharic hate speech classification tasks. The potential sources of errors in Amharic hate speech classification are explored in detail. Out of a total of more than 18 million tweets in our Twitter corpus, we annotate over 15k tweets with two independent Amharic native-speaker annotators. A Cohen's kappa score of 48% for inter-annotator agreement is achieved. An adjudicator is employed to decide on disputed annotations and determine the final gold labels. Logistic regression, Linear SVM, and the two fine-tuned contextual embedding transformer models: AmFLAIR and AmRoBERTa are employed. Among all the models, AmFLAIR achieves the best performance result with an F1-score of 72%.

workshop paper

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

YORC: Yoruba Reading Comprehension dataset

Due to significant increases in internet penetration and the development of smartphone technology during the preceding couple of decades, many people have started using social media as a communication platform. Social media has grown to be one of the most significant components, with several benefits. However, technology also poses a number of threats, challenges, and barriers, such as hate speech, disinformation, and fake news. all most all social media platforms can be accused of not doing enough to thwart hate speech on their platform. Amharic and Afaan Oromo language speakers frequently mix the two languages when conversing and posting on social media. 
In this work, a bilingual hate speech detection for Amharic and Afaan Oromo languages were conducted using four different deep learning classifiers (CNN, BiLSTM, CNN-BiLSTM, and BiGRU) and three feature extraction (Keras word embedding, word2vec, and FastText) techniques. According to the experiment result, BiLSTM with FastText feature extraction is an outperforming the other algorithm by achieving a 78.05 % percent of accuracy for Bilingual Amharic Afaan Oromo hate speech detection. Furthermore, we are working towards multilingual hate speech detection for other under-represented Ethiopian languages.

Bilingual Hate Speech Detection on Social Media Amharic and Afaan Oromo

Large Language Models (LLMs) have taken center stage in recent years in the field of NLP. But while they have shown significant performance gains in many NLP tasks, their performance is concentrated in Anglo-Centric languages. Although previous studies have highlighted shortcomings of LLMs---including perpetuating harms and biases---we know little about their failure cases in low-resourced languages. We investigate how ChatGPT responds in Amharic and Afan Oromo, two low-resourced languages spoken in Ethiopia. We taxonomize the failure cases we observed and provide examples to illustrate the shortcomings. Our results show text generated in these languages has 6 failure modes and additionally exhibits political utterances that are not related to the prompts. We hope our work will contribute to the broader understanding of LLMs for low-resourced languages.

Evaluating Large Language Models in Low-Resource Settings

Multilingual language models (MLMs) have democratized access to information and artificial intelligence. However, low-resource languages remain underrepresented. This study introduces code-switching as a test to assess the low-resource multilingual capabilities of MLMs using standard Basque-English code-switching as a benchmark. The models were evaluated focusing on output naturalness, code-switching aptitude, and hallucination frequency. Employing codeswitching as an evaluative measure revealed significant differences in naturalness and the ability to produce grammatical codeswitched outputs across models. The paper finds that models have accurate knowledge of language contact, as shown by the mixing of languages out of the prompt domain.

Do You Speak Basquenglish? Assessing Low-resource Multilingual Proficiency of Pretrained Language Models

Social media data has become a crucial re- source for understanding and detecting mental health challenges. However, there is a significant gap in our understanding of age-specific linguistic markers associated with classifying depression. This study bridges the gap by analyzing 25,241 text samples from 15,156 Reddit users with self-reported depression across two age groups: adolescents (13-20 year olds) and adults (21+). Through a quantitative exploratory analysis using LIWC, topic modeling, and data visualization, distinct patterns and topical differences emerged in the language of depression for adolescents and adults, including social concerns, temporal focuses, emotions, and cognition. These findings enhance our understanding of how depression is expressed on social media, bearing implications for accurate classification and tailored interventions across different age groups.

An Exploratory Analysis of Differential Linguistic Features of Depression in Adolescents and Adults via Social Media | VIDEO

Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation

The main goal of this work is to develop a
bi-directional machine translation between Amharic
and Kistanigna. To increase the content of the language
on the web, to address the issue of the endangered
Kistanigna language and to share the information
between the language, it is essential to develop machine
translation between Amharic and Kistanigna language.
In this study, the translation is implemented using
neural machine translation. We conduct the
experiments using LSTM, Bi-LSTM, LSTM +
attention, CNN + attention, and the Transformer
encoder-decoder model. For our experiments, we have
used 9,225 parallel sentences with morpheme-based
translation units. To segment our morpheme data, we
used the morfessor tool. We considered training time,
memory usage, and BLEU sco re when proposing an
optimal model. Finally, we suggested morpheme-based
bidirectional machine translation using a Transformer
with BLEU scores of 21.31 and 22.40 for AmharicKistanigna and Kistanigna-Amharic translation
respectively. The major weakness of the study is the
unavailability of enough datasets to conduct an
extensive experiment. As a result, there is a need to
prepare parallel corpora for conducting similar
research.

Morpheme-based Bi-directional Machine Translation using Deep Learning

Previous studies have established that pre-trained language models inherently manifest various biases captured from the training corpora. Though several debiasing strategies, such as fine-tuning the whole model with counterfactuals, prompt tuning, and representation projection, have been introduced, they often fall short of effectively altering the models' intrinsic parameters without retaining language modeling ability. To address this, we propose EditBias, changing models' biased properties by modifying partial parameters while preserving meaningless associations. Experiments show that EditBias achieves excellent performance on debiasing.

Debiasing Stereotyped Models via Model Editing

Large Language Models (LLMs) have showcased remarkable advancements in diverse language tasks. However, their performance in mathematical reasoning, particularly in solving Mathematical Word Problems (MWPs), remains a challenge. To address this, we propose a novel masking technique tailored to enhance LLMs' MWPs-solving abilities. Unlike random masking, our method employs a balanced and strategic approach, improving the model's discrimination between word tokens and numerical values. This enables LLMs to better understand MWPs intricacies, resulting in significant performance improvements. Our approach bridges the gap between linguistic comprehension and mathematical problem-solving, boosting LLMs' efficacy in solving MWPs and holds potential for broader applications in various knowledge domains.

Downloads

Next from EMNLP 2023

YORC: Yoruba Reading Comprehension dataset