Singapore

Large Language Models (LLMs) have taken center stage in recent years in the field of NLP. But while they have shown significant performance gains in many NLP tasks, their performance is concentrated in Anglo-Centric languages. Although previous studies have highlighted shortcomings of LLMs---including perpetuating harms and biases---we know little about their failure cases in low-resourced languages. We investigate how ChatGPT responds in Amharic and Afan Oromo, two low-resourced languages spoken in Ethiopia. We taxonomize the failure cases we observed and provide examples to illustrate the shortcomings. Our results show text generated in these languages has 6 failure modes and additionally exhibits political utterances that are not related to the prompts. We hope our work will contribute to the broader understanding of LLMs for low-resourced languages.

EMNLP 2023

Evaluating Large Language Models in Low-Resource Settings

afan oromo

low-resourced languaged

chatgpt

amharic

large language models

workshop paper

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

Multilingual language models (MLMs) have democratized access to information and artificial intelligence. However, low-resource languages remain underrepresented. This study introduces code-switching as a test to assess the low-resource multilingual capabilities of MLMs using standard Basque-English code-switching as a benchmark. The models were evaluated focusing on output naturalness, code-switching aptitude, and hallucination frequency. Employing codeswitching as an evaluative measure revealed significant differences in naturalness and the ability to produce grammatical codeswitched outputs across models. The paper finds that models have accurate knowledge of language contact, as shown by the mixing of languages out of the prompt domain.

Do You Speak Basquenglish? Assessing Low-resource Multilingual Proficiency of Pretrained Language Models

Social media data has become a crucial re- source for understanding and detecting mental health challenges. However, there is a significant gap in our understanding of age-specific linguistic markers associated with classifying depression. This study bridges the gap by analyzing 25,241 text samples from 15,156 Reddit users with self-reported depression across two age groups: adolescents (13-20 year olds) and adults (21+). Through a quantitative exploratory analysis using LIWC, topic modeling, and data visualization, distinct patterns and topical differences emerged in the language of depression for adolescents and adults, including social concerns, temporal focuses, emotions, and cognition. These findings enhance our understanding of how depression is expressed on social media, bearing implications for accurate classification and tailored interventions across different age groups.

An Exploratory Analysis of Differential Linguistic Features of Depression in Adolescents and Adults via Social Media | VIDEO

Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation

The main goal of this work is to develop a
bi-directional machine translation between Amharic
and Kistanigna. To increase the content of the language
on the web, to address the issue of the endangered
Kistanigna language and to share the information
between the language, it is essential to develop machine
translation between Amharic and Kistanigna language.
In this study, the translation is implemented using
neural machine translation. We conduct the
experiments using LSTM, Bi-LSTM, LSTM +
attention, CNN + attention, and the Transformer
encoder-decoder model. For our experiments, we have
used 9,225 parallel sentences with morpheme-based
translation units. To segment our morpheme data, we
used the morfessor tool. We considered training time,
memory usage, and BLEU sco re when proposing an
optimal model. Finally, we suggested morpheme-based
bidirectional machine translation using a Transformer
with BLEU scores of 21.31 and 22.40 for AmharicKistanigna and Kistanigna-Amharic translation
respectively. The major weakness of the study is the
unavailability of enough datasets to conduct an
extensive experiment. As a result, there is a need to
prepare parallel corpora for conducting similar
research.

Morpheme-based Bi-directional Machine Translation using Deep Learning

Previous studies have established that pre-trained language models inherently manifest various biases captured from the training corpora. Though several debiasing strategies, such as fine-tuning the whole model with counterfactuals, prompt tuning, and representation projection, have been introduced, they often fall short of effectively altering the models' intrinsic parameters without retaining language modeling ability. To address this, we propose EditBias, changing models' biased properties by modifying partial parameters while preserving meaningless associations. Experiments show that EditBias achieves excellent performance on debiasing.

Debiasing Stereotyped Models via Model Editing

Large Language Models (LLMs) have showcased remarkable advancements in diverse language tasks. However, their performance in mathematical reasoning, particularly in solving Mathematical Word Problems (MWPs), remains a challenge. To address this, we propose a novel masking technique tailored to enhance LLMs' MWPs-solving abilities. Unlike random masking, our method employs a balanced and strategic approach, improving the model's discrimination between word tokens and numerical values. This enables LLMs to better understand MWPs intricacies, resulting in significant performance improvements. Our approach bridges the gap between linguistic comprehension and mathematical problem-solving, boosting LLMs' efficacy in solving MWPs and holds potential for broader applications in various knowledge domains.

Premium content

Evaluating Large Language Models in Low-Resource Settings

Downloads

Next from EMNLP 2023

Do You Speak Basquenglish? Assessing Low-resource Multilingual Proficiency of Pretrained Language Models