Spain

Arabic, like other highly inflected languages, encodes a large amount of information in its morphology and word structure. In this work, we propose two embedding strategies that modify the tokenization phase of traditional word embedding models (Word2Vec) and contextual word embedding models (BERT) to take into account Arabic&#39;s relatively complex morphology. In Word2Vec, we segment words into subwords during training time and then compose word-level representations from the subwords during test time. We train our embeddings on Arabic Wikipedia and show that they perform better than a Word2Vec model on multiple Arabic natural language processing datasets while being approximately 60% smaller in size. Moreover, we showcase our embeddings&#39; ability to produce accurate representations of some out-of-vocabulary words that were not encountered before. In BERT, we modify the tokenization layer of Google&#39;s pretrained multilingual BERT model by incorporating information on morphology. By doing so, we achieve state of the art performance on two Arabic NLP datasets without pretraining.

COLING 2020

On the Importance of Tokenization in Arabic Embedding Models

arabic

embeddings

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

Arabic, like other highly inflected languages, encodes a large amount of information in its morphology and word structure. In this work, we propose two embedding strategies that modify the tokenization phase of traditional word embedding models (Word2Vec) and contextual word embedding models (BERT) to take into account Arabic's relatively complex morphology. In Word2Vec, we segment words into subwords during training time and then compose word-level representations from the subwords during test time. We train our embeddings on Arabic Wikipedia and show that they perform better than a Word2Vec model on multiple Arabic natural language processing datasets while being approximately 60% smaller in size. Moreover, we showcase our embeddings' ability to produce accurate representations of some out-of-vocabulary words that were not encountered before. In BERT, we modify the tokenization layer of Google's pretrained multilingual BERT model by incorporating information on morphology. By doing so, we achieve state of the art performance on two Arabic NLP datasets without pretraining.

poster

We present our work on automatically detecting isnads, the chains of authorities for a report that serve as citations in hadith and other classical Arabic texts. We experiment with both sequence labeling methods for identifying isnads in a single pass and a hybrid “retrieve-and-tag” approach.

Tracing Traditions: Automatic Extraction of Isnads from Classical Arabic Texts

This research focuses on improving sequence labelling (SL) tasks by feeding neural networks with language specific characteristics in the form of embeddings.
Arabic is a morphologically rich language.
Arabic NLP researchers have used two types of approaches to work with Arabic text. 

Our Contribution: 
We propose a diacritic-aware architecture for SL which extends and outperforms the current standard character-aware architecture of a Bi-directional long short-term memory network (Bi-LSTM) with a CRF. 
We also propose a combination architecture for NER that combines word, character, diacritic, and POS information and outperforms the standard character-aware architecture and our own diacritic-aware architecture.





Embed More Ignore Less (EMIL): Exploiting Enriched Representations for Arabic NLP


Social media user-generated text is actually the main resource for many NLP tasks. This text however, does not follow the standard rules of writing. Moreover, the use of dialects such as Moroccan Arabic in written communications increases further NLP tasks complexity. A dialect is a verbal language that does not have a standard orthography, which leads users to improvise spelling while writing. Thus, for the same word we can find multiple forms of transliterations. Subsequently, it is mandatory to normalize these different transliterations to one canonical word form. To reach this goal, we have exploited the powerfulness of word embedding models generated with a corpus of YouTube comments. Besides, using a Moroccan Arabic dialect dictionary that provides the canonical forms, we have built a normalization dictionary that we refer to as MANorm . We have conducted several experiments to demonstrate the efficiency of MANorm, which have shown its usefulness in dialect normalization.

MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script

While online Arabic is primarily written using the Arabic script, a Roman-script variety called Arabizi is often seen on social media. Although this representation captures the phonology of the language, it is not a one-to-one mapping with the Arabic script version. This issue is exacerbated by the fact that Arabizi on social media is Dialectal Arabic which does not have a standard orthography. Furthermore, Arabizi tends to include a lot of code mixing between Arabic and English (or French). To map Arabizi text to Arabic script in the context of complete utterances, previously published efforts have split Arabizi detection and Arabic script target in two separate tasks. In this paper, we present the first effort on a unified model for Arabizi detection and transliteration into a code-mixed output with consistent Arabic spelling conventions, using a sequence-to-sequence deep learning model. Our best system achieves 80.6% word accuracy and 58.7% BLEU on a blind test set.

A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models

We propose a multi-task sequence prediction system, based on recurrent neural networks and used to annotate a Tunisian Arabizi corpus on multiple levels. The performed annotation are text classification, tokenization, PoS tagging and encoding of Tunisian Arabizi into CODA* Arabic orthography. The system is learned to predict all the annotation levels in cascade, starting from Arabizi input. We evaluate the system on the TIGER German corpus, suitably converting data to have a multi-task problem, in order to show the effectiveness of our neural architecture. We show how we used the system in order to annotate a Tunisian Arabizi corpus, which has been subsequently manually corrected and used to further evaluate sequence models on Tunisian data. Our system is developed for the Fairseq framework, which allows for a fast and easy use for any other sequence prediction problem.


Multi-Task Sequence Prediction for Tunisian Arabizi Multi-Level Annotation

Recent work has shown that distributional word vector spaces often encode human biases like sexism or racism. In this work, we conduct an extensive analysis of biases in Arabic word embeddings by applying a range of recently introduced bias tests on a variety of embedding spaces induced from corpora in Arabic. We measure the presence of biases across several dimensions, namely: embedding models (SKIP-GRAM, CBOW, and FASTTEXT) and vector sizes, types of text (encyclopedic text, and news vs. user-generated content), dialects (Egyptian Arabic vs. Modern Standard Arabic), and time (diachronic analyses over corpora from different time periods). Our analysis yields several interesting findings, e.g., that implicit gender bias in embeddings trained on Arabic news corpora steadily increases over time (between 2007 and 2017). We make the Arabic bias specifications (AraWEAT) publicly available.

AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings

The difficulty of processing dialects is clearly observed in the high cost of building representative corpus, in particular for machine translation. Indeed, all machine translation systems requirea huge amount and good management of training data, which represents a challenge in a low-resource setting such as the Tunisian Arabic dialect. The paper present a data augmentation technique to create a parallel corpus for Tunisian Arabic dialect written in social media and standard Arabic in order to build a MachineTranslation model. The created corpus was used to build a sentence-based translation model for testing data. This model reached a BLEU score of 15.03%on a test set, while it was limited to 13.27% utilizing the corpus without augmentation.

Parallel resources for Tunisian Arabic dialect translation

During the last two decades, we have progressively turned to the Internet and social media to
find news, entertain conversations and share opinion. Recently, OpenAI has developed a machine
learning system called GPT-2 for Generative Pre-trained Transformer-2, which can produce
deepfake texts. It can generate blocks of text based on brief writing prompts that look
like they were written by humans, facilitating the spread false or auto-generated text. In line
with this progress, and in order to counteract potential dangers, several methods have been proposed
for detecting text written by these language models. In this paper, we propose a transfer
learning based model that will be able to detect if an Arabic sentence is written by humans or
automatically generated by bots. Our dataset is based on tweets from a previous work, which
we have crawled and extended using the Twitter API. We used GPT2-Small-Arabic to generate
fake Arabic Sentences. For evaluation, we compared different recurrent neural network (RNN)
word embeddings based baseline models, namely: LSTM, BI-LSTM, GRU and BI-GRU, with a
transformer-based model. Our new transfer-learning model has obtained an accuracy up to 98%.
To the best of our knowledge, this work is the first study where ARABERT and GPT2 were
combined to detect and classify the Arabic auto-generated texts.

BERT Transformer model for Detecting Arabic GPT2 Auto-Generated Tweets

Globalization has caused the rise of the code-switching phenomenon among multilingual societies. In Arab countries, code-switching between Arabic and English has become frequent, especially through social media platforms. Consequently, research in Natural Language Processing (NLP) systems increased to tackle such a phenomenon. One of the significant challenges of developing code-switched NLP systems is the lack of data itself. We propose an open source trained bilingual contextual word embedding models of FLAIR, BERT, and ELECTRA.
We also propose a novel contextual word embedding model called KERMIT, which can efficiently map Arabic and English words inside one vector space in terms of data usage. We applied intrinsic and extrinsic evaluation methods to compare the performance of the models. Our results show that FLAIR and FastText achieve the highest results in the sentiment analysis task. However, KERMIT is the best-achieving model on the intrinsic evaluation and named entity recognition. Also, it outperforms the other transformer-based models on question answering task.

Contextual Embeddings for Arabic-English Code-Switched Data

Automatic categorization of short texts, such as news headlines and social media posts, has many applications ranging from content analysis to recommendation systems. In this paper, we use such text categorization i.e., labeling the social media posts to categories like `sports', `politics', `human-rights' among others, to showcase the efficacy of models across different sources and varieties of Arabic. In doing so, we show that diversifying the training data, whether by using diverse training data for the specific task (an increase of 21% macro F1) or using diverse data to pre-train a BERT model (26% macro F1), leads to overall improvements in classification effectiveness.
In our work, we also introduce two new Arabic text categorization datasets, where the first is composed of social media posts from a popular Arabic news channel that cover Twitter, Facebook, and YouTube, and the second is composed of tweets from popular Arabic accounts. The posts in the former are nearly exclusively authored in modern standard Arabic (MSA), while the tweets in the latter contain both MSA and dialectal Arabic.

On the Importance of Tokenization in Arabic Embedding Models

Downloads

Next from COLING 2020

Tracing Traditions: Automatic Extraction of Isnads from Classical Arabic Texts

Similar lecture

Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy

On the Importance of Tokenization in Arabic Embedding Models

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Tracing Traditions: Automatic Extraction of Isnads from Classical Arabic Texts

Similar lecture

Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy

Downloads