Spain

We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.

COLING 2020

Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

technical paper

We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about SpanIdentification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles.

Detection of Propaganda Techniques in News Articles

Domain knowledge is important to understand
both the lexical and relational associations of
words in natural language text, especially for
domain-specific tasks like Natural Language
Inference (NLI) in the medical domain, where
due to the lack of a large annotated dataset
such knowledge cannot be implicitly learned
during training. However, because of the linguistic
idiosyncrasies of clinical texts (e.g.,
shorthand jargon), solely relying on domain
knowledge from an external knowledge base
(e.g., UMLS) can lead to wrong inference predictions
as it disregards contextual information
and, hence, does not return the most relevant
mapping. To remedy this, we devise
a knowledge adaptive approach for medical
NLI that encodes the premise/hypothesis texts
by leveraging supplementary external knowledge,
alongside the UMLS, based on the word
contexts. By incorporating refined domain
knowledge at both the lexical and relational levels
through a multi-source attention mechanism,
it is able to align the token-level interactions
between the premise and hypothesis more
effectively. Comprehensive experiments and
case study on the recently released MedNLI
dataset are conducted to validate the effectiveness
of the proposed approach.

Improving Medical NLI Using Context-Aware
Domain Knowledge

Collecting modality exclusivity norms for lexical items has recently become a common practice in psycholinguistics and cognitive research. 
However, these norms are available only for a relatively small number of languages and often involve a costly and time-consuming collection of ratings.
In this work, we aim at learning a mapping between word embeddings and modality norms. 
Our experiments focused on crosslingual word embeddings, in order to predict modality association scores by training on a high-resource language and testing on a low-resource one.
We ran two experiments, one in a monolingual and the other one in a crosslingual setting. Results show that modality prediction using off-the-shelf crosslingual embeddings indeed has moderate-to-high correlations with human ratings even when regression algorithms are trained on an English resource and tested on a completely unseen language.

Automatic Learning of Modality Exclusivity Norms with Crosslingual Word Embeddings

Emotion stimulus detection is the task of finding the cause of an emotion in a textual description, similar to target or aspect detection for sentiment analysis. Previous work approached this in three ways, namely (1) as text classification into an inventory of predefined possible stimuli (“Is the stimulus category A or B?”), (2)as sequence labeling of tokens (“Which tokens describe the stimulus?”), and (3) as clause classification (“Does this clause contain the emotion stimulus?”). So far, setting (3) has been evaluated broadly on Mandarin and (2) on English, but no comparison has been performed.Therefore, we analyze whether clause classification or token sequence labeling is better suited for emotion stimulus detection in English. We propose an integrated framework which enables us to evaluate the two different approaches comparably, implement models inspired by state-of-the-art approaches in Mandarin, and test them on four English data sets from different domains. Our results show that token sequence labeling is superior on three out of four datasets, in both clause-based and token sequence-based evaluation. The only case in which clause classification performs better is one data set with a high density of clause annotations. Our error analysis further confirms quantitatively and qualitatively that clauses are not the appropriate stimulus unit in English.

Emotion Stimulus Detection in English -- Token Sequence Labeling vs. Clause Classification

In this paper, we propose a novel method for learning cross-lingual
word embeddings, that incorporates sub-word information during
training, and is able to learn high-quality embeddings from modest
amounts of monolingual data and a bilingual lexicon. This method could
be particularly well-suited to learning cross-lingual embeddings for
lower-resource, morphologically-rich languages, enabling knowledge to
be transferred from rich- to lower-resource languages. We evaluate our
proposed approach simulating lower-resource languages for bilingual
lexicon induction, monolingual word similarity, and document
classification. Our results indicate that incorporating sub-word
information indeed leads to improvements, and in the case of document
classification, performance better than, or on par with, strong
benchmark approaches.

Joint Training for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora

Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing approaches for DST mainly fall into one of two categories, namely, ontology-based and ontology-free methods. An ontology-based method selects a value from a candidate-value list for each target slot, while an ontology-free method extracts spans from dialog contexts. Recent work introduced a BERT-based model to strike a balance between the two methods by pre-defining categorical and non-categorical slots. However, it is not clear enough which slots are better handled by either of the two slot types, and the way to use the pre-trained model has not been well investigated. In this paper, we propose a simple yet effective dual-strategy model for DST, by adapting a single BERT-style reading comprehension model to jointly handle both the categorical and non-categorical slots. Our experiments on the MultiWOZ datasets show that our method significantly outperforms the BERT-based counterpart, finding that the key is a deep interaction between the domain-slot and context information. When evaluated on noisy (MultiWOZ 2.0) and cleaner (MultiWOZ 2.1) settings, our method performs competitively and robustly across the two different settings. Our method sets the new state of the art in the noisy setting, while performing more robustly than the best model in the cleaner setting. We also conduct a comprehensive error analysis on the dataset, including the effects of the dual strategy for each slot, to facilitate future research.

Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking

Our paper offers a computational model of the semantic recoverability of verb arguments, tested in particular on direct objects and Instruments. Our fully distributional model is intended to improve on older taxonomy-based models, which require a lexicon in addition to the training corpus. We computed the selectional preferences of 99 transitive verbs and 173 Instrument verbs as the mean value of the pairwise cosine similarity between their arguments (a weighted mean between all the arguments, or an unweighted mean with the topmost k arguments). Results show that our model can predict the recoverability of objects and Instruments, providing a similar result to that of taxonomy-based models but at a much cheaper computational cost.

PISA: A Measure of Preference in Selection of Arguments to Model Verb Argument Recoverability

We present a semi-supervised model which learns the semantics of negation purely through analysis of syntactic structure. Linguistic theory posits that the semantics of negation can be understood purely syntactically, though recent research relies on combining a variety of features including part-of-speech tags, word embeddings, and semantic representations to achieve high task performance. Our simplified model returns to syntactic theory and achieves state-of-the-art performance on the task of Negation Scope Detection while demonstrating the tight relationship between the syntax and semantics of negation.

Learning Negation Scope from Syntactic Structure

There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a mixed effects model by incorporating annotator random effects into any existing neural model, improves performance over models that do not incorporate such effects.

Natural Language Inference with Mixed Effects

Co-predication is one of the most frequently used linguistic tests to tell apart shifts in polysemic sense from changes in homonymic meaning. It is increasingly coming under criticism as evidence is accumulating that it tends to mis-classify specific cases of polysemic sense alteration as homonymy. In this paper, we collect empirical data to investigate these accusations. We asses how co-predication acceptability relates to explicit ratings of polyseme word sense similarity, and how well either measure can be predicted through the distance between target words' contextualised word embeddings. 
We find that sense similarity appears to be a major contributor in determining co-predication acceptability, but that co-predication judgements tend to rate less similar sense interpretations as being as unacceptable as homonym pairs, effectively mis-classifying these instances. The tested contextualised word embeddings fail to predict word sense similarity consistently, but the similarities between BERT embeddings show a significant correlation with co-predication ratings. We take this finding as evidence that BERT embeddings might be better representations of context than encodings of word meaning.

Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

Downloads

Next from COLING 2020

Detection of Propaganda Techniques in News Articles

Similar lecture

Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale

Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Detection of Propaganda Techniques in News Articles

Similar lecture

Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale

Downloads