Spain

In this paper, we present SemEval-2020 Task 4, CommonsenseValidation, and explanation(ComVE), which includes three subtasks, aiming to evaluate whether a system can distinguish a natural language statement that makes sense to humans from one that does not, and provide the reasons. Specifically, in our first subtask, the participating systems are required to choose from two natural language statements of similar wording the one that makes sense and the one does not. The second subtask additionally asks a system to select the key reason from three options why a given statement does not make sense. In the third subtask, a participating system needs to generate reason. We finally attracted 39 teams participating in at least one of the three subtasks. For SubtaskA and Subtask B, the performances of top-ranked systems are close to that of humans. However, for Subtask C, there is still a relatively large gap between systems and human performance. The dataset used in our task can be found at https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation;The leaderboard can be found athttps://competitions.codalab.org/competiti

COLING 2020

Commonsense Validation and Explanation

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

technical paper

Lexical entailment (LE) is a fundamental asymmetric lexico-semantic relation, supporting the hierarchies in lexical resources (e.g., WordNet, ConceptNet) and applications like natural language inference and taxonomy induction. Multilingual and cross-lingual NLP applications warrant models for LE detection that go beyond language boundaries. As part of SemEval 2020, we carried out a shared task (Task 2) on multilingual and cross-lingual LE. The shared task spans three dimensions: (1) monolingual LE in multiple languages versus cross-lingual LE, (2) binary versus graded LE, and (3) a set of 6 diverse languages (and 15 corresponding language pairs). We offered two different evaluation tracks: (a) distributional (Dist): for unsupervised, fully distributional models that capture LE solely on the basis of unannotated corpora, and (b) Any: for externally informed models, allowed to leverage any resources, including lexico-semantic networks (e.g., WordNet or BabelNet). In the Any track, we received system runs that push state-of-the-art across all languages and language pairs, for both binary LE detection and graded LE prediction.

Predicting Multilingual and Cross-Lingual
(Graded) Lexical Entailment

Information on social media comprises of various modalities such as textual, visual and audio.NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, the computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content any-more. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis released approx 10K an-notated memes- with human-annotated labels namely sentiment(positive, negative, neutral), type of emotion(sarcastic, funny,offensive, motivation) and their corresponding intensity. The challenge consisted of three subtasks: sentiment (positive, negative, and neutral) analysis of memes, overall emotion (humour, sarcasm, offensive, and motivational) classification of memes, and classifying intensity of meme emotion. The best performances achieved were F1(macro average)scores of 0.35, 0.51 and 0.32, respectively for each of the three subtasks.

The Visuo-Lingual Metaphor!

We present the results of the SemEval-2020 Task 9 on Sentiment Analysis of CodeMixed Tweets (SentiMix 2020). We also release and describe our Hinglish (Hindi-English) and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels. These corpora are comprised of 20K and 19K examples, respectively. The sentiment labels are - Positive, Negative, and Neutral. SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submitted systems to the Spanglish competition. The best performance achieved was 75.0% F1 score
for Hinglish and 80.6% F1 for Spanglish. We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants.

Overview of Sentiment Analysis of
Code-Mixed Tweets (Sentimix)

We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.

Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about SpanIdentification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles.

Detection of Propaganda Techniques in News Articles

Domain knowledge is important to understand
both the lexical and relational associations of
words in natural language text, especially for
domain-specific tasks like Natural Language
Inference (NLI) in the medical domain, where
due to the lack of a large annotated dataset
such knowledge cannot be implicitly learned
during training. However, because of the linguistic
idiosyncrasies of clinical texts (e.g.,
shorthand jargon), solely relying on domain
knowledge from an external knowledge base
(e.g., UMLS) can lead to wrong inference predictions
as it disregards contextual information
and, hence, does not return the most relevant
mapping. To remedy this, we devise
a knowledge adaptive approach for medical
NLI that encodes the premise/hypothesis texts
by leveraging supplementary external knowledge,
alongside the UMLS, based on the word
contexts. By incorporating refined domain
knowledge at both the lexical and relational levels
through a multi-source attention mechanism,
it is able to align the token-level interactions
between the premise and hypothesis more
effectively. Comprehensive experiments and
case study on the recently released MedNLI
dataset are conducted to validate the effectiveness
of the proposed approach.

Improving Medical NLI Using Context-Aware
Domain Knowledge

Collecting modality exclusivity norms for lexical items has recently become a common practice in psycholinguistics and cognitive research. 
However, these norms are available only for a relatively small number of languages and often involve a costly and time-consuming collection of ratings.
In this work, we aim at learning a mapping between word embeddings and modality norms. 
Our experiments focused on crosslingual word embeddings, in order to predict modality association scores by training on a high-resource language and testing on a low-resource one.
We ran two experiments, one in a monolingual and the other one in a crosslingual setting. Results show that modality prediction using off-the-shelf crosslingual embeddings indeed has moderate-to-high correlations with human ratings even when regression algorithms are trained on an English resource and tested on a completely unseen language.

Automatic Learning of Modality Exclusivity Norms with Crosslingual Word Embeddings

Emotion stimulus detection is the task of finding the cause of an emotion in a textual description, similar to target or aspect detection for sentiment analysis. Previous work approached this in three ways, namely (1) as text classification into an inventory of predefined possible stimuli (“Is the stimulus category A or B?”), (2)as sequence labeling of tokens (“Which tokens describe the stimulus?”), and (3) as clause classification (“Does this clause contain the emotion stimulus?”). So far, setting (3) has been evaluated broadly on Mandarin and (2) on English, but no comparison has been performed.Therefore, we analyze whether clause classification or token sequence labeling is better suited for emotion stimulus detection in English. We propose an integrated framework which enables us to evaluate the two different approaches comparably, implement models inspired by state-of-the-art approaches in Mandarin, and test them on four English data sets from different domains. Our results show that token sequence labeling is superior on three out of four datasets, in both clause-based and token sequence-based evaluation. The only case in which clause classification performs better is one data set with a high density of clause annotations. Our error analysis further confirms quantitatively and qualitatively that clauses are not the appropriate stimulus unit in English.

Emotion Stimulus Detection in English -- Token Sequence Labeling vs. Clause Classification

In this paper, we propose a novel method for learning cross-lingual
word embeddings, that incorporates sub-word information during
training, and is able to learn high-quality embeddings from modest
amounts of monolingual data and a bilingual lexicon. This method could
be particularly well-suited to learning cross-lingual embeddings for
lower-resource, morphologically-rich languages, enabling knowledge to
be transferred from rich- to lower-resource languages. We evaluate our
proposed approach simulating lower-resource languages for bilingual
lexicon induction, monolingual word similarity, and document
classification. Our results indicate that incorporating sub-word
information indeed leads to improvements, and in the case of document
classification, performance better than, or on par with, strong
benchmark approaches.

Joint Training for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora

Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing approaches for DST mainly fall into one of two categories, namely, ontology-based and ontology-free methods. An ontology-based method selects a value from a candidate-value list for each target slot, while an ontology-free method extracts spans from dialog contexts. Recent work introduced a BERT-based model to strike a balance between the two methods by pre-defining categorical and non-categorical slots. However, it is not clear enough which slots are better handled by either of the two slot types, and the way to use the pre-trained model has not been well investigated. In this paper, we propose a simple yet effective dual-strategy model for DST, by adapting a single BERT-style reading comprehension model to jointly handle both the categorical and non-categorical slots. Our experiments on the MultiWOZ datasets show that our method significantly outperforms the BERT-based counterpart, finding that the key is a deep interaction between the domain-slot and context information. When evaluated on noisy (MultiWOZ 2.0) and cleaner (MultiWOZ 2.1) settings, our method performs competitively and robustly across the two different settings. Our method sets the new state of the art in the noisy setting, while performing more robustly than the best model in the cleaner setting. We also conduct a comprehensive error analysis on the dataset, including the effects of the dual strategy for each slot, to facilitate future research.

Commonsense Validation and Explanation

Downloads

Next from COLING 2020

Predicting Multilingual and Cross-Lingual (Graded) Lexical Entailment

Similar lecture

Measuring correlation-to-causation exaggeration in press releases

Commonsense Validation and Explanation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Predicting Multilingual and Cross-Lingual (Graded) Lexical Entailment

Similar lecture

Measuring correlation-to-causation exaggeration in press releases

Downloads