Spain

In this paper, we present language model system submitted to SemEval-2020 Task 4 competition: ”Commonsense Validation and Explanation”. We participate in two subtasks for subtask A: validation and subtask B: Explanation. We implemented with transfer learning using pretrained language models (BERT, XLNet, RoBERTa, and ALBERT) and fine-tune them on this task. Then we compared their characteristics in this task to help future researchers understand and use these models more properly. The ensembled model better solves this problem, making the model’s accuracy reached 95.9% on subtask A, which just worse than human’s by only 3% accuracy.

COLING 2020

Commonsense Validation and Explanation system based on ensemble of language model

pretrain model

ensemble

commonsense

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

poster

We present the results of our system for SemEval-2020 Task 1 that exploits a commonly used lexical semantic change detection model based on Skip-Gram with Negative Sampling. Our system focuses on Vector Initialization (VI) alignment, compares VI to the currently top-ranking models for Subtask 2 and demonstrates that these can be outperformed if we optimize VI dimensionality. We demonstrate that differences in performance can largely be attributed to model-specific sources of noise, and we reveal a strong relationship between dimensionality and frequency-induced noise in VI alignment. Our results suggest that lexical semantic change models integrating vector space alignment should pay more attention to the role of the dimensionality parameter.

How low can you go? Dimensionality in Lexical Semantic Change Detection

In this work, we tackled seamEval-2020 Task A, B, and C of Offensive Language Challenge at SemEval2020. We handled offensive language in five languages: English, Greek, Danish, Arabic, and Turkish. Specifically, we pre-processed all
provided datasets and developed an appropriate strategy to handle Tasks (A, B, & C) for identifying the presence/absence, type, and the target of
offensive language in social media. For this purpose, we used OLID2019 [1], OLID2020 [2] datasets, and generated new datasets using Google translation API (Fig. 1), which we made publicly available. We discussed the limitations
and the success of our machine learning-based approach for all five different languages. Our results for identifying offensive posts (Task A)
yielded satisfactory accuracy of 0.92 for English, 0.81 for Danish, 0.84 for Turkish, 0.85 for Greek, and 0.89 for Arabic. For the type detection (Task
B), the results are significantly higher (.87 accuracy) compared to target detection (Task C), which yields .81 accuracy. Moreover, after using automated Google translation, the overall efficiency improved by 2% for Greek, Turkish, and Danish.

Multilingual Identification of Offensive Language, Type, and Target of Twitter Post Using Translated Datasets.

We describe our contribution to two of the subtasks of SemEval 2020 Task 6, DeftEval: Extracting term-definition pairs in free text. The system for Subtask 1: Sentence Classification is based on a transformer architecture where we use transfer learning to fine-tune a pretrained model on the downstream task, and the one for Subtask 3: Relation Classification uses a Random Forest classifier with handcrafted dedicated features. Our systems respectively achieve 0.830 and 0.994 F1-scores on the official test set, and we believe that the insights derived from our study are potentially relevant to help advance the research on definition extraction.

A Supervised Approach for Definition Extraction

Poster for system that was designed for Humor evaluation within the SemEval-2020 Task 7. The system is based on convolutional neural network architecture. In our paper we investigate the system on the official dataset, and we provide more insight to model itself to see how the learned inner features look.

Convolutional humor

This paper describes our contribution to SemEval-2020 Task 8: Memotion Analysis. Our system learns multi-modal embeddings from text and images in order to classify Internet memes by sentiment. Our model learns text embeddings using BERT and extracts features from images with DenseNet, subsequently combining both features through concatenation. We also compare our results with those produced by DenseNet, ResNet, BERT, and BERT-ResNet. Our results show that image classification models have the potential to help classifying memes, with DenseNet outperforming ResNet. Adding text features is however not always helpful for Memotion Analysis.

Utilizing BERT and DenseNet for Internet Meme Emotion Analysis

In this paper, we present our contribution in SemEval-2020 Task 1: Unsupervised Lexical SemanticChange Detection, where we systematically combine existing models for unsupervised capturingof lexical semantic change across time in text corpora of German, English, Latin and Swedish.In particular, we analyze the score distribution of existing models. Then we define a general classification threshold, adjust it independently to each of the models and measure the models’score certainty. Finally, using both the threshold and score certainty, we aggregate the models forthe two sub-tasks: binary classification and ranking.

Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection

We describe our system for SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. We developed ensemble models using RoBERTa-based neural architectures, additional CRF layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. We achieved sizable improvements over baseline fine-tuned RoBERTa models, and the official evaluation ranked our system 3rd (almost tied with the 2nd) out of 36 teams on the span identification subtask with an F1 score of 0.491, and 2nd (almost tied with the 1st) out of 31 teams on the technique classification subtask with an F1 score of 0.62.

It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning

We present the MULTISEM systems submitted to SemEval 2020 Task 3: Graded Word Similarity in Context (GWSC). We experiment with injecting semantic knowledge into pre-trained BERT models through fine-tuning on lexical semantic tasks related to GWSC. We use existing semantically annotated datasets and propose to approximate similarity through automatically generated lexical substitutes in context. We participate in both GWSC subtasks and address two languages, English and Finnish. Our best English models occupy the third and fourth positions in the ranking for the two subtasks. Performance is lower for the Finnish models which are mid-ranked in the respective subtasks, highlighting the important role of data availability for fine-tuning.

Fine-tuning BERT for Lexical Meaning

CoSimLex is a dataset that can be used to evaluate the ability of context-dependent word embeddings for modeling subtle, graded changes of meaning, as perceived by humans during reading. At SemEval-2020, task 3, subtask 1 is about ”predicting the (graded) effect of context in word similarity”, using CoSimLex to quantify such a change of similarity for a pair of words, from one context to another. Here, a meaning shift is composed of two aspects, a) discrete changes observed between different word senses, and b) more subtle changes of meaning representation that are not captured in those discrete changes. Therefore, this SemEval task was designed to allow the evaluation of systems that can deal with a mix of both situations of semantic shift, as they occur in the human perception of meaning. The described system was developed to improve the BERT baseline provided with the task, by reducing distortions in the BERT semantic space, compared to the human semantic space. To this end, complementarity between 768- and 1024-dimensional BERT embeddings, and average word sense vectors were used. With this system, after some fine-tuning, the baseline performance of 0.705 (uncentered Pearson correlation with human semantic shift data from 27 annotators) was enhanced by more than 6%, to 0.7645. We hope that this work can make a contribution to further our understanding of the semantic vector space of human perception, as it can be modeled with context-dependent word embeddings in natural language processing systems.

Combining BERT with WordNet sense embeddings to predict graded word similarity changes

Applying pre-trained transformer to select emphasis for written text in visual media [1]. We propose PointWise Regression and PairWise Ranking for the emphasis modeling. We also apply data augmentation and hand-craft features to boost the performance. We have a full investigation across different pre-trained transformers, such as ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. And we finally take the 1st place in this competition.


Downloads

Next from COLING 2020

How low can you go? Dimensionality in Lexical Semantic Change Detection

Similar lecture

CommonSense Explanation, Reasoning and Prediction with Multi-task Learning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

How low can you go? Dimensionality in Lexical Semantic Change Detection

Similar lecture

CommonSense Explanation, Reasoning and Prediction with Multi-task Learning

Downloads