Spain

We describe our contribution to two of the subtasks of SemEval 2020 Task 6, DeftEval: Extracting term-definition pairs in free text. The system for Subtask 1: Sentence Classification is based on a transformer architecture where we use transfer learning to fine-tune a pretrained model on the downstream task, and the one for Subtask 3: Relation Classification uses a Random Forest classifier with handcrafted dedicated features. Our systems respectively achieve 0.830 and 0.994 F1-scores on the official test set, and we believe that the insights derived from our study are potentially relevant to help advance the research on definition extraction.

COLING 2020

A Supervised Approach for Definition Extraction

deft corpus

sentence classifier

definition extraction

transformer

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

poster

Poster for system that was designed for Humor evaluation within the SemEval-2020 Task 7. The system is based on convolutional neural network architecture. In our paper we investigate the system on the official dataset, and we provide more insight to model itself to see how the learned inner features look.

Convolutional humor

This paper describes our contribution to SemEval-2020 Task 8: Memotion Analysis. Our system learns multi-modal embeddings from text and images in order to classify Internet memes by sentiment. Our model learns text embeddings using BERT and extracts features from images with DenseNet, subsequently combining both features through concatenation. We also compare our results with those produced by DenseNet, ResNet, BERT, and BERT-ResNet. Our results show that image classification models have the potential to help classifying memes, with DenseNet outperforming ResNet. Adding text features is however not always helpful for Memotion Analysis.

Utilizing BERT and DenseNet for Internet Meme Emotion Analysis

In this paper, we present our contribution in SemEval-2020 Task 1: Unsupervised Lexical SemanticChange Detection, where we systematically combine existing models for unsupervised capturingof lexical semantic change across time in text corpora of German, English, Latin and Swedish.In particular, we analyze the score distribution of existing models. Then we define a general classification threshold, adjust it independently to each of the models and measure the models’score certainty. Finally, using both the threshold and score certainty, we aggregate the models forthe two sub-tasks: binary classification and ranking.

Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection

We describe our system for SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. We developed ensemble models using RoBERTa-based neural architectures, additional CRF layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. We achieved sizable improvements over baseline fine-tuned RoBERTa models, and the official evaluation ranked our system 3rd (almost tied with the 2nd) out of 36 teams on the span identification subtask with an F1 score of 0.491, and 2nd (almost tied with the 1st) out of 31 teams on the technique classification subtask with an F1 score of 0.62.

It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning

We present the MULTISEM systems submitted to SemEval 2020 Task 3: Graded Word Similarity in Context (GWSC). We experiment with injecting semantic knowledge into pre-trained BERT models through fine-tuning on lexical semantic tasks related to GWSC. We use existing semantically annotated datasets and propose to approximate similarity through automatically generated lexical substitutes in context. We participate in both GWSC subtasks and address two languages, English and Finnish. Our best English models occupy the third and fourth positions in the ranking for the two subtasks. Performance is lower for the Finnish models which are mid-ranked in the respective subtasks, highlighting the important role of data availability for fine-tuning.

Fine-tuning BERT for Lexical Meaning

CoSimLex is a dataset that can be used to evaluate the ability of context-dependent word embeddings for modeling subtle, graded changes of meaning, as perceived by humans during reading. At SemEval-2020, task 3, subtask 1 is about ”predicting the (graded) effect of context in word similarity”, using CoSimLex to quantify such a change of similarity for a pair of words, from one context to another. Here, a meaning shift is composed of two aspects, a) discrete changes observed between different word senses, and b) more subtle changes of meaning representation that are not captured in those discrete changes. Therefore, this SemEval task was designed to allow the evaluation of systems that can deal with a mix of both situations of semantic shift, as they occur in the human perception of meaning. The described system was developed to improve the BERT baseline provided with the task, by reducing distortions in the BERT semantic space, compared to the human semantic space. To this end, complementarity between 768- and 1024-dimensional BERT embeddings, and average word sense vectors were used. With this system, after some fine-tuning, the baseline performance of 0.705 (uncentered Pearson correlation with human semantic shift data from 27 annotators) was enhanced by more than 6%, to 0.7645. We hope that this work can make a contribution to further our understanding of the semantic vector space of human perception, as it can be modeled with context-dependent word embeddings in natural language processing systems.

Combining BERT with WordNet sense embeddings to predict graded word similarity changes

Applying pre-trained transformer to select emphasis for written text in visual media [1]. We propose PointWise Regression and PairWise Ranking for the emphasis modeling. We also apply data augmentation and hand-craft features to boost the performance. We have a full investigation across different pre-trained transformers, such as ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. And we finally take the 1st place in this competition.


Learning Word Emphasis Selection by Pre-trained Language Model

Memes have become an ubiquitous social media entity and the processing and analysis of such multimodal data is currently an active area of research. This poster presents our work on the Memotion Analysis shared task of SemEval 2020, involving the sentiment and humor analysis of memes. We propose a system which uses different bimodal fusion techniques to leverage the inter-modal dependency for sentiment and humor classification tasks. Out of all our experiments, the best system improved the baseline with macro F1 scores of 0.357 on Sentiment Classification (Task A), 0.510 on Humor Classification (Task B) and 0.312 on Scales of Semantic Classes (Task C).

Bi-Fusion Techniques for Deep Meme Emotion Analysis

This paper presents the team BRUMS submission to SemEval-2020 Task 3: Graded Word Similarity in Context. The system utilises state-of-the-art contextualised word embeddings, which have some task-specific adaptations, including stacked embeddings and average embeddings. Overall, the approach achieves good evaluation scores across all the languages, while maintaining simplicity. Following the final rankings, our approach is ranked within the top 5 solutions of each language while preserving the 1st position of Finnish subtask 2.

Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity

The poster presents our work on Semeval Task 4. Aside from creating competitive English models, we evaluate different approaches to using the system in the Czech language. We also show how models from subtask B can be used to rerank and improve outputs of models in subtask C. Inversely, we demonstrate that accuracy of subtask C models on subtask B correlates better with human judgement than BLEU score, which was used for the official evaluation.

Downloads

Next from COLING 2020

Convolutional humor

Similar lecture

Fine-tuning BERT for Domain-Specific Definition Classification

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Convolutional humor

Similar lecture

Fine-tuning BERT for Domain-Specific Definition Classification

Downloads