Spain

In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models&#39; dynamics.

COLING 2020

Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection

counterfactual detection

causal reasoning

self-attention

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models' dynamics.

poster

- Proposing a domain-tuned contextualized language model for
Offensive Language Detection task.
- Experimenting with different components and configurations (e.g.,
a multi-view SVM) stacked upon BERT models for specific sub-tasks.
- Achieving F1 scores of 91.7% in Sub-task A, 66.5% in Sub-task B,
and 63.2% in Sub-task C, ranking 5th, 6th, and 11th respectively.
- Performing error analysis to show common misclassification errors
made by our model and outlining research directions for future.

Domain-Tuned Contextualized Models for Offensive Language Detection

We describe the system submitted by the SWAGex team to the SemEval-2020 Commonsense Validation and Explanation Task. We used multiple methods on the pre-trained language model BERT (Devlin et al., 2018) for tasks that require the system to recognize sentences against commonsense and justify the reasoning behind this decision. Our best performing model is BERT trained on SWAG fine-tuned for the task. We investigate the ability to transfer commonsense knowledge from SWAG to SemEval-2020 by training a model for the Explanation task with Next Event Prediction data.

Commonsense Explanation as Next Event Prediction

This poster describes the NTUAAILS submission for SemEval 2020 Task 11 Detection of Propaganda Techniques in News Articles. Using deep learning techniques to detect and classify propaganda techniques, we can automatically provide the means to address propaganda and fake news.
In order to achieve this goal, two separate models were created using bidirectional LSTMs and pre-trained word embeddings like ELMo and GloVe. 
At first, the detection sub-task (Span identification) was addressed as Named Entity Recognition problem (NER). The entities consisted of two categories (token-level) propaganda and no-propaganda.
After the detection, the second sub-task's (Technique Classification) goal was to classify the given propaganda fragments in the 14 propaganda classes.
The models achieved rank 28 among 36 teams with F1 score of 0.335 and ranked 25 among 31 teams with 0.463 F1 scores for Span Identification and Technique Classification sub-tasks respectively. Our results indicate that the proposed deep learning models, although relatively simple in architecture and fast to train, achieve satisfactory results in the tasks on hand.

Propaganda Detection and Classification: BiLSTMs and ELMo approach

OffensEval 2: Multilingual Offensive Language Identification in Social Media is Task 12 of SemEval-2020 international workshop on Semantic Evaluation. Our team created a solution dedicated to this task by leveraging two pre-trained state-of-the-art models: BERT and XLNet. The models were fine-tuned for each subtask separately and features taken from their hidden layers were combined and fed into a fully connected neural network. The model using aggregated Transformer features can serve as a powerful tool for offensive language identification problem. Our team was ranked 7th out of 40 in Sub-task C - Offense target identification with 64.727% macro F1-score and 64th out of 85 in Sub-task A - Offensive language identification (89.726% F1-score).

Aggregation of Transformer-based models’ features for offensive language recognition

We consider detection of the span of antecedents and consequents in argumentative prose a structural, grammatical task. Our system comprises a set of stacked Bi-LSTMs trained on two complementary linguistic annotations. We explore the effectiveness of grammatical features (POS and clause type) through ablation. The reported experiments suggest that a multi-task learning approach using this external, grammatical knowledge is useful for detecting the extent of antecedents and consequents and performs nearly as well without the use of word embeddings.

Multi-task Stacked Bi-LSTMs

We present our systems to solve Task 3 of Semeval-2020, which aims to predict the effect that context has on human perception of similarity of words. The task consists of two subtasks in English, Croatian, Finnish and Slovenian: (1) predicting the change of similarity, and (2) predicting the human scores of similarity, both of them for a pair of words within two different contexts. We tackled the problem by developing two systems, the first one uses a centroid approach and word vectors. The second one uses the ELMo language model, which is trained for each pair of words with the given context. Our approach achieved the highest score in subtask 2 for the English language.

Predicting Contextual Word Similarity Using a Centroid based Approach and Word Embeddings

This poster describes our approach to "DeftEval: Extracting Definitions from Free Text in Textbooks" competition held as a part of SemEval 2020. The task was devoted to finding and labeling definitions in texts.

Multi-task learning for Definition Extraction

We propose hybrid models (HybridE and HybridW) for meme analysis (SemEval 2020 Task 8), which involves sentiment classification (Subtask A), humor classification (Subtask B), and scale of semantic classes (Subtask C). The hybrid model consists of BLSTM and CNN for text and image processing respectively. HybridE provides equal weight to BLSTM and CNN performance, while HybridW provides weightage based on the performance of BLSTM and CNN on a validation set. The performances (macro F1) of our hybrid model on Subtask A are 0.329 (HybridE),0.328 (HybridW), on Subtask B are 0.507 (HybridE), 0.512 (HybridW), and on Subtask C are0.309 (HybridE), 0.311 (HybridW).

You’re not the only one
cursed with knowledge –Multi branch model memotion analysis

In this paper, we assess the ability of BERT and its derivative models (RoBERTa, DistilBERT, and ALBERT) for short-edits-based humor grading. We test these models for humor grading and classification tasks on the Humicroedit and the FunLines dataset. We perform extensive experiments with these models to test their language modeling and generalization abilities via zero-shot inference and cross dataset inference-based approaches. Further, we also inspect the role of self-attention layers in humor-grading by performing a qualitative analysis over the self-attention weights from the final layer of the trained BERT model. Our experiments show that all the pre-trained BERT derivative models show significant generalization capabilities for humor-grading related tasks.

Assessing the Ability of BERT and Derivative Models to Perform Short-Edits based Humor Grading

This work presents our contribution to the Offensive Language Classification Task (EnglishSubTask A) of Semeval 2020. We propose different Bert models trained on several offensive language classification and profanity datasets, and combine their output predictions in an ensemble model. We experimented with different ensemble approaches, such as SVMs, Gradient boosting,AdaBoosting and Logistic Regression. We further propose an under-sampling approach of thecurrent SOLID dataset, which removed the most uncertain partitions of the dataset, increasing therecall of the dataset. Our best model, an average ensemble of four different Bert models, achieved11th place out of 82 participants with a macro F1 score of 0.91344 in the English SubTask A.

Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection

Downloads

Next from COLING 2020

Domain-Tuned Contextualized Models for Offensive Language Detection

Similar lecture

Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention

Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Domain-Tuned Contextualized Models for Offensive Language Detection

Similar lecture

Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention

Downloads