Spain

This paper introduces our efforts at the FinCasual shared task for modeling causality in financial utterances.
Our approach uses the commonly and successfully applied strategy of fine-tuning a transformer-based language model with a twist, i.e. we modified the training and inference mechanism such that our model produces multiple predictions for the same instance.
By designing such a model that returns k&gt;1 predictions at the same time, we not only obtain a more resource efficient training (as opposed to fine-tuning some pre-trained language model k independent times), but our results indicate that we are also capable of obtaining comparable or even better evaluation scores that way.
We compare multiple strategies for combining the k predictions of our model.
Our submissions got ranked third on both subtasks of the shared task.

COLING 2020

Modeling causality in financial texts using multi-headed transformers

multi-head

bert

transformer

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

This paper introduces our efforts at the FinCasual shared task for modeling causality in financial utterances.
Our approach uses the commonly and successfully applied strategy of fine-tuning a transformer-based language model with a twist, i.e. we modified the training and inference mechanism such that our model produces multiple predictions for the same instance.
By designing such a model that returns k>1 predictions at the same time, we not only obtain a more resource efficient training (as opposed to fine-tuning some pre-trained language model k independent times), but our results indicate that we are also capable of obtaining comparable or even better evaluation scores that way.
We compare multiple strategies for combining the k predictions of our model.
Our submissions got ranked third on both subtasks of the shared task.

poster

We explore a section-based extractive summarization method tailored to the structure of financial reports: our best system parses the report Table of Contents (ToC), splits the report into narrative sections based on the ToC, and applies a BERT-based classifier to each section to determine whether it should be included in the summary. Our best system ranks 4th, 1st, 2nd, and 17th on the Rouge-1, Rouge-2, Rouge-SU4, and Rouge-L official metrics, respectively.

SUMSUM@FNS-2020 Shared Task

Identifying semantic relationships has become increasingly important for multiple downstream applications like Question Answering and Event Prediction. We tackle the problem of causal relationship extraction from financial news using the FinCausal 2020 dataset in two parts: 1) Detect the presence of causal relations. 2) Extract segments corresponding to cause and effect from news snippets.

Causal Extraction System

The paper describes the work that the team submitted to FinCausal 2020 Shared Task. This work is associated with the first sub-task of identifying causality in sentences. The various models used in the experiments tried to obtain a latent space representation for each of the sentences. Linear regression was performed on these representations to classify whether the sentence is causal or not. The experiments have shown BERT (Large) performed the best, giving a F1 score of 0.958, in the task of detecting the causality of sentences in financial texts and reports. The class imbalance was dealt with a modified loss function to give a better metric score for the evaluation.

Causality detection using Sentence Embeddings in Financial Reports

We present our contributions for the FinTOC'2 Shared Tasks: Table of Content (ToC) extraction in English documents and French documents. 
For ToC Extraction, we propose an approach that combines information from multiple sources: the table of contents, the wording of the document, and lexical domain knowledge.
For the title detection part, we compare surface features to character-based features on various training configurations. We show that title detection results are very sensitive to the kind of training dataset used.

Title Detection and Structure Extraction

In this paper, we present three methods developed by the Sorbonne Team for the NADI shared task on Arabic Dialect Identification for tweets.
The first and the second method use respectively a machine learning model based on a Voting Classifier with words and character level features and a deep learning model at the word level. The third method uses only character-level features. We explored different text representation such as TF-IDF (first model) and word embeddings (second model). The Voting Classifier was the most powerful prediction model, achieving the best macro-average F1 score of 18.8% and an accuracy of 36.54\% on the official test.
Our model ranked 9 on the challenge and in conclusion we propose some ideas to improve its results.

Voting Classifier vs Deep learning method in Arabic Dialect Identification

Arabic dialect detection is an important subdomain for studying various indigenous features of different societies. Limitation of the labeled data-set for training models drives us for alternative techniques. We introduce a semi-supervised learning approach for generating pseudo-labeled data for fine-tuning neural network models. This strategy improves performance when compared to training classifiers in a supervised fashion and also produces better results with reduced training time.

Semi-supervised Fine-grained Approach for Arabic Dialect Detection

Arabic is one of the most important and growing languages in the world. With the rise of the social media platforms such as like Twitter, Arabic spoken dialects have become more in use. In this paper, we describe our approach on the NADI Shared Task 1 that requires us to build a system to differentiate between different 21
Arabic dialects, we introduce a deep learning semi-supervised fashion approach along with pre-processing that was reported on NADI shared Task 1 Corpus. Our system ranks 4th in NADI’s shared task competition achieving 23.09% F1 macro average score with a simple yet an efficient approach on differentiating between 21 Arabic Dialects given tweets.

Arabic Dialect Identification Using BERT-Based Domain Adaptation

Around the Arab world, different Arabic dialects are spoken by more than 300M persons and are increasingly popular in social media texts. However, Arabic dialects are considered to be low-resource languages, limiting the development of machine-learning-based systems for these dialects. In this paper, we investigate the Arabic dialect identification task, from two perspectives: country-level dialect identification from 21 Arab countries, and province-level dialect identification from 100 provinces. We introduce an unified pipeline of state-of-the-art models, that can handle the two subtasks. Our experimental studies applied to the NADI shared task under the team name BERT-NGRAMS, show promising results both at the country-level (F1-score of 25.99%) and the province-level (F1-score of 6.39%), and thus allow us to be ranked 2nd for the country-level subtask and 1st in the province-level subtask. 


Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification

This paper presents the ArabicProcessors team’s deep learning system designed for the NADI 2020 Subtask 1 (country-level dialect identification) and Subtask 2 (province-level dialect identification). We used Arabic-Bert in combination with data augmentation and ensembling methods. Unlabeled data provided by task organizers (10 Million tweets) was split into multiple subparts, to which we applied semi-supervised learning method, and finally ran a specific ensembling process on the resulting models. This system ranked 3rd in Subtask 1 with 23.26% F1-score and 2nd in Subtask 2 with 5.75% F1-score.

Arabic dialect identification: 
An Arabic-BERT model with data augmentation and ensembling strategy

We describe Faheem (adj. of understand), our submission to NADI (Nuanced
Arabic Dialect Identification) shared task. With so many Arabic dialects
being understudied due to the scarcity of the resources, the objective is to
identify the Arabic dialect used in the tweet, at the country-level. We propose
a machine learning approach where we utilize word-level n-gram (n=1-3) and
tf-idf features and feed them to six different classifiers. We train the system
using a data set of 21,000 tweets covering 21 Arab countries. Our top
performing classifiers are: Logistic Regression, Support Vector Machines, and
Multinomial Naive Bayes (MNB). We achieved our best result of macro-F1=
0.151 using the MNB classifier.

Modeling causality in financial texts using multi-headed transformers

Downloads

Next from COLING 2020

SUMSUM@FNS-2020 Shared Task

Similar lecture

Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings

Modeling causality in financial texts using multi-headed transformers

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

SUMSUM@FNS-2020 Shared Task

Similar lecture

Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings

Downloads