Spain

Contextualized word representations have become a driving force in NLP, motivating widespread interest in understanding their capabilities and the mechanisms by which they operate. Particularly intriguing is their ability to identify and encode conceptual abstractions. Past work has probed BERT representations (Devlin et al., 2019) for this competence, finding that BERT can correctly retrieve noun hypernyms in cloze tasks. In this work, we ask the question: do probing studies shed light on systematic knowledge in BERT representations? As a case study, we examine hypernymy knowledge encoded in BERT representations. In particular, we demonstrate through a simple consistency probe that the ability to correctly retrieve hypernyms in cloze tasks, as used in prior work, does not correspond to systematic knowledge in BERT. Our main conclusion is cautionary: even if BERT demonstrates high probing accuracy for a particular competence, it does not necessarily follow that BERT‘understands’ a concept, and it cannot be expected to systematically generalize across applicable contexts.

COLING 2020

On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

technical paper

The 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration tasks participants with regenerating large detailed multi-fact explanations for standardized science exam questions. Given a question, correct answer, and knowledge base, models must rank each fact in the knowledge base such that facts most likely to appear in the explanation are ranked highest. Explanations consist of an average of 6 (and as many as 16) facts that span both core scientific knowledge and world knowledge, and form an explicit lexically-connected "explanation graph" describing how the facts interrelate. In this second iteration of the explanation regeneration shared task, participants are supplied with more than double the training and evaluation data of the first shared task, as well as a knowledge base nearly double in size, both of which expand into more challenging scientific topics that increase the difficulty of the task. In total 10 teams participated, and 5 teams submitted system description papers. The best-performing teams significantly increased state-of-the-art performance both in terms of ranking (mean average precision) and inference speed on this challenge task.

TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration

Languages with large vocabularies pose a challenge for lexical induction especially if they are too low in resources to train sophisticated language models. The situation becomes even more challenging if the language lacks a standard orthography, such that spelling variants cannot be referenced with a standard format for lexical simplification. We propose a simple approach to induce morphological and orthographic forms in low-resourced NLP and prove its potential on a non-standard language as a use case.

Lexical Induction of Orthographic and Morphological forms for Low-Resourced Languages

Conversational voice assistants are rapidly developing from purely transactional systems to social companions with “personality”. UNESCO recently stated that the female and submissive personality of current digital assistants gives rise for concern as it reinforces gender stereotypes. In this work, we present results from a participatory design workshop, wherein we invite people to submit their preferences for what their ideal persona might look like, both in drawings as well as in a multiple choice questionnaire. We find no clear consensus which suggests that one possible solution is to let people configure/personalise their assistants. We then outline a multi-disciplinary project of how we plan to address the complex question of gender and stereotyping in digital assistants.

Conversational Assistants and Gender Stereotypes: Public Perceptions and Desiderata for Voice Personas

Causality represents the most important kind of correlation between events. Extracting causality from text has become a promising hot topic in NLP. However, there is no mature research systems, evaluation rules and datasets for public evaluation. Moreover, there is a lack of unified causal sequence labeling methods, which constitute the key factors that hinder the progress of causality extraction research. We survey the limitations and shortcomings of existing causality
research field comprehensively from the aspects of basic concepts, extraction methods, experimental data, and labeling methods, so as to provide reference for future research on causality extraction. We summarize the existing causality datasets, explore their practicability and extensibility from multiple perspectives. Aiming at the problem of causal sequence labeling, we analyze the existing methods of causal sequence labeling, with a summarizations of its regulation. Multiple candidate causal labeling sequences are put forward according to labeling controversy to explore the optimal labeling method through experiments, and suggestions are provided for selecting labeling method.

A Review of dataset and labeling methods for causality extraction

We present a system for the task of unsupervised lexical change detection. Given a target word and two corpora spanning different periods of time, automatically detects whether the word has lost or gained senses from one corpus to another. Our system employs the temporal referencing method to obtain compatible representations of target words in different periods of time. This is done by concatenating corpora of different periods and performing a temporal referencing of target words i.e., treating occurrences of target words in different periods as two independent tokens. Afterwards, we train word embeddings on the joint corpus and compare the referenced vectors of each target word using cosine similarity. Our submission was ranked 6th among 33teams for subtask 1, obtaining an average accuracy of 0.637, only 0.050 points behind the first ranked system

Temporal Referencing Word Embeddings

As the name implies, contextualized representations of language are typically motivated by their ability to encode context. Which aspects of context are captured by such representations? We introduce an approach to address this question using Representational Similarity Analysis (RSA). As case studies, we investigate the degree to which a verb embedding encodes the verb’s subject, a pronoun embedding encodes the pronoun’s antecedent, and a full-sentence representation encodes the sentence’s head word (as determined by a dependency parse). In all cases, we show that BERT’s contextualized embeddings reflect the linguistic dependency being studied, and that BERT encodes these dependencies to a greater degree than it encodes less linguistically-salient controls. These results demonstrate the ability of our approach to adjudicate between hypotheses about which aspects of context are encoded in representations of language.

Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis

Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to extract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language glosses. Previous work improves the first system but little work has focused on the second. This paper focuses on the translation system and introduces the STMC-Transformer which improves on the current state-of-the-art by over 5 and 7 BLEU respectively on gloss-to-text and video-to-text translation of the PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase of over 16 BLEU.
We also demonstrate the problem in current methods that rely on gloss supervision. The video-to-text translation of our STMC-Transformer outperforms translation of ground truth (GT) glosses. This contradicts previous claims that GT gloss translation acts as an upper bound for SLT performance and reveals that glosses are an inefficient representation of sign language. For future SLT research, we suggest an end-to-end training of the recognition and translation models or using a different sign language annotation scheme.

Better Sign Language Translation with STMC-Transformer

In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. Our trained models will be made publicly available to the research community.

German’s Next Language Model

Emotion-cause pair extraction (ECPE), which aims at simultaneously extracting emotion-cause pairs that express emotions and their corresponding causes in a document, plays a vital role in understanding natural languages. Considering that a cause usually appears around its corresponding emotion, we construct a pair graph and a Pair Graph Convolutional Network (PairGCN) to model dependency relations among local neighborhood candidate pairs. Moreover, in our proposed graph, there are three types of dependency relations and each type of dependency relations has its own way to propagate contextual information. Experiments on a benchmark Chinese emotion-cause pair extraction corpus demonstrate the effectiveness of the proposed model.

End-to-End Emotion-Cause Pair Extraction with Graph Convolutional Network

This paper describes the system designed by the Baidu PGL Team which achieved the first place in the TextGraphs 2020 Shared Task. The task focuses on generating explanations for elementary science questions. Given a question and its corresponding correct answer, we are asked to select the facts that can explain why the answer is correct for that question and answering (QA) from a large knowledge base. To address this problem, we use a pre-trained language model to recall the top-K relevant explanations for each question. Then, we adopt a re-ranking approach based on a pre-trained language model to rank the candidate explanations. To further improve the rankings, we also develop an architecture consisting both powerful pre-trained transformers and GNNs to tackle the multi-hop inference problem. The official evaluation shows that, our system can outperform the second best system by 1.91 points. 

On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

Downloads

Next from COLING 2020

TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration

Similar lecture

Tutorial: Endangered Languages meet Modern NLP

On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration

Similar lecture

Tutorial: Endangered Languages meet Modern NLP

Downloads