United States

Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages with entity representations and show the model consistently outperforms word-based pretrained models in various cross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a multilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual knowledge more likely than using only word representations.

NAACL 2022

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

multilingual language model

entity

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

workshop paper

We describe our two-stage system for the Multilingual Information Access (MIA) 2022 Shared Task on Cross-Lingual Open-Retrieval Question Answering. The first stage consists of multilingual passage retrieval with a hybrid dense and sparse retrieval strategy. The second stage consists of a reader which outputs the answer from the top passages returned by the first stage. We show the efficacy of using entity representations, sparse retrieval signals to help dense retrieval, and Fusion-in-Decoder. On the development set, we obtain 43.46 F1 on XOR-TyDi QA and 21.99 F1 on MKQA, for an average F1 score of 32.73. On the test set, we obtain 40.93 F1 on XOR-TyDi QA and 22.29 F1 on MKQA, for an average F1 score of 31.61. We improve over the official baseline by over 4 F1 points on both the development and test sets.

Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

Recent advances in NLP often stem from large transformer-based pre-trained models, which rapidly grow in size and use more and more training data. Such models are often released to the public so that end users can fine-tune them on a task dataset. While it is common to treat pre-training data as public, it may still contain personally identifiable information (PII), such as names, phone numbers, and copyrighted material. Recent findings show that the capacity of these models allows them to memorize parts of the training data, and suggest differentially private (DP) training as a potential mitigation. While there is recent work on DP fine-tuning of NLP models, the effects of DP pre-training are less well understood: it is not clear how downstream performance is affected by DP pre-training, and whether DP pre-training mitigates some of the memorization concerns. We focus on T5 and show that by using recent advances in JAX and XLA we can train models with DP that do not suffer a large drop in pre-training utility, nor in training speed, and can still be fine-tuned to high accuracies on downstream tasks (e.g. GLUE). Moreover, we show that T5’s span corruption is a good defense against data memorization.

"Training Text-to-Text Transformers with Privacy Guarantees"

By saying "Maria is tall", a human speaker typically implies that Maria is evaluatively tall from the speaker’s perspective. However, by using a different construction "Maria is taller than Sophie", we cannot infer from Maria and Sophie’s relative heights that Maria is evaluatively tall because it is possible for Maria to be taller than Sophie in a context in which they both count as short. Can pre-trained language models (LMs) “understand” evaulativity (EVAL) inference? To what extent can they discern the EVAL salience of different constructions in a conversation? Will it help LMs’ implicitness performance if we give LMs a persona such as chill, social, and pragmatically skilled? Our study provides an approach to probing LMs’ interpretation of EVAL inference by incorporating insights from experimental pragmatics and sociolinguistics. We find that with the appropriate prompt, LMs can succeed in some pragmatic level language understanding tasks. Our study suggests that socio-pragmatics methodology can shed light on the challenging questions in NLP.

Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

An intelligent system is expected to perform reasonable inferences, accounting for both the literal meaning of a word and the meanings a word can acquire in different contexts. A specific kind of inference concerns the connective and, which in some cases gives rise to a temporal succession or causal interpretation in contrast with the logic, commutative one (Levinson, 2000). In this work, we investigate the phenomenon by creating a new dataset for evaluating the interpretation of and by NLI systems, which we use to test three Transformer-based models. Our results show that all systems generalize patterns that are consistent with both the logical and the pragmatic interpretation, perform inferences that are inconsistent with each other, and show clear divergences with both theoretical accounts and humans’ behavior.

Pragmatic and Logical Inferences in NLI Systems: The Case of Conjunction Buttressing

Prior studies have raised concerns over specificity issues in clinical advice. Lacking specificity — explicitly discussed detailed information — may affect the quality and implementation of clinical advice in medical practice. In this study, we developed and validated a fine-grained annotation schema to describe differ- ent aspects of specificity in clinical advice extracted from the medical research literature. We also presented our initial annotation effort and discussed future directions towards an NLP- based specificity analysis tool for summarizing and verifying the details in clinical advice.


"Devils are in the Details'' — Annotating Specificity of Clinical Advice from Medical Literature

This paper presents a linguistically driven proof of concept for finding potentially euphemistic terms, or PETs. Acknowledging that PETs tend to be commonly used expressions for a certain range of sensitive topics, we make use of distributional similarities to select and filter phrase candidates from a sentence and rank them using a set of simple sentiment-based metrics. We present the results of our approach tested on a corpus of sentences containing euphemisms, demonstrating its efficacy for detecting single and multi-word PETs from a broad range of topics. We also discuss future potential for sentiment-based methods on this task.

Searching for PETs: Using Distributional and Sentiment-Based Methods to Find Potentially Euphemistic Terms

We report results of experiments using BART (Lewis et al., 2019) and the Penn Discourse Tree Bank (Webber et al., 2019) (PDTB) to generate texts with correctly realized discourse relations. We address a question left open by previous research (Yung et al., 2021; Ko and Li, 2020) concerning whether conditioning the model on the intended discourse relation—which corresponds to adding explicit discourse relation information into the input to the model—improves its performance. Our results suggest that including discourse relation information in the input of the model significantly improves the consistency with which it produces a correctly realized discourse relation in the output. We exemplify and argue for the integration of metrics inspired by hypotheses in psycholinguistics into our evaluation

Generating Discourse Connectives with Pre-trained Language Models: Discourse Relations Help Yet Again

Semantic Role Labeling (SRL) has long been referred to as the task of detecting who did what to whom, where, when, and how? Yet most research efforts looked for answers to these questions only at pre-defined syntactical positions in the sentence, disregarding what language users typically understand from discourse and wider document contexts. These implicit arguments are a naturally occurring phenomenon that can be widely beneficial for many natural language understanding challenges. In this work, we propose a method for crowdsourcing implicit semantic arguments on a large scale. To further explore the conceptualization of such arguments by crowd workers, we let them annotate questions and clauses for a given predicate and all its roles. We hope that this dataset will help overcome the data bottleneck for implicit SRL and will allow machines to perform frame semantics on the discourse level.

Looking Beyond Syntax: Detecting Implicit Semantic Arguments

Despite significant progress in Vision-and-Language Navigation (VLN) over the years, existing agents are evaluated on fine-grained instructions that are neither practical nor efficient. In this study, we consider underspecification as a ubiquitous and important phenomenon to bridge the gap between curated instructions and natural human instructions in reality. We introduce a new setting, namely Underspecified vision-and-Language Navigation (ULN). We verify that our instructions are more practical and efficient from the users' perspective. We propose a goal-aware agent with an Exploitation-to-Exploration (E2E) module that estimates step-wise uncertainty. Experimental results show that existing VLN models are still brittle to underspecification. Empirically, our method outperforms the baseline by a large margin in success rate while maintaining a comparable performance on R2R.

ULN: Towards Underspecified Vision-and-Language Navigation

Some of the information that is conveyed in text is not explicitly stated, but instead, is inferred by the reader. While current state-of-the-art language models are very good at recovering explicit information, they still struggle with inferring implicit information from text, and as a result, that information is lost and NLU quality suffers. One such implicit information phenomenon is gapping, a linguistic phenomenon in coordination structures containing VPs, in which a VP that appears in the initial clause is elided from a non-initial clause and can be inferred from the context of the sentence.

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Downloads

Next from NAACL 2022

Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

Similar lecture

A system for named entity recognition using data augmentation, transformer-based sequence labeling model, and EnsembleCRF

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAACL 2022

Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering

Similar lecture

A system for named entity recognition using data augmentation, transformer-based sequence labeling model, and EnsembleCRF

Downloads