United States

By saying &quot;Maria is tall&quot;, a human speaker typically implies that Maria is evaluatively tall from the speaker’s perspective. However, by using a different construction &quot;Maria is taller than Sophie&quot;, we cannot infer from Maria and Sophie’s relative heights that Maria is evaluatively tall because it is possible for Maria to be taller than Sophie in a context in which they both count as short. Can pre-trained language models (LMs) “understand” evaulativity (EVAL) inference? To what extent can they discern the EVAL salience of different constructions in a conversation? Will it help LMs’ implicitness performance if we give LMs a persona such as chill, social, and pragmatically skilled? Our study provides an approach to probing LMs’ interpretation of EVAL inference by incorporating insights from experimental pragmatics and sociolinguistics. We find that with the appropriate prompt, LMs can succeed in some pragmatic level language understanding tasks. Our study suggests that socio-pragmatics methodology can shed light on the challenging questions in NLP.

NAACL 2022

Pre-trained Language Models’ Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

socio-pragmatics

degree semantics

pre-trained language models

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

By saying "Maria is tall", a human speaker typically implies that Maria is evaluatively tall from the speaker’s perspective. However, by using a different construction "Maria is taller than Sophie", we cannot infer from Maria and Sophie’s relative heights that Maria is evaluatively tall because it is possible for Maria to be taller than Sophie in a context in which they both count as short. Can pre-trained language models (LMs) “understand” evaulativity (EVAL) inference? To what extent can they discern the EVAL salience of different constructions in a conversation? Will it help LMs’ implicitness performance if we give LMs a persona such as chill, social, and pragmatically skilled? Our study provides an approach to probing LMs’ interpretation of EVAL inference by incorporating insights from experimental pragmatics and sociolinguistics. We find that with the appropriate prompt, LMs can succeed in some pragmatic level language understanding tasks. Our study suggests that socio-pragmatics methodology can shed light on the challenging questions in NLP.

workshop paper

An intelligent system is expected to perform reasonable inferences, accounting for both the literal meaning of a word and the meanings a word can acquire in different contexts. A specific kind of inference concerns the connective and, which in some cases gives rise to a temporal succession or causal interpretation in contrast with the logic, commutative one (Levinson, 2000). In this work, we investigate the phenomenon by creating a new dataset for evaluating the interpretation of and by NLI systems, which we use to test three Transformer-based models. Our results show that all systems generalize patterns that are consistent with both the logical and the pragmatic interpretation, perform inferences that are inconsistent with each other, and show clear divergences with both theoretical accounts and humans’ behavior.

Pragmatic and Logical Inferences in NLI Systems: The Case of Conjunction Buttressing

Prior studies have raised concerns over specificity issues in clinical advice. Lacking specificity — explicitly discussed detailed information — may affect the quality and implementation of clinical advice in medical practice. In this study, we developed and validated a fine-grained annotation schema to describe differ- ent aspects of specificity in clinical advice extracted from the medical research literature. We also presented our initial annotation effort and discussed future directions towards an NLP- based specificity analysis tool for summarizing and verifying the details in clinical advice.


"Devils are in the Details'' — Annotating Specificity of Clinical Advice from Medical Literature

This paper presents a linguistically driven proof of concept for finding potentially euphemistic terms, or PETs. Acknowledging that PETs tend to be commonly used expressions for a certain range of sensitive topics, we make use of distributional similarities to select and filter phrase candidates from a sentence and rank them using a set of simple sentiment-based metrics. We present the results of our approach tested on a corpus of sentences containing euphemisms, demonstrating its efficacy for detecting single and multi-word PETs from a broad range of topics. We also discuss future potential for sentiment-based methods on this task.

Searching for PETs: Using Distributional and Sentiment-Based Methods to Find Potentially Euphemistic Terms

We report results of experiments using BART (Lewis et al., 2019) and the Penn Discourse Tree Bank (Webber et al., 2019) (PDTB) to generate texts with correctly realized discourse relations. We address a question left open by previous research (Yung et al., 2021; Ko and Li, 2020) concerning whether conditioning the model on the intended discourse relation—which corresponds to adding explicit discourse relation information into the input to the model—improves its performance. Our results suggest that including discourse relation information in the input of the model significantly improves the consistency with which it produces a correctly realized discourse relation in the output. We exemplify and argue for the integration of metrics inspired by hypotheses in psycholinguistics into our evaluation

Generating Discourse Connectives with Pre-trained Language Models: Discourse Relations Help Yet Again

Semantic Role Labeling (SRL) has long been referred to as the task of detecting who did what to whom, where, when, and how? Yet most research efforts looked for answers to these questions only at pre-defined syntactical positions in the sentence, disregarding what language users typically understand from discourse and wider document contexts. These implicit arguments are a naturally occurring phenomenon that can be widely beneficial for many natural language understanding challenges. In this work, we propose a method for crowdsourcing implicit semantic arguments on a large scale. To further explore the conceptualization of such arguments by crowd workers, we let them annotate questions and clauses for a given predicate and all its roles. We hope that this dataset will help overcome the data bottleneck for implicit SRL and will allow machines to perform frame semantics on the discourse level.

Looking Beyond Syntax: Detecting Implicit Semantic Arguments

Despite significant progress in Vision-and-Language Navigation (VLN) over the years, existing agents are evaluated on fine-grained instructions that are neither practical nor efficient. In this study, we consider underspecification as a ubiquitous and important phenomenon to bridge the gap between curated instructions and natural human instructions in reality. We introduce a new setting, namely Underspecified vision-and-Language Navigation (ULN). We verify that our instructions are more practical and efficient from the users' perspective. We propose a goal-aware agent with an Exploitation-to-Exploration (E2E) module that estimates step-wise uncertainty. Experimental results show that existing VLN models are still brittle to underspecification. Empirically, our method outperforms the baseline by a large margin in success rate while maintaining a comparable performance on R2R.

ULN: Towards Underspecified Vision-and-Language Navigation

Some of the information that is conveyed in text is not explicitly stated, but instead, is inferred by the reader. While current state-of-the-art language models are very good at recovering explicit information, they still struggle with inferring implicit information from text, and as a result, that information is lost and NLU quality suffers. One such implicit information phenomenon is gapping, a linguistic phenomenon in coordination structures containing VPs, in which a VP that appears in the initial clause is elided from a non-initial clause and can be inferred from the context of the sentence.

Bridging the Gap: Recovering Elided VPs in Coordination Structures

To tackle the rising phenomenon of hate speech, efforts have been made towards data curation and analysis. When it comes to analysis of bias, previous work has focused predominantly on race. In our work, we further investigate bias in hate speech datasets along racial, gender and intersectional axes. We identify strong bias against African American English (AAE), masculine and AAE+Masculine tweets, which are annotated as disproportionately more hateful and offensive than from other demographics. We provide evidence that BERT-based models propagate this bias and show that balancing the training data for these protected attributes can lead to fairer models with regards to gender, but not race.

Analyzing Hate Speech Data along Racial, Gender and Intersectional Axes


The representations in large language models contain multiple types of gender information. We focus on two types of such signals in English texts: factual gender information, which is a grammatical or semantic property, and gender bias, which is the correlation between a word and specific gender. We can disentangle
the model’s embeddings and identify components encoding both types of information with probing. We aim to diminish the stereotypical bias in the representations while preserving the factual gender signal. Our filtering method
shows that it is possible to decrease the bias of gender-neutral profession names without significant deterioration of language modeling capabilities. The findings can be applied to language generation to mitigate reliance on stereotypes while preserving gender agreement in coreferences

Don’t Forget About Pronouns: Removing Gender Bias in Language Models Without Losing Factual Gender Information

We investigate whether it’s feasible to de-gender resumes to mitigate potential gender bias in algorithmic resume screening. Using a corpus of 709k resumes from IT firms, we first train a series of models to classify the self­ reported gender of the applicant, thereby measuring the extent and nature of gendered information encoded in resumes. We then conduct a series of gender obfuscation experiments, where we iteratively remove gendered information from resumes while preserving the job­-relevant content. Finally, we investigate the trade­off between gender obfuscation and the performance of a resume screening algorithm. We find three main results: (1) there is a significant amount of gendered information in resumes. (2) lexicon-­based gender obfuscation method can reduce the amount of gendered information to a large extent. However, after a certain point, the performance of the resume screening algorithm starts suffering. (3) general-­purpose gender debiasing methods for NLP models such as removing gender subspace from embeddings are not effective in obfuscating gender.


Downloads

Next from NAACL 2022

Pragmatic and Logical Inferences in NLI Systems: The Case of Conjunction Buttressing

Similar lecture

A Data Cartography based MixUp for Pre-trained Language Models