United States

People frequently interact with information retrieval (IR) systems, however, IR models exhibit biases and discrimination towards various demographics. The in-processing fair ranking methods provide a trade-offs between accuracy and fairness through adding a fairness-related regularization term in the loss function. However, there haven’t been intuitive objective functions that depend on the click probability and user engagement to directly optimize towards this. In this work, we propose the In-Batch Balancing Regularization (IBBR) to mitigate the ranking disparity among subgroups. In particular, we develop a differentiable normed Pairwise Ranking Fairness (nPRF) and leverage the T-statistics on top of nPRF over sub-groups as a regularization to improve fairness. Empirical results with the BERT-based neural rankers on the MS MARCO Passage Retrieval dataset with the human-annotated non-gendered queries benchmark (Rekabsaz and Schedl, 2020) show that our IBBR method with nPRF achieves significantly less bias with minimal degradation in ranking performance compared with the baseline

NAACL 2022

Debiasing Neural Retrieval via In-batch Balancing Regularization

retrieval system

regularization

fairness

## Welcome to NAACL 2022!
The Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) is taking place July 10-15, 2022 as a hybrid event, in Seattle, WA and online. We are happy to welcome all of you to conference! 
 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, keynote presentations, a plenary panel on the place of linguistics and symbolic structures in NLP, and an Industrial Track panel on Careers in NLP.
 
Posters (including Findings of NAACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop and will have an oral session and a poster session.
 
The program also features 6 Tutorials and 26 Workshops.
 
 We wish you a wonderful conference!
 
[The NAACL 2022 Organizing Committee](https://2022.naacl.org/committees/organization/)

2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics

workshop paper

Have you ever wondered where one can find gender biases? They hide within language models (LM) and we will show how to find them! Our new framework can identify regions of a LM that contain bias – it exploits movement pruning to do so. We also show the utility of our framework by conducting gender bias study in BERT. Based on our observations, show an improvement to an existing debiasing method.

Gender Biases and Where to Find Them

Word embeddings learned from massive text collections have demonstrated significant levels of discriminative biases.
However, debiasing on the Chinese language, one of the most spoken languages, has been less explored.
Meanwhile, existing literature relies on manually created supplementary data, which is time- and energy-consuming.
In this work, we propose the first Chinese Gender-neutral word Embedding model (CGE) based on Word2vec, which learns gender-neutral word embeddings without any labeled data.
Concretely, CGE utilizes and emphasizes the rich feminine and masculine information contained in radicals, \ie a kind of component in Chinese characters, during the training procedure.
This consequently alleviates discriminative gender biases.
Experimental results show that our unsupervised method outperforms the state-of-the-art supervised debiased word embedding models without sacrificing the functionality of the embedding model.

Unsupervised Mitigation of Gender Bias by Character Components: A Case Study of Chinese Word Embedding

Pre-trained word embedding models are easily distributed and applied, as they alleviatevusers from the effort to train models themselves. With widely distributed models, it is important to ensure that they do not exhibit undesired behaviour, such as biases against population groups. For this purpose, we carry outvan empirical study on evaluating the bias of 15 publicly available, pre-trained word embeddings model based on three training algorithms (GloVe, word2vec, and fastText) with
regard to four bias metrics (WEAT, SEMBIAS, DIRECT BIAS, and ECT). The choice of word embedding models and bias metrics is motivated by a literature survey over 37 publications which quantified bias on pre-trained word embeddings. Our results indicate that fastText is the least biased model (in 8 out of 12 cases) and small vector lengths lead to a higher bias.

An Empirical Study on the Fairness of Pre-trained Word Embeddings

Considerable efforts to measure and mitigate gender bias in recent years have led to the introduction of an abundance of tasks, datasets, and metrics used in this vein. 
In this position paper, we assess the current paradigm of gender bias evaluation and identify several flaws in it. First, we highlight the importance of extrinsic bias metrics that measure how a model's performance on some task is affected by gender, as opposed to intrinsic evaluations of model representations, which are less strongly connected to specific harms to people interacting with systems. We find that only a few extrinsic metrics are measured in most studies, although more can be measured. Second, we find that datasets and metrics are often coupled, and discuss how their coupling hinders the ability to obtain reliable conclusions, and how one may decouple them. We then investigate how the choice of the dataset and its composition, as well as the choice of the metric, affect bias measurement, finding significant variations across each of them. Finally, we propose several guidelines for more reliable gender bias evaluation.

Choose Your Lenses: Flaws in Gender Bias Evaluation

The growing capability and availability of generative
language models has enabled a wide
range of new downstream tasks. Academic research
has identified, quantified and mitigated
biases present in language models but is rarely
tailored to downstream tasks where wider impact
on individuals and society can be felt. In
this work, we leverage one popular generative
language model, GPT-3, with the goal of writing
unbiased and realistic job advertisements.
We first assess the bias and realism of zero-shot
generated advertisements and compare them to
real-world advertisements. We then evaluate
prompt-engineering and fine-tuning as debiasing
methods. We find that prompt-engineering
with diversity-encouraging prompts gives no
significant improvement to bias, nor realism.
Conversely, fine-tuning, especially on unbiased
real advertisements, can improve realism and
reduce bias.

Looking for a Handsome Carpenter! Debiasing GPT-3 Job Advertisements

Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model.
However, after knowledge distillation, it was found that the smaller model is more biased by gender compared to the source large model.
This paper studies what causes gender bias to increase after the knowledge distillation process.
Moreover, we suggest applying a variant of the mixup on knowledge distillation, which is used to increase generalizability during the distillation process, not for augmentation.
By doing so, we can significantly reduce the gender bias amplification after knowledge distillation.
We also conduct an experiment on the GLUE benchmark to demonstrate that even if the mixup is applied, it does not have a significant adverse effect on the model's performance.

Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT

Sentiment analysis is increasingly viewed as a vital task both from an academic and a commercial standpoint. In this paper, we focus on the structured sentiment analysis task that is released on SemEval-2022 Task 10. The task aims to extract the structured sentiment information (e.g., holder, target, expression and sentiment polarity) in a text. We propose a simple and unified model for both the monolingual and crosslingual structured sentiment analysis tasks. We translate this task into an event extraction task by regrading the expression as the trigger word and the other elements as the arguments of the event. Particularly, we first extract the expression by judging its start and end indices. Then, to consider the expression, we design a conditional layer normalization algorithm to extract the holder and target based on the extracted expression. Finally, we infer the sentiment polarity based on the extracted structured information. Pre-trained language models are utilized to obtain the text representation. We conduct the experiments on seven datasets in five languages. It attracted 233 submissions in monolingual subtask and crosslingual subtask from 32 teams. Finally, we obtain the top 5 place on crosslingual tasks.

ECNU_ICA at SemEval-2022 Task 10: A Simple and Unified Model for Monolingual and Crosslingual Structured Sentiment Analysis

This presentation presents our submission to task 10, Structured Sentiment Analysis of the SemEval 2022 competition. The task aims to extract all elements of the fine-grained sentiment in a text. We cast structured sentiment analysis to the prediction of the sentiment graphs following (Barnes et al., 2021), where nodes are spans of sentiment holders, targets and expressions, and directed edges denote the relation types be- tween them. Our approach closely follows that of semantic dependency parsing (Dozat and Manning, 2018). The difference is that we use pre-trained language models (e.g., BERT and RoBERTa) as text encoder to solve the prob- lem of limited annotated data. Additionally, we make improvements on the computation of cross attention and present the suffix mask- ing technique to make further performance im- provement. Substantially, our model achieved the Top-1 average Sentiment Graph F1 score on seven datasets in five different languages in the monolingual subtask.

ZHIXIAOBAO at SemEval-2022 Task 10: Apporoaching Structured Sentiment with Graph Parsing

Large scale pre-training models have been widely used in named entity recognition (NER) task. However, model ensemble through param- eter averaging or voting can not full play to the differentiation advantages of different models, expecially in open domain. In this paper, we describes our NER system in the SemEval 2022 task11: MultiCoNER. We proposed an effec- tive system to adaptively ensemble pre-trained language models by a Transformer layer. By assigning different weights to each model for different inputs, we adopted the Transformer layer to effectively integrate the advantages of diverse models. Experimental results show that our method achieves superior performances on Farsi and Dutch.

SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models

Patronizing and Condescending Language (PCL) towards vulnerable communities in general media has been shown to have potentially harmful effects. Due to its subtlety and the good intentions behind its use, the audience is not aware of the language's toxicity. In this paper, we present our method for the SemEval-2022 Task4 titled "Patronizing and Condescending Language Detection". In Subtask A, a binary classification task, we introduce adversarial training based on Fast Gradient Method (FGM) and employ pre-trained model in a unified architecture. For Subtask B, framed as a multi-label classification problem, we utilize various improved multi-label cross-entropy loss functions and analyze the performance of our method. In the final evaluation, our system achieved official rankings of 17/79 and 16/49 on Subtask A and Subtask B, respectively. In addition, we explore the relationship between PCL and emotional polarity and intensity it contains.

Content not yet available

Debiasing Neural Retrieval via In-batch Balancing Regularization

Downloads

Next from NAACL 2022

Gender Biases and Where to Find Them

Similar lecture

An Empirical Study on the Fairness of Pre-trained Word Embeddings

Content not yet available

Debiasing Neural Retrieval via In-batch Balancing Regularization

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAACL 2022

Gender Biases and Where to Find Them

Similar lecture

An Empirical Study on the Fairness of Pre-trained Word Embeddings

Downloads