Ireland

We study extractive question answering task and systematically compare 9 models on MRQA benchmarks which have 12 QA datasets. In this presentation we highlight three interesting findings, the first one is that language models based on encoder-decoder architecture (e.g. BART and T5) can be used as extractive readers, while previous work used to use encoder-only architecture model (e.g. RoBERTa). The second finding is that in general extractive readers has better generalization capacity than generative readers. And the last one is that the generative reader is better than extractive readers if the context is long, and the extractive readers are better when the context is short.

ACL 2022

Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering

generative model.

extractive model

question answering

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

workshop paper

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. Here we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval: recurring span retrieval. We use recurring spans across passages in a document to create pseudo examples for contrastive learning. The resulting model -- Spider -- performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, when combined with BM25, it is often competitive with in-domain DPR models, which are trained on tens of thousands of examples.

Learning to Retrieve Passages without Supervision

Introduction to our recent work "Hyperlink-induced Pre-training
for Passage Retrieval in Open-domain Question Answering", a pre-training method to learn question-passage relevance induced by hyperlink-based structures within Web documents for OpenQA passage retrieval.

Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering

Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs.

Towards Continual Knowledge Learning of Language Models

We present a new cross-lingual information retrieval (CLIR) system trained using multi-stage knowledge distillation (KD). The teacher relies on a highly effective but expensive two-stage process consisting of query translation and monolingual IR, while the student executes a single CLIR step. We teach the student powerful multilingual encoding as well as CLIR by optimizing two corresponding KD objectives. Learning useful non-English representations from an English-only retriever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual language model. In both in-domain and zero-shot evaluation, the proposed method demonstrates far superior accuracy over direct fine-tuning with labeled CLIR data. One of our systems is also the current best single-model system on the XOR-TyDi leaderboard.

Learning Cross-Lingual IR from an English Retriever

We consider the problem of pretraining a two-stage open-domain question answering (QA) system (retriever + reader) with strong transfer capabilities. The key challenge is how to construct a large amount of high-quality question-answer-context triplets without task-specific annotations. Specifically, the triplets should align well with downstream tasks by: (i) covering a wide range of domains (for open-domain applications), (ii) linking a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identifying the correct answer in the context (for training the reader). Previous pretraining approaches generally fall short of one or more of these requirements. In this work, we automatically construct a large-scale corpus that meets all three criteria by consulting millions of references cited within Wikipedia. The well-aligned pretraining signals benefit both the retriever and the reader significantly. Our pretrained retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our pretrained reader, the entire system improves by up to 4% in exact match.

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References

Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking.
Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the recall@100. Furthermore, when a few thousands random examples are available, fine-tuning our model on these examples significantly improves performance over BM25. The contrastive learning method used in this work also provides improvements on the BEIR benchmark when used as pre-training before fine-tuning on the MS~MARCO dataset. Finally, large annotated information retrieval datasets are generally not available in languages other than English. The development of alternatives to supervised training for information retrieval is therefore all the more crucial from a multilingual perspective. We show that the contrastive learning method used in our work enables to train multilingual retrievers reaching state-of-the-art performance.

Towards Unsupervised Dense Information Retrieval with Contrastive Learning

In this work, we aim to capitalize on the unique few-shot capabilities offered by large-scale language models to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models, which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition language models on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any language model, offering like this a strong baseline. Indeed, we find that language models conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage, alleviates generally decreased performance of smaller few-shot language models. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift the attention towards finding more effective ways to use models, including but not limited to better prompting or increasing inference-time compute.

Internet-augmented language models through few-shot prompting for open-domain question answering

When a large language model produces undesirable output after deployment, recent semi-parametric methods have relied on a growing database of how a similar error was resolved in the past. This paper highlights current issues in retrieving similar errors from the database, and addresses these issues with a novel generation inspired retrieval technique, called GUD-IR (Generated UnDerstanding for explainable IR). On two existing datasets: SocialChem101 (ethical reasoning) and OBQA (science QA), we found that off-the-shelf modern retrieval methods fail to distinguish lexically dissimilar but semantically similar erroneous situations. On both the datasets, GUD-IR provides nearly 10% absolute precision gains in retrieval quality over strong baselines. Our approach is a step towards deploying semiparametric methods as a low-cost utility enhancement for very large pre-trained LMs.

GUD-IR: Generative Retrieval for Semiparametric Models

Despite the success of large pre-trained language models (LMs) such as Codex, they show below-par performance on the larger and more complicated programming related questions. We show that LMs benefit from the summarized version of complicated questions. Our findings show that superfluous information often present in problem description such as human characters, background stories, names (which are included to help humans in understanding a task) does not help models in understanding a task. To this extent, we create a meta-dataset from the frequently used APPS dataset for the program synthesis task. Our meta-dataset consists of human and synthesized summary of the long and complicated programming questions. Experimental results on Codex show that our proposed approach outperforms baseline by 8.13% on an average in terms of strict accuracy. Our analysis shows that summary significantly improve performance for introductory (9.86%) and interview (11.48%) related programming questions. However, it shows improvement by a small margin (~ 2) for competitive programming questions, implying the scope for future research direction.

Less is More: Summary of Long Instructions is Better for Program Synthesis

Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state of the art task specific models. Conventional approaches to improve model performance via creating large datasets with lots of task instances or architectural/training changes in model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augumentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that these significantly improve model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to ~200 data samples on average across tasks.

Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering

Downloads

Next from ACL 2022

Learning to Retrieve Passages without Supervision

Similar lecture

Clickbait Spoiling via Question Answering and Passage Retrieval

Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2022

Learning to Retrieve Passages without Supervision

Similar lecture

Clickbait Spoiling via Question Answering and Passage Retrieval

Downloads