Thailand

Data-to-text generation systems are trained on large datasets, such as WebNLG, RotoWire, E2E or DART. Beyond traditional token-overlap evaluation metrics (BLEU or METEOR), a key concern faced by recent generators is to control 
the factuality of the generated text with respect to the input data specification. We report on our experience when developing an automatic factuality evaluation system for data-to-text generation that we are testing on WebNLG and E2E data. We aim to prepare gold data annotated manually to identify cases where the text communicates more information than is warranted based on the input data (extra) or fails to communicate data that is part of the input (missing). While analyzing reference (data, text) samples, we encountered a range of systematic uncertainties that are related to cases on implicit phenomena in text, and the nature of non-linguistic knowledge we expect to be involved when assessing factuality. 
We derive from our experience a set of evaluation guidelines to reach high inter-annotator agreement on such cases

ACL-IJCNLP 2021

Evaluation Guidlines to Deal with Implicit Phenomena to Assess Factuality in Data-to-Text Generation

human annotations

factuality

data-to-text

evaluation

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

workshop paper

In this paper, we focus on the problem of citing sentence generation, which entails generating a short text to capture the salient information in a cited paper and the connection between the citing and cited paper. We present BACO, a BAckground knowledge- and COntent-based framework for citing sentence generation, which considers two types of information: (1) background knowledge by leveraging structural information from a citation network; and (2) content, which represents in-depth information about what to cite and why to cite. First, a citation network is encoded to provide background knowledge. Second, we apply salience estimation to identify what to cite by estimating the importance of sentences in the cited paper. During the decoding stage, both types of information are combined to facilitate the text generation, and then we conduct a joint training for the generator and citation function classification to make the model aware of why to cite. Our experimental results show that our framework outperforms comparative baselines.

BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation

Current dialogue summarization systems usually encode the text with a number of general semantic features (e.g., keywords and topics) to gain more powerful dialogue modeling capabilities. However, these features are obtained via open-domain toolkits that are dialog-agnostic or heavily relied on human annotations. In this paper, we show how DialoGPT, a pre-trained model for conversational response generation, can be developed as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAMSum and AMI, and employ pre-trained and non pre-trained models as our summarizers. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the-art performance on the SAMSum dataset.

Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

The growth of online consumer health questions has led to the necessity for reliable and
accurate question answering systems. A recent study showed that manual summarization of consumer health questions brings significant improvement in retrieving relevant answers. However, the automatic summarization
of long questions is a challenging task due to
the lack of training data and the complexity of
the related subtasks, such as the question focus and type recognition. In this paper, we introduce a reinforcement learning-based framework for abstractive question summarization.
We propose two novel rewards obtained from
the downstream tasks of (i) question-type identification and (ii) question-focus recognition
to regularize the question generation model.
These rewards ensure the generation of semantically valid questions and encourage the
inclusion of key medical entities/foci in the
question summary. We evaluated our proposed method on two benchmark datasets and
achieved higher performance over state-of-the-art models. The manual evaluation of the
summaries reveals that the generated questions are more diverse and have fewer factual
inconsistencies than the baseline summaries.
The source code is available here: https://github.com/shwetanlp/CHQ-Summ.

Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards

Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. 
We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. 
Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. 
However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation.
Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development. Our code and annotated data is publicly available at https://github.com/AkariAsai/unanswerable_qa.

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Relation linking is a crucial component of Knowledge Base Question Answering systems. Existing systems use a wide variety of heuristics, or ensembles of multiple systems, heavily relying on the surface question text. However, the explicit semantic parse of the question is a rich source of relation information that is not taken advantage of. We propose a simple transformer-based neural model for relation linking that leverages the AMR semantic parse of a sentence. Our system significantly outperforms the state-of-the-art on 4 popular benchmark datasets. These are based on either DBpedia or Wikidata, demonstrating that our approach is effective across KGs.

A Semantics-aware Transformer Model of Relation Linking for Knowledge Base Question Answering

Users of medical question answering systems often submit long and detailed questions, making it hard to achieve high recall in answer retrieval. To alleviate this problem, we propose a novel Multi-Task Learning (MTL) method with data augmentation for medical question understanding. We first establish an equivalence between the tasks of question summarization and Recognizing Question Entailment (RQE) using their definitions in the medical domain. Based on this equivalence, we propose a data augmentation algorithm to use just one dataset to optimize for both tasks, with a weighted MTL loss. We introduce gradually soft parameter-sharing: a constraint for decoder parameters to be close, that is gradually loosened as we move to the highest layer. We show through ablation studies that our proposed novelties improve performance. Our method outperforms existing MTL methods across 4 datasets of medical question pairs, in ROUGE scores, RQE accuracy and human evaluation. Finally, we show that our method fares better than single-task learning under 4 low-resource settings.

A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding

While aggregate performance metrics can generate valuable insights at a large scale, their dominance means more complex and nuanced language phenomena, such as vagueness, may be overlooked. Focusing on vague terms (e.g. sunny, cloudy, young, etc.) we inspect the behavior of visually grounded and text-only models, finding systematic divergences from human judgments even when a model’s overall performance is high. To help explain this disparity, we identify two assumptions made by the datasets and models examined and, guided by the philosophy of vagueness, isolate cases where they do not hold.

Human-Model Divergence in the Handling of Vagueness

We conclude that the implicit UCCA parser does not address NFHs consistently, which could result either from inconsistent annotation, insufﬁcient training data or a modelling limitation. We consider this phenomenon important, as it is pervasive in text and critical for correct inference. Careful design and ﬁne-grained annotation of NFHs in Meaning Representations would beneﬁt downstream tasks such as machine translation, natural language inference and question answering, particularly when they require numeric reasoning, as recovering and categorizing them. We are investigating the treatment of this phenomenon by other meaning representations, such as AMR. We encourage researchers in meaning representations, and computational linguistics in general, to address this phenomenon in future research.

Meaning Representation of Numeric Fused-Heads in UCCA

Transformer-based language models (LMs) have shown strong performance on an array of natural language understanding tasks. However, the question of how these models react to implicit meaning has been largely unexplored. We investigate this using the complement coercion phenomenon, which involves sentences like "The student finished the book about sailing" where the action "read" is implicit. We compare LMs' surprisal estimates at various critical sentence regions in sentences with and without implicit meaning. Effects associated with recovering implicit meaning were found at a critical region other than where sentences minimally differ. We then use follow-up experiments to factor out potential confounds, revealing different perspectives that offer a richer and more accurate picture.

Transformer-based language models and complement coercion: Experimental studies

Noun phrases (NPs) in text are linked via discourse relations. One such well-known relation is bridging , in which an NP indirectly refers to an entity mentioned elsewhere in the text. 
Our aim is to construct a high-quality large-scale manually annotated corpus, that covers bridging relations. To this end, we generalize and simplify the definition of bridging relations, to make them amenable to crowdsourcing. In our updated definition NPs are connected to co-reference clusters via natural-language prepositions.
The result is an annotation scheme in which each NP is potentially linked to other NPs via coreference links and/or via a preposition. While the collection process is ongoing, we already have 2790 documents annotated (each including 3 paragraphs), covering 504,250 bridging links and 14,958 coreference clusters (not including singletons).



Downloads

Next from ACL-IJCNLP 2021

BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation

Similar lecture

GO FIGURE: A Meta Evaluation of Factuality in Summarization

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL-IJCNLP 2021

BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation

Similar lecture

GO FIGURE: A Meta Evaluation of Factuality in Summarization

Downloads