Thailand

How does the input segmentation of pretrained language models (PLMs) 
affect their interpretations of complex words?
We present the first study investigating this question,
taking BERT as the example PLM and focusing on its semantic representations
of English derivatives.
We show that PLMs can be interpreted as serial dual-route models, i.e.,
the meanings of complex words are either stored or else need to 
be computed from the subwords,
which implies that maximally meaningful input tokens should
allow for the best generalization on new words. This hypothesis
is confirmed by a series of semantic probing 
tasks on which DelBERT (Derivation leveraging BERT), a model with 
derivational input segmentation, substantially outperforms
BERT with WordPiece segmentation. Our results suggest
that the generalization capabilities of PLMs
could be further improved if a morphologically-informed vocabulary 
of input tokens were used.

IJCNLP-AACL 2021

Superbizarre Is Not Superb: Derivational Morphology Improves BERT&#39;s Interpretation of Complex Words

derivational morphology

input segmentation

pretrained language models

bert

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

technical paper

This paper presents a novel method for nested named entity recognition. As a layered method, our method extends the prior second-best path recognition method by explicitly excluding the influence of the best path. Our method maintains a set of hidden states at each time step and selectively leverages them to build a different potential function for recognition at each level. In addition, we demonstrate that recognizing innermost entities first results in better performance than the conventional outermost entities first scheme. We provide extensive experimental results on ACE2004, ACE2005, and GENIA datasets to show the effectiveness and efficiency of our proposed method.

Nested Named Entity Recognition via Explicitly Excluding the Influence of the Best Path

Modern models for event causality identification (ECI) are mainly based on supervised learning, which are prone to the data lacking problem. Unfortunately, the existing NLP-related augmentation methods cannot directly produce available data required for this task. To solve the data lacking problem, we introduce a new approach to augment training data for event causality identification, by iteratively generating new examples and classifying event causality in a dual learning framework. On the one hand, our approach is knowledge guided, which can leverage existing knowledge bases to generate well-formed new sentences. On the other hand, our approach employs a dual mechanism, which is a learnable augmentation framework, and can interactively adjust the generation process to generate task-related sentences. Experimental results on two benchmarks EventStoryLine and Causal-TimeBank show that 1) our method can augment suitable task-related training data for ECI; 2) our method outperforms previous methods on EventStoryLine and Causal-TimeBank (+2.5 and +2.1 points on F1 value respectively).

LearnDA: Learnable Knowledge-Guided Data Augmentation for Event Causality Identification

Distantly supervision automatically generates plenty of training samples for relation extraction. 
However, it also incurs two major problems: noisy labels and imbalanced training data.
Previous works focus more on reducing wrongly labeled relations (false positives) while few explore the missing relations that are caused by incompleteness of knowledge base (false negatives).
Furthermore, the quantity of negative labels overwhelmingly surpasses the positive ones in previous problem formulations.
In this paper, we first provide a thorough analysis of the above challenges caused by negative data.
Next, we formulate the problem of relation extraction into as a positive unlabeled learning task to alleviate false negative problem. 
Thirdly, we propose a pipeline approach, dubbed \textsc{ReRe}, that first performs sentence classification with relational labels and then extracts the subjects/objects.
Experimental results show that the proposed method consistently outperforms existing approaches and remains excellent performance even learned with a large quantity of false positive samples.
Source code is available online at https://github.com/redreamality/RERE-relation-extraction.

Revisiting the Negative Data of Distantly Supervised Relation Extraction

This paper studies a new problem setting of entity alignment for knowledge graphs (KGs). Since KGs possess different sets of entities, there could be entities that cannot find alignment across them, leading to the problem of dangling entities. As the first attempt to this problem, we construct a new dataset and design a multi-task learning framework for both entity alignment and dangling entity detection. The framework can opt to abstain from predicting alignment for the detected dangling entities. We propose three techniques for dangling entity detection that are based on the distribution of nearest-neighbor distances, i.e., nearest neighbor classification, marginal ranking and background ranking. After detecting and removing dangling entities, an incorporated entity alignment model in our framework can provide more robust alignment for remaining entities. Comprehensive experiments and analyses demonstrate the effectiveness of our framework. We further discover that the dangling entity detection module can, in turn, improve alignment learning and the final performance. The contributed resource is publicly available to foster further research.

Knowing the No-match: Entity Alignment with Dangling Cases

We present TIMERS - a TIME, Rhetorical and Syntactic-aware model for document-level temporal relation classification in the English language. Our proposed method leverages rhetorical discourse features and temporal arguments from semantic role labels, in addition to traditional local syntactic features, trained through a Gated Relational-GCN. Extensive experiments show that the proposed model outperforms previous methods by 5-18\% on the TDDiscourse, TimeBank-Dense, and MATRES datasets due to our discourse-level modeling.

TIMERS: Document-level Temporal Relation Extraction

Relational fact extraction aims to extract semantic triples from unstructured text. In this work, we show that all of the relational fact extraction models can be organized according to a graph-oriented analytical perspective. An efficient model, aD}acency lIst oRiented rElational faCT (DIRECT), is proposed based on this analytical framework. To alleviate challenges of error propagation and sub-task loss equilibrium, DIRECT employs a novel adaptive multi-task learning strategy with dynamic sub-task loss balancing. Extensive experiments are conducted on two benchmark datasets, and results prove that the proposed model outperforms a series of state-of-the-art (SoTA) models for relational triple extraction.

Adjacency List Oriented Relational Fact Extraction via Adaptive Multi-task Learning

We present a novel model for the problem of ranking a collection of documents according to their semantic similarity to a source (query) document. While the problem of document-to-document similarity ranking has been studied, most modern methods are limited to relatively short documents or rely on the existence of "ground-truth'' similarity labels. Yet, in most common real-world cases, similarity ranking is an unsupervised problem as similarity labels are unavailable. Moreover, an ideal model should not be restricted by documents' length. Hence, we introduce SDR, a self-supervised method for document similarity that can be applied to documents of arbitrary length. 
Importantly, SDR can be effectively applied to extremely long documents, exceeding the $4,096$ maximal token limit of Longformer.
Extensive evaluations on large document datasets show that SDR significantly outperforms its alternatives across all metrics. To accelerate future research on unlabeled long document similarity ranking, and as an additional contribution to the community, we herein publish two human-annotated test sets of long documents similarity evaluation. The SDR code and datasets are publicly available.

Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference

Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with vast real-world applications. Noting precursor applications in other machine learning and AI techniques with pervasive societal impact, we anticipate the rising importance of developing NLP technologies for social good. Inspired by theories in moral philosophy and global priority research, we aim to promote a future guideline for social good in the context of NLP. We lay the foundations via \textit{moral philosophy}'s definition of social good, and propose a framework to provide a categorization of NLP tasks on the basis of real-world impact, along with metrics to calculate the expected social impact of NLP technology. Based on these fundamental frameworks, we adopt the methodology of global priority research to identify priority causes for NLP research, and use thought experiments to illustrate research priority under hypothetical situations. Finally, we use our theoretical framework to quantify ACL 2020 papers on the basis of societal impact, and provide a checklist for ensuring the creation of NLP technologies that advance social good. Based on these tools, we envision new usage of NLP to effectively improve social good and call attention to important NLP problems with the potential for strong social impact.

How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact

In this work, we propose a new task called Image-guided Story Ending Generation (IgSEG). Given a multi-sentence story plot and an ending-related image, IgSEG aims to generate a story ending that conforms to the contextual logic and the relevant visual concepts. In contrast to the story ending generation task, which generates open-ended endings, the major challenges of IgSEG are to comprehend the given context and image sufficiently, and mine the appropriate semantics from the image to make the generated story ending informative, reasonable, and coherent. To address the challenges, we propose a Multi-layer Graph convolution and Cascade-LSTM (MGCL) based model 
which mainly comprises of two collaborative modules: i) a multi-layer graph convolutional network to learn the dependency relations of sentences and the logical clue of the context; ii) a multiple context-image attention module to generate the story endings by gradually incorporating textual and visual semantic concepts. Our MGCL is thus capable of building logically consistent and semantically rich story endings. To evaluate the proposed model, we modify the existing VIST dataset to obtain the VIST-Ending dataset. Empirically, our MGCL outperforms all the strong baselines on both automatic and human evaluation.

IgSEG: Image-guided Story Ending Generation

Query focused summarization (QFS) models aim to generate summaries from source documents that can answer the given query. Most previous work on QFS only considers the query relevance criterion when producing the summary. However, studying the effect of answer relevance in the summary generating process is also important. In this paper, we propose QFS-BART, a model that incorporates the explicit answer relevance of the source documents given the query via a question answering model, to generate coherent and answer-related summaries. Furthermore, our model can take advantage of large pre-trained models which improve the summarization performance significantly. Empirical results on the Debatepedia dataset show that the proposed model achieves the new state-of-the-art performance.

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Downloads

Next from IJCNLP-AACL 2021

Nested Named Entity Recognition via Explicitly Excluding the Influence of the Best Path

Similar lecture

BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2021

Nested Named Entity Recognition via Explicitly Excluding the Influence of the Best Path

Similar lecture

BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

Downloads