Dominican Republic

I survey work on linking Large Language Models to brain imaging data. I argue &#39;interpretability&#39; is crucial for using this work to reveal more about how the human brain processes language. I then survey some of our contributions to interpretability methods, and discuss the relevance of these methods for the big questions from cognitive neuroscience.

EMNLP 2021

Language, Brains &amp; Interpretability

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

I survey work on linking Large Language Models to brain imaging data. I argue 'interpretability' is crucial for using this work to reveal more about how the human brain processes language. I then survey some of our contributions to interpretability methods, and discuss the relevance of these methods for the big questions from cognitive neuroscience.

Language, Brains & Interpretability

workshop paper

Entity retrieval, which aims at disambiguating mentions to canonical entities from massive KBs, is essential for many tasks in natural language processing. Recent progress in entity retrieval shows that the dual-encoder structure is a powerful and efficient framework to nominate candidates if entities are only identified by descriptions. However, they ignore the property that meanings of entity mentions diverge in different contexts and are related to various portions of descriptions, which are treated equally in previous works. In this work, we propose Multi-View Entity Representations (MuVER), a novel approach for entity retrieval that constructs multi-view representations for entity descriptions and approximates the optimal view for mentions via a heuristic searching method. Our method achieves the state-of-the-art performance on ZESHEL and improves the quality of candidates on three standard Entity Linking datasets

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Event detection (ED) aims at identifying event instances of specified types in given texts, which has been formalized as a sequence labeling task. As far as we know, existing neural-based ED models make decisions relying on the contextual semantic features of each word in the input text, which we find is easy to get confused by varied contexts in the test stage. To this end, we come up with the idea of introducing a set of statistical features from word-event co-occurrence frequencies in the entire training set to cooperate with the contextual features. Specifically, we propose a Semantic and Statistic-Joint Discriminative Network (SS-JDN) consisting of a semantic feature extractor, a statistical feature extractor, and a joint event discriminator. In experiments, SS-JDN effectively exceeds ten recent state-ofthe-art (SOTA) baseline methods on ACE2005 and KBP2015 benchmark datasets. Further, we perform extensive experiments to investigate the effectiveness of SS-JDN.

Treasures Outside Contexts: Improving Event Detection via Global Statistics

Conventional entity typing approaches are based on independent classification paradigms, which make them difficult to recognize inter-dependent, long-tailed and fine-grained entity types. In this paper, we argue that the implicitly entailed extrinsic and intrinsic dependencies between labels can provide critical knowledge to tackle the above challenges. To this end, we propose Label Reasoning Network(LRN), which sequentially reasons fine-grained entity labels by discovering and exploiting label dependencies knowledge entailed in the data. Specifically, LRN utilizes an auto-regressive network to conduct deductive reasoning and a bipartite attribute graph to conduct inductive reasoning between labels, which can effectively model, learn and reason complex label dependencies in a sequence-to- set, end-to-end manner. Experiments show that LRN achieves the state-of-the-art performance on standard ultra fine-grained entity typing benchmarks, and can also resolve the long tail label problem effectively.

Fine-grained Entity Typing via Label Reasoning

In this paper, we propose Sequence Span Rewriting (SSR), a self-supervised task for sequence-to-sequence (Seq2Seq) pre-training. SSR learns to refine the machine-generated imperfect text spans into ground truth text. SSR provides more fine-grained and informa- tive supervision in addition to the original text- infilling objective. Compared to the preva- lent text infilling objectives for Seq2Seq pre- training, SSR is naturally more consistent with many downstream generation tasks that re- quire sentence rewriting (e.g., text summariza- tion, question generation, grammatical error correction, and paraphrase generation). We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

Generating long text conditionally depending on the short input text has recently attracted more and more research efforts. Most existing approaches focus more on introducing extra knowledge to supplement the short input text, but ignore the coherence issue of the generated texts. To address aforementioned research issue, this paper proposes a novel twostage approach to generate coherent long text. Particularly, we first build a document-level path for each output text with each sentence embedding as its node, and a revised selforganising map (SOM) is proposed to cluster similar nodes of a family of document-level paths to construct the directed semantic graph. Then, three subgraph alignment methods are proposed to extract the maximum matching paths or subgraphs. These directed subgraphs are considered to well preserve extra but relevant content to the short input text, and then they are decoded by the employed pre-trained model to generate coherent long text. Extensive experiments have been performed on three real-world datasets, and the promising results demonstrate that the proposed approach is superior to the state-of-the-art approaches w.r.t. a number of evaluation criteria.

Building the Directed Semantic Graph for Coherent Long Text Generation

We study continual learning for natural language instruction generation, by observing human users' instruction execution. We focus on a collaborative scenario, where the system both acts and delegates tasks to human users using natural language. We compare user execution of generated instructions to the original system intent as an indication to the system's success communicating its intent. We show how to use this signal to improve the system's ability to generate instructions via contextual bandit learning. In interaction with real users, our system demonstrates dramatic improvements in its ability to generate language over time.

Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

Distantly supervised relation extraction (RE) automatically aligns unstructured text with relation instances in a knowledge base (KB). Due to the incompleteness of current KBs, sentences implying certain relations may be annotated as N/A instances, which causes the socalled false negative (FN) problem. Current RE methods usually overlook this problem, inducing improper biases in both training and testing procedures. To address this issue, we propose a two-stage approach. First, it finds out possible FN samples by heuristically leveraging the memory mechanism of deep neural networks. Then, it aligns those unlabeled data with the training data into a unified feature space by adversarial training to assign pseudo labels and further utilize the information contained in them. Experiments on two wildlyused benchmark datasets demonstrate the effectiveness of our approach.

Knowing False Negatives: An Adversarial Training Method for Distantly Supervised Relation Extraction

Where next? Towards multi-text consumption via three inspired research lines

Human explanations are responses to "why P rather than Q?" where P is an observed event and Q an imagined, counterfactual event that did not occur. Since people assign human-like traits to AI models, they expect explanations of models' behavior to follow the same conceptual framework used to explain human behavior. The consequence of not meeting this expectation is that people using AI technology do not have agency and hence one of the XAI objectives is not achieved. NLP researchers are accepting the perspective of the social and behavioral sciences as evident by the number of papers that have been published in the last and this year on contrastive explanations. In this talk, I will present a categorization of methods for producing contrastive explanations of NLP models and go into details of one such method.

Contrastive Explanations of NLP Models

Research in financial economics utilizing natural language processing has modeled how U.S. companies compete across product markets, how they innovate, and how supply chains are organized. This presentation will summarize these models, how they have evolved to the present day, and new work in progress. A key technical foundation is that these concepts of market structure, which were developed by economic theorists, have relatedness and overlap as their very intellectual foundation. Hence NLP methods (new and old) that focus on textual relatedness make up an ideal toolkit for analyzing these economic concepts, which are also central to many applications aimed at understanding and predicting business success.
One of the most compelling properties of an NLP approach is that market structures can be updated and redrawn every year at low cost as textual corpora are updated. This observation indicates the use of NLP to measure innovation. This has led to many insights on how companies manage growth, whether and when they go public, and how they manage the risk of disruption. This work continues to evolve with a focus on innovation, disaggregating complex product portfolios, and to build NLP measures for private firms through their websites. Among other economic findings, this work reveals that U.S. companies are becoming increasingly complex and broad in their product offerings, a result that is not seen using standard numerical databases in the public domain. Work using these same methods has also been deployed in related settings such as modeling systemic risk.

Downloads

Next from EMNLP 2021

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Similar lecture

Probing Pre-trained Language Models for Semantic Attributes and their Values

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2021

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Similar lecture

Probing Pre-trained Language Models for Semantic Attributes and their Values

Downloads