Dominican Republic

I survey work on linking Large Language Models to brain imaging data. I argue &#39;interpretability&#39; is crucial for using this work to reveal more about how the human brain processes language. I then survey some of our contributions to interpretability methods, and discuss the relevance of these methods for the big questions from cognitive neuroscience.

EMNLP 2021

Language, Brains &amp; Interpretability

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

I survey work on linking Large Language Models to brain imaging data. I argue 'interpretability' is crucial for using this work to reveal more about how the human brain processes language. I then survey some of our contributions to interpretability methods, and discuss the relevance of these methods for the big questions from cognitive neuroscience.

Language, Brains & Interpretability

workshop paper

Entity retrieval, which aims at disambiguating mentions to canonical entities from massive KBs, is essential for many tasks in natural language processing. Recent progress in entity retrieval shows that the dual-encoder structure is a powerful and efficient framework to nominate candidates if entities are only identified by descriptions. However, they ignore the property that meanings of entity mentions diverge in different contexts and are related to various portions of descriptions, which are treated equally in previous works. In this work, we propose Multi-View Entity Representations (MuVER), a novel approach for entity retrieval that constructs multi-view representations for entity descriptions and approximates the optimal view for mentions via a heuristic searching method. Our method achieves the state-of-the-art performance on ZESHEL and improves the quality of candidates on three standard Entity Linking datasets

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Event detection (ED) aims at identifying event instances of specified types in given texts, which has been formalized as a sequence labeling task. As far as we know, existing neural-based ED models make decisions relying on the contextual semantic features of each word in the input text, which we find is easy to get confused by varied contexts in the test stage. To this end, we come up with the idea of introducing a set of statistical features from word-event co-occurrence frequencies in the entire training set to cooperate with the contextual features. Specifically, we propose a Semantic and Statistic-Joint Discriminative Network (SS-JDN) consisting of a semantic feature extractor, a statistical feature extractor, and a joint event discriminator. In experiments, SS-JDN effectively exceeds ten recent state-ofthe-art (SOTA) baseline methods on ACE2005 and KBP2015 benchmark datasets. Further, we perform extensive experiments to investigate the effectiveness of SS-JDN.

Treasures Outside Contexts: Improving Event Detection via Global Statistics

Conventional entity typing approaches are based on independent classification paradigms, which make them difficult to recognize inter-dependent, long-tailed and fine-grained entity types. In this paper, we argue that the implicitly entailed extrinsic and intrinsic dependencies between labels can provide critical knowledge to tackle the above challenges. To this end, we propose Label Reasoning Network(LRN), which sequentially reasons fine-grained entity labels by discovering and exploiting label dependencies knowledge entailed in the data. Specifically, LRN utilizes an auto-regressive network to conduct deductive reasoning and a bipartite attribute graph to conduct inductive reasoning between labels, which can effectively model, learn and reason complex label dependencies in a sequence-to- set, end-to-end manner. Experiments show that LRN achieves the state-of-the-art performance on standard ultra fine-grained entity typing benchmarks, and can also resolve the long tail label problem effectively.

Fine-grained Entity Typing via Label Reasoning

Table-based fact verification task aims to verify whether the given statement is supported by the given semi-structured table. Symbolic reasoning with logical operations plays a crucial role in this task. Existing methods leverage programs that contain rich logical information to enhance the verification process. However, due to the lack of fully supervised signals in the program generation process, spurious programs can be derived and employed, which leads to the inability of the model to catch helpful logical operations. To address the aforementioned problems, in this work, we formulate the tablebased fact verification task as an evidence retrieval and reasoning framework, proposing the Logic-level Evidence Retrieval and Graphbased Verification network (LERGV). Specifically, we first retrieve logic-level program-like evidence from the given table and statement as supplementary evidence for the table. After that, we construct a logic-level graph to capture the logical relations between entities and functions in the retrieved evidence, and design a graph-based verification network to perform logic-level graph-based reasoning based on the constructed graph to classify the final entailment relation. Experimental results on the large-scale benchmark TABFACT show the effectiveness of the proposed approach.

Logic-level Evidence Retrieval and Graph-based Verification Network for Table-based Fact Verification

We present a novel partition filter encoder designed for joint entity and relation extraction, in an attempt to reinforce bilateral interaction between NER and RE.

A Partition Filter Network for Joint Entity and Relation Extraction

To alleviate label scarcity in Named Entity Recognition (NER) task, distantly supervised NER methods are widely applied to automatically label data and identify entities. Although the human effort is reduced, the generated incomplete and noisy annotations pose new challenges for learning effective neural models. In this paper, we propose a novel dictionary extension method which extracts new entities through the type expanded model. Moreover, we design a multi-granularity boundary- aware network which detects entity boundaries from both local and global perspectives. We conduct experiments on different types of datasets, the results show that our model outperforms previous state-of-the-art distantly supervised systems and even surpasses the supervised models.

TEBNER: Domain Specific Named Entity Recognition with Type Expanded Boundary-aware Network

Distantly supervised relation extraction (RE) automatically aligns unstructured text with relation instances in a knowledge base (KB). Due to the incompleteness of current KBs, sentences implying certain relations may be annotated as N/A instances, which causes the socalled false negative (FN) problem. Current RE methods usually overlook this problem, inducing improper biases in both training and testing procedures. To address this issue, we propose a two-stage approach. First, it finds out possible FN samples by heuristically leveraging the memory mechanism of deep neural networks. Then, it aligns those unlabeled data with the training data into a unified feature space by adversarial training to assign pseudo labels and further utilize the information contained in them. Experiments on two wildlyused benchmark datasets demonstrate the effectiveness of our approach.

Knowing False Negatives: An Adversarial Training Method for Distantly Supervised Relation Extraction

Where next? Towards multi-text consumption via three inspired research lines

Human explanations are responses to "why P rather than Q?" where P is an observed event and Q an imagined, counterfactual event that did not occur. Since people assign human-like traits to AI models, they expect explanations of models' behavior to follow the same conceptual framework used to explain human behavior. The consequence of not meeting this expectation is that people using AI technology do not have agency and hence one of the XAI objectives is not achieved. NLP researchers are accepting the perspective of the social and behavioral sciences as evident by the number of papers that have been published in the last and this year on contrastive explanations. In this talk, I will present a categorization of methods for producing contrastive explanations of NLP models and go into details of one such method.

Contrastive Explanations of NLP Models

Research in financial economics utilizing natural language processing has modeled how U.S. companies compete across product markets, how they innovate, and how supply chains are organized. This presentation will summarize these models, how they have evolved to the present day, and new work in progress. A key technical foundation is that these concepts of market structure, which were developed by economic theorists, have relatedness and overlap as their very intellectual foundation. Hence NLP methods (new and old) that focus on textual relatedness make up an ideal toolkit for analyzing these economic concepts, which are also central to many applications aimed at understanding and predicting business success.
One of the most compelling properties of an NLP approach is that market structures can be updated and redrawn every year at low cost as textual corpora are updated. This observation indicates the use of NLP to measure innovation. This has led to many insights on how companies manage growth, whether and when they go public, and how they manage the risk of disruption. This work continues to evolve with a focus on innovation, disaggregating complex product portfolios, and to build NLP measures for private firms through their websites. Among other economic findings, this work reveals that U.S. companies are becoming increasingly complex and broad in their product offerings, a result that is not seen using standard numerical databases in the public domain. Work using these same methods has also been deployed in related settings such as modeling systemic risk.

Language, Brains & Interpretability

Downloads

Next from EMNLP 2021

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Similar lecture

Probing Pre-trained Language Models for Semantic Attributes and their Values

Language, Brains & Interpretability

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2021

MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Similar lecture

Probing Pre-trained Language Models for Semantic Attributes and their Values

Downloads