Singapore

EMNLP 2023

Estimating Large Language Model Capabilities without Labeled Test Data

accuracy prediction

in-context learning

large language models

workshop paper

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

The Less the Merrier? Investigating Language Representation in Multilingual Models

Test-time Augmentation for Factual Probing

KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models

PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale

Relation extraction (RE) aims to extract the relations between entity names from the textual context. In principle, textual context determines the ground-truth relation and the RE models should be able to correctly identify the relations reflected by the textual context. However, existing work has found that the RE models memorize the entity name patterns to make RE predictions while ignoring the textual context. This motivates us to raise the question: are RE models robust to the entity replacements? In this work, we operate the random and type-constrained entity replacements over the RE instances in TACRED and evaluate the state-of-the-art RE models under the entity replacements. We observe the 30\% - 50\% F1 score drops on the state-of-the-art RE models under entity replacements. These results suggest that we need more efforts to develop effective RE models robust to entity replacements. We release the source code at https://github.com/wangywUST/RobustRE.

How Fragile is Relation Extraction under Entity Replacements?

This paper addresses structural ambiguity in Dutch relative clauses. By investigating the task of disambiguation by grounding, we study how the presence of a prior sentence can resolve relative clause ambiguities. We apply this method to two parsing architectures in an attempt to demystify the parsing and language model components of two present-day neural parsers. Results show that a neurosymbolic parser, based on proof nets, is more open to data bias correction than an approach based on universal dependencies, although both set-ups suffer from a comparable initial data bias.

Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization

Image-Text-Matching (ITM) is one of the defacto methods of learning generalized representations from a large corpus in Vision and Language (VL). However, due to the weak association between the web-collected image–text pairs, models fail to show fine-grained understanding of the combined semantics of these modalities. To this end, we propose Hard Negative Captions (HNC): an automatically created
dataset containing foiled hard negative captions for ITM training towards achieving fine-grained cross-modal comprehension in VL. Additionally, we provide a challenging manually-created test set for benchmarking models on a fine-grained cross-modal mismatch with varying levels of compositional complexity. Our results show the effectiveness of training on HNC by improving the models' zero-shot capabilities in detecting mismatches on diagnostic tasks and performing robustly under noisy visual input scenarios. Also, we demonstrate that HNC models yield a comparable or better initialization for fine-tuning. Our code and data are publicly available.

HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities

Semantic parsing aims to map natural language sentences to predefined formal languages,
such as logic forms and programming languages, as the semantic annotation.
From the theoretic views of linguistic and programming language,
structures play an important role in both languages,
which had motivated semantic parsers since the task was proposed in the beginning.
But in the neural era, semantic parsers treating both natural and formal language as sequences,
such as Seq2Seq and LLMs, have got more attentions.
On the other side, lots of neural progress have been made for grammar induction,
which only focuses on natural languages.
Although closely related in the sense of structural modeling,
these techniques hadn't been jointly analyzed on the semantic parsing testbeds.
To gain the better understanding on structures for semantic parsing,
we design a taxonomy of structural modeling methods,
and evaluate some representative techniques on semantic parsing,
including both compositional and i.i.d. generalizations.
In addition to the previous opinion that structures will help in general,
we find that (1) structures must be designed
for the specific dataset and generalization level,
and (2) what really matters is not the structure choice of either source or target side,
but the choice combination of both sides.
Based on the finding, we further propose a metric that can evaluate the structure choice,
which we believe can boost the automation of grammar designs for specific datasets and domains.

On the Effects of Structural Modeling for Neural Semantic Parsing

NLP models have progressed drastically in recent years, according to numerous datasets proposed to evaluate performance. Questions remain, however, about how particular dataset design choices may impact the conclusions we draw about model capabilities. In this work, we investigate this question in the domain of compositional generalization. We examine the performance of six modeling approaches across 4 datasets, split according to 8 compositional splitting strategies, ranking models by 18 compositional generalization splits in total. Our results show that:
i) the datasets, although all designed to evaluate compositional generalization, rank modeling approaches differently;
ii) datasets generated by humans align better with each other than with synthetic datasets, or than the latter among themselves;
iii) generally, whether datasets are sampled from the same source is more predictive of the resulting model ranking than whether they maintain the same interpretation of compositionality; and
iv) specific lexical items in dataset impacts the measurement consistency.
Overall, our results demonstrate that much work remains to be done when it comes to assessing whether popular evaluation datasets measure what they intend to measure, and suggests that elucidating more rigorous standards for establishing the validity of evaluation sets could benefit the field.

The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks

This paper presents the formalization of tree-shape uncertainty that enables us to analyze the inherent branching bias of unsupervised parsing models using raw texts alone. Previous work analyzed the branching bias of unsupervised parsing models by comparing the outputs of trained parsers with gold syntactic trees. However, such approaches do not consider the fact that texts can be generated by different grammars with different syntactic trees, possibly failing to clearly separate the inherent bias of the model and the bias in train data learned by the model. To this end, we formulate tree-shape uncertainty and derive sufficient conditions that can be used for creating texts that are expected to contain no biased information on branching. In the experiment, we show that training parsers on such unbiased texts can effectively detect the branching bias of existing unsupervised parsing models. Such bias may depend only on the algorithm, or it may depend on seemingly unrelated dataset statistics such as sequence length and vocabulary size.

Premium content

Estimating Large Language Model Capabilities without Labeled Test Data

Downloads

Next from EMNLP 2023

The Less the Merrier? Investigating Language Representation in Multilingual Models