Singapore

EMNLP 2023

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

data contamination

evaluation

workshop paper

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

Narrative Order Aware Story Generation via Bidirectional Pretraining Model with Optimal Transport Reward

Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs

A Closer Look into Using Large Language Models for Automatic Evaluation

Improving Pacing in Long-Form Story Planning

“Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters

Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models

Extracting relational triples from text is a crucial task for constructing knowledge bases. Recent advancements in joint entity and relation extraction models have demonstrated remarkable F1 scores ($\ge 90\%$) in accurately extracting relational triples from free text. However, these models have been evaluated under restrictive experimental settings and unrealistic datasets. They overlook sentences with zero triples (zero-cardinality), thereby simplifying the task. In this paper, we present a benchmark study of state-of-the-art joint entity and relation extraction models under a more realistic setting. We include sentences that lack any triples in our experiments, providing a comprehensive evaluation. Our findings reveal a significant decline (approximately 10-15\% in one dataset and 6-14\% in another dataset) in the models' F1 scores within this realistic experimental setup. Furthermore, we propose a two-step modeling approach that utilizes a simple BERT-based classifier. This approach leads to overall performance improvement in these models within the realistic experimental setting.

90% F1 Score in Relation Triple Extraction: Is it Real?

Code datasets used for training language models consists mainly of the most popular languages and evaluation is predominantly carried out on similar distributions, often overlooking low-resource programming languages.
Motivated by the NLP generalization taxonomy proposed by Hupkes et.al., we propose a new benchmark dataset called GenCodeSearchNet (GeCS) which builds upon existing natural language code search datasets to systemically evaluate the programming language understanding generalization capabilities of language models. As part of the full dataset, we introduce a new, manually curated subset StatCodeSearch that focuses on R, a popular but so far underrepresented programming language that is often used by researchers outside the field of computer science.
For evaluation and comparison we provide two measures with text-code matching and ranking. We collect several baseline results using fine-tuned BERT-style models and GPT-style large language models in a zero-shot setting.

GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding

The success of neural language models (LMs) on many technological tasks has brought about their potential relevance as scientific theories of language despite some clear differences be- tween LM training and child language acqui- sition. In this paper we argue that some of the most prominent benchmarks for evaluat- ing the syntactic capacities of LMs may not be sufficiently rigorous. In particular, we show that the template-based benchmarks lack the structural diversity commonly found in the the- oretical and psychological studies of language. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We advocate for the use of the readily available, carefully curated datasets that have been eval- uated for gradient acceptability by large pools of native speakers and are designed to probe the structural basis of grammar specifically. On one such dataset, the LI-Adger dataset, LMs evaluate sentences in a way inconsistent with human language users. We conclude with sug- gestions for better connecting LMs with the empirical study of child language acquisition.

Evaluating Neural Language Models as Cognitive Models of Language Acquisition

This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable names to explore whether models truly understand code semantics or merely rely on textual cues. We have also introduced adversaries like dead code and commented code across three programming languages (Python, Javascript, and Java) to further scrutinize the model's understanding. Ultimately, our research aims to offer valuable insights into the inner workings of transformer-based LMs, enhancing their ability to understand code and contributing to more efficient software development practices and maintenance workflows.

Premium content

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

Downloads

Next from EMNLP 2023

Narrative Order Aware Story Generation via Bidirectional Pretraining Model with Optimal Transport Reward