Canada

The task of noun compound interpretation, or NCI, aims to uncover the implicit relation between constituent nouns. One way of performing this task is via paraphrasing. For example, given the noun compound “chocolate bunny”, acceptable paraphrases would be “a bunny made of chocolate” or “a bunny shaped chocolate. We also introduce a task similar to NCI called noun compound conceptualization (NCC). The difference is that NCC involves infrequent or novel noun compounds, for which it may be more difficult to come up with a plausible meaning. Our work involves evaluating how well modern large language models perform on both NCI and NCC. We also investigate how much of the responses of large language models are copied from the web.

ACL 2023

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

noun compound

gpt-3

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

workshop paper

SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the question of when these abilities begin to emerge largely unanswered. In this paper, we investigate whether the effects of pre-training can be observed when the problem size is reduced, modeling a smaller, reduced-vocabulary language. We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters, and establish a strong correlation between pre-training perplexity and downstream performance (GLUE benchmark). We examine downscaling effects, extending scaling laws to models as small as 1M parameters. At this scale, we observe a break of the power law for compute-optimal models and show that the MLM loss does not scale smoothly with compute-cost (FLOPs) below 2.2e15 FLOPs. We also find that adding layers does not always benefit downstream performance.

Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Recurrent Attention Networks for Long-text Modeling

RHO ($\rho$): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding

Accounts of human language processing have long appealed to implicit “situation models” that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent trans- former models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a rela- tively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the can- didate noun phrases the pronoun ultimately at- tends to. We then compare how this circuit be- haves in a closely matched “syntactic” control where the situation model is not strictly nec- essary. These analyses suggest distinct path- ways through which implicit situation models are constructed to guide pronoun resolution.

Causal interventions expose implicit situation models for commonsense language understanding

Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities. An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps. Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps. This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps. We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective. We define metrics to evaluate both generation and inductive quality. Experiment results demonstrate that our approach significantly outperforms strong baselines.

Multimedia Generative Script Learning for Task Planning

Prior work has shown that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen. Despite learning a small subset of parameters, this approach is not compute-efficient, as training the new embeddings requires a full forward and backward pass over the entire model. In this work, we propose mini-model adaptation, a compute-efficient alternative that builds a shallow mini-model from a fraction of a large model's parameters. New language-specific embeddings can then be efficiently trained over the mini-model, and plugged into the aligned large model for rapid cross-lingual transfer. We explore two approaches to learn mini-models: MiniJoint, which jointly pretrains the primary model and the mini-model using a single transformer with a secondary MLM head at a middle layer; and MiniPost, where we start from a regular pretrained model and build a mini-model by extracting and freezing a few layers and learning a small number of parameters on top. Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the performance of the standard approach using up to 2.3x less compute.

Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training

Since conventional knowledge embedding models
cannot take full advantage of the abundant
textual information, there have been extensive
research efforts in enhancing knowledge embedding
using texts. However, existing enhancement
approaches cannot apply to temporal
knowledge graphs (tKGs), which contain
time-dependent event knowledge with complex
temporal dynamics. Specifically, existing
enhancement approaches often assume knowledge
embedding is time-independent. In contrast,
the entity embedding in tKG models usually
evolves, which poses the challenge of aligning
temporally relevant texts with entities. To
this end, we propose to study enhancing temporal
knowledge embedding with textual data
in this paper. As an approach to this task, we
propose Enhanced Temporal Knowledge Embeddings
with Contextualized Language Representations
(ECOLA), which takes the temporal
aspect into account and injects textual information
into temporal knowledge embedding.
To evaluate ECOLA, we introduce three new
datasets for training and evaluating ECOLA.
Extensive experiments show that ECOLA significantly
enhances temporal KG embedding
models with up to 287% relative improvements
regarding Hits@1 on the link prediction task.
The code and models are publicly available.

Downloads

Next from ACL 2023

SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Similar lecture

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?