Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as \textbf{t}ask and \textbf{a}rgument \textbf{g}rounding for \textbf{E}mbodied agents (\system). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.

tagE: Enabling an Embodied Agent to Understand Human Instructions​

The prevalence and salience of LMs has changed the way many researchers in NLP think about our work, causing many issues for model evaluation. This presentation presents a retrospective of recent past approaches to modeling text, and discusses what has changed, and what the current evaluation challenges are. Next, it discusses common pitfalls that LM-based research falls into, highlighting several fallacies, and suggesting ways to avoid them. Finally, this presentation highlights the importance of meta-evaluation research (the evaluation of our evaluations) as one crucial component to overcoming LM evaluation challenges in the post LM-boom.

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentations of Contextual Bangla Neural Stemmer: Finding Contextualized Root Word Representations for Bangla Words

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

Large language models capture knowledge and abilities learned from their pre-training data, which is typically limited to text and code. We present approaches for *grounding* these LLMs to carry out tasks that involve visual perception and reasoning. Our methods leverage LLMs' abilities to generate (1) dialogue (2) questions and (3) code, putting these to use in a series of visual dialogue tasks where people interact with models in perceptually-complex settings.

Interacting with LLMs for Grounded Tasks

Identifying intents from dialogue utterances forms an integral component of task-oriented dialogue systems. Intent-related tasks are typically formulated either as a classification task, where the utterances are classified into predefined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be modeled in a multiclass (MC) or multilabel (ML) setup. While typically these tasks are modeled as separate tasks, we propose INTENDD a unified approach leveraging a shared utterance encoding backbone. INTENDD uses an entirely unsupervised contrastive learning strategy for representation learning, where pseudo labels for the unlabeled utterances are generated based on their lexical features. Additionally, we introduce a two-step post-processing setup for the classification tasks using modified adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, INTENDD reports percentage improvements of 2.32 %, 1.26 %, and 1.52 % in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism’s key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token’s importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model’s performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Inverse Text Normalization (ITN) involves rewriting the verbalised form of text from spoken transcripts to its corresponding written form. The task inherently expects challenges in identifying ITN entries due to spelling variations in words arising out of dialects, transcription errors etc. Additionally, in Tamil, word boundaries between adjacent words in a sentence often get obscured due to Punarchi, i.e. phonetic transformation of these boundaries. Being morphologically rich, the words in Tamil show a high degree of agglutination due to inflection and clitics. The combination of such factors leads to a high degree of surfaceform variations, making scalability with pure rule-based approaches difficult. Instead, we experiment with fine-tuning three pre-trained neural LMs, consisting of a seq2seq model (s2s), a non-autoregressive text editor (NAR) and a sequence tagger + rules combination (tagger). While the tagger approach works best in a fully-supervised setting, s2s performs the best (98.05 F-Score) when augmented with additional data, via bootstrapping and data augmentation (DA&B). S2S reports a cumulative percentage improvement of 20.1 %, and statistically significant gains for all our models with DA&B. Compared to a fully supervised setup, bootstrapping alone reports a percentage improvement as high as 14.12 %, even with a small seed set of 324 ITN entries.

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence.

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

Transliteration is very important in the Indian
language context due to the usage of multi-
ple scripts and the widespread use of roman-
ized inputs. However, few training and evalu-
ation sets are publicly available. We introduce
Aksharantar1, the largest publicly available
transliteration dataset for Indian languages cre-
ated by mining from monolingual and paral-
lel corpora, as well as collecting data from hu-
man annotators. The dataset contains 26 mil-
lion transliteration pairs for 21 Indic lan-
guages from 3 language families using 12
scripts. Aksharantar is 21 times larger than ex-
isting datasets and is the first publicly available
dataset for 7 languages and 1 language fam-
ily. We also introduce the Aksharantar testset
comprising 103k word pairs spanning 19 lan-
guages that enables a fine-grained analysis of
transliteration models on native origin words,
foreign words, frequent words, and rare words.
Using the training set, we trained IndicXlit,
a multilingual transliteration model that im-
proves accuracy by 15% on the Dakshina test
set, and establishes strong baselines on the Ak-
sharantar testset introduced in this work. The
models, mining scripts, transliteration guide-
lines, and datasets are available at https://
github.com/AI4Bharat/IndicXlit under
open-source licenses.

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

Open-domain dialogue system usually requires different sources of knowledge to generate more informative and evidential responses. However, existing knowledge-grounded dialogue systems either focus on a single knowledge source or overlook the dependency between multiple sources of knowledge, which may result in generating inconsistent or even paradoxical responses. To incorporate multiple knowledge sources and dependencies between them, we propose SAFARI, a novel framework that leverages the exceptional capabilities of large language models (LLMs) in planning, understanding, and incorporating under both supervised and unsupervised settings. Specifically, SAFARI decouples the knowledge grounding into multiple sources and response generation, which allows easy extension to various knowledge sources including the possibility of not using any sources. To study the problem, we construct a personalized knowledge-grounded dialogue dataset Knowledge Behind Persona (KBP), which is the first to consider the dependency between persona and implicit knowledge. Experimental results on the KBP dataset demonstrate that the SAFARI framework can effectively produce persona-consistent and knowledge-enhanced responses.

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines.1 Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

Visually Grounded Continual Language Learning with Selective Specialization

One of the frequent points in the mainstream narrative about large language models is that they have “emergent properties” (sometimes even dangerous enough to be considered existential risk to mankind). However, there is much disagreement about even the very definition of such properties. If they are understood as a kind of generalization beyond training data - as something that a model does without being explicitly trained for it - I argue that we have not in fact established the existence of any such properties, and at the moment we do not even have the methodology for doing so.

Underline digital video library

tagE: Enabling an Embodied Agent to Understand Human Instructions

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

Interacting with LLMs for Grounded Tasks

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

Visually Grounded Continual Language Learning with Selective Specialization

A sanity check on emergent properties

Underline digital video library

tagE: Enabling an Embodied Agent to Understand Human Instructions​

Evaluation after the LM boom: Frustrations, fallacies andthe future

Oral Presentation of CNS Model-Paper ID: 68 | VIDEO

Interacting with LLMs for Grounded Tasks

IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Scaling Neural ITN for Numbers and Temporal Expressions in Tamil: Findings for an Agglutinative Low-resource Language | VIDEO

CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models

Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues

Visually Grounded Continual Language Learning with Selective Specialization

A sanity check on emergent properties

tagE: Enabling an Embodied Agent to Understand Human Instructions