Indonesia

Fluency is a crucial goal of all Natural Language Generation (NLG) systems. Widely
used automatic evaluation metrics fall short in capturing the fluency of machine-generated
text. Assessing the fluency of NLG systems poses a challenge since these models are not limited
to simply reusing words from the input but may also generate abstractions. Existing reference-
based fluency evaluations, such as word overlap measures, often exhibit weak correlations with
human judgments. This paper presents an unsupervised technique for measuring text fluency
without the need for any reference. Our approach leverages various word embeddings and
trains language models using Recurrent Neural Network (RNN) architectures. We also

experiment with other available multilingual Language Models (LMs). To assess the
performance of the models, we conduct a comparative analysis across 10 Indic languages,
correlating the obtained fluency scores with human judgments. We also release a human-
annotated benchmark test set for fluency.

IJCNLP-AACL 2023

Unsupervised approach to evaluate sentence-level fluency: Do we really need reference? 

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

workshop paper

### Welcome to IJCNLP-AACL 2023!
I am honored and happy to welcome you to IJCNLP-AACL 2023! The conference hosts both the 13th
International Joint Conference on Natural Language Processing (IJCNLP) and the 3rd Conference of
the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL). 
The joint conferences of IJCNLP and AACL have been organized by alternating leadership in the
Asia-Pacific region, in odd years by Asian Federation of Natural Language Processing (AFNLP) and
in even years by AACL. The conference is organized by AFNLP this year. 

Considering the current situation that we are not yet fully over the coronavirus pandemic, the
conference is offered in a hybrid mode, off-line in Nusa Dua, Bali, Indonesia, and on-line with the help
of Underline, during November 1-4, 2023. For those attending off-line, I hope that you will find time to
explore the exciting local area, which remains quite warm and pleasant at this time of the year. For
those attending on-line, I hope that the attendance is not much hampered, sometimes even enhanced
and comfortable, though the on-line arrangements have been made without ample time. 

Message from Jong C. Park, Korea Advanced Institute of Science and Technology (KAIST) 
IJCNLP-AACL 2023 General Chair 
###### [Conference Handbook](https://drive.google.com/file/d/1S2JSXa105xJxIkNBfhYmGH20Om6wARBx/view?usp=sharing)

In order to acces this event page, you need to register [here](http://www.ijcnlp-aacl2023.org/)

The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics

Controllable text generation is a fundamental aspect of natural language generation, with numerous methods proposed for different constraint types. However, these approaches often require significant architectural or decoding modifications, making them challenging to apply to additional constraints or resolve different constraint combinations. To address this, our paper introduces Regular Expression Instruction (REI), which utilizes an instruction-based mechanism to fully exploit regular expressions' advantages to uniformly model diverse constraints. Specifically, our REI supports all popular fine-grained controllable generation constraints, i.e., lexical, positional, and length, as well as their complex combinations, via regular expression-style instructions. Our method only requires fine-tuning on medium-scale language models or few-shot, in-context learning on large language models, and requires no further adjustment when applied to various constraint combinations. Experiments demonstrate that our straightforward approach yields high success rates and adaptability to various constraints while maintaining competitiveness in automatic metrics and outperforming most previous baselines.

Toward Unified Controllable Text Generation via Regular Expression Instruction

Solving math word problems (MWP) is a challenging task for natural language processing systems, as it requires to not only identify and comprehend the problem description within the context, but also to deduce a solution in accordance with the posed question. Previous solvers have been found to prioritize the context over the question, resulting in low performance when solving multiple questions under the same context. In this paper, we present a question-oriented strategy to address this issue and improve the generalizability of MWP solvers. Our approach features an entity-aware encoder that enhances the connection between MWP context and question via entities in established dependency graphs, aiming at obtaining better problem representations. Then, a question-guided decoder is trained using a contrastive learning strategy to enhance the question representations. Empirical evaluations on four benchmarks demonstrate that our method outperforms previous solvers and exhibits a favorable balance between efficacy and efficiency in MWP solving. In addition, our solver is not reliant on any specific pre-trained model and demonstrates seamless compatibility with different pre-trained model backbones.

Don't be Blind to Questions: Question-Oriented Math Word Problem Solving

Detecting out-of-scope (OOS) utterances is crucial in task-oriented dialogue systems, but obtaining enough annotated OOS dialogues to train a binary classifier directly is difficult in practice. Existing data augmentation methods generate OOS dialogues automatically, but their performance usually depends on an external corpus. This dependence not only induces uncertainty, but also reduces the quality of generated dialogues. Specifically, all of them are out-of-domain (OOD). 

Herein we propose SILVER, a self data augmentation method that does not use external data. It addresses issues of previous research and improves the accuracy of OOS detection (false positive rate: 90.5% to 47.4%). Furthermore, SILVER successfully generates high-quality in-domain (IND) OOS dialogues in terms of naturalness (percentage: 8% to 68%) and OOS correctness (percentage: 74% to 88%), as evaluated by human workers.

SILVER: Self Data Augmentation for Out-of-Scope Detection in Dialogues

The strong capabilities of conversation-based large language models in healthcare applications are attracting an increasingly larger audience. However, the reliability of these models in consistently and accurately providing medical advice based on user-inputted symptoms is a critical concern. This study explores the sensitivity of LLMs to variations in user input, focusing specifically on how different symptom descriptions and prior users' beliefs can potentially lead to different diagnoses. We test two GPT models with five different prompt templates to assess their ability of mentioning the true patient condition based on the symptoms. Our findings reveal a substantial sensitivity to input variations---especially when users have prior assumptions and beliefs---indicating potential inconsistencies in the diagnoses generated by these models.

“Dr LLM, what do I have?”: The Impact of User Beliefs and Prompt Formulation on Health Diagnoses

In clinical visits, clinical note writing is a time-consuming and cost-prohibitive manual task for clinicians. Although virtual medical scribes have been proposed to generate clinical notes (semi-)automatically, the data sparsity issue is still a challenging problem in practice. Identifying the topic of clinical utterances in doctor-patient conversations is one of the key strategies for automation. In this paper, we propose an utterance-level note section classification method for the situation of the limited amount of in-house data. We leverage an external, unsupervised corpus of medical conversations to transfer knowledge using the framework of Unsupervised Meta-learning with Task Augmentation (UMTA). Our experiments are performed on both manual transcripts and machine transcripts generated by automatic speech recognition (ASR). The results show that our strategies achieve substantial gains in prediction accuracy over several baseline approaches and are robust to ASR errors.

Clinical note section classification on doctor-patient conversations in low-resourced settings

Rumors have emerged as significant global challenges in recent years. Once praised as means of information, unity, and global connectivity, social media platforms and online communication tools have now become breeding grounds for the spread of false, misleading, and manipulative information on an unprecedented scale. Rumors, composed of untrue information and deceptive messages, are designed in a way that manipulates emotions. Therefore, emotions play a vital role in determining the veracity of rumors. In this paper, we present a novel emotion-enriched and psycholinguistics features-based approach for rumor detection on online social media. The proposed approach encompasses detecting rumors using lexicons and various linguistics features-based learning approaches, mainly extracting the psychological association of words and their emotions. The emotion-related and psycholinguistic features are extracted for both posts and comments to augment the approach and facilitate detecting rumors more effectively. The semantic relations of a post and its comments with their underlying emotions are preserved using word-level GloVe embedding. The proposed approach is evaluated over a publicly available PHEME dataset and compared with various baselines and SOTA approaches, which show significantly better performance for rumor detection on online social media.

An Emotion-Enriched and Psycholinguistics Features-Based Approach for Rumor Detection on Online Social Media

Biomedical entity linking, also known as biomedical concept normalization, has recently witnessed the rise to prominence of zero-shot contrastive models. However, the pretraining material used for these models has, until now, largely consisted of specialist biomedical content such as MIMIC-III clinical notes and PubMed papers. While the resulting in-domain models have shown promising results for many biomedical tasks, adverse drug event normalization on social media texts has so far remained challenging for them.

In this paper, we propose a new approach for adverse drug event normalization on social media relying on general-purpose model initialization via BioLORD and a semantic-text-similarity finetuning named STS.

Our experimental results on several social media datasets demonstrate the effectiveness of our proposed approach, by achieving state-of-the-art performance. Based on its strong performance across all the tested datasets, we believe this work could emerge as a turning point for the task of adverse drug event normalization on social media and has the potential to serve as a benchmark for future research in the field.

Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contexts

Identifying mental health conditions using user-generated text on social media paved the way for automated computational methods for mental health surveillance on social media. Inferring the accurate state of mental health markers requires understanding the user's emotions and history of mental health conditions for identifying the mental health landscape of at-risk users. Existing works have exploited social media data by using users' most recent posts; however, previous works ignored users' emotional historical activity on social media, indicating their mental state over time. In this work, we address this gap and present a novel emotion- and time-aware architecture that jointly learns social media users' emotional historical context and temporal posting irregularities for mental health surveillance on social media. We conduct classification experiments to identify social media users with mental conditions, i.e., (i) healthy users and users affected with depression and (ii) healthy users and users prone to self-harm. Experimental results demonstrate proposed method outperforms recent competitive methods, demonstrating the importance of capturing the user's temporal emotional spectrum and time-aware emotional context through historical posts for social media users' mental health surveillance.


Temporal Tides of Emotional Resonance: A Novel Approach to Identify Mental Health on Social Media

Huge amounts of sentiment-rich data are produced on social networks, the rate of which is growing year by year. As such, a global view of sentiment on social media platforms is only possible at scale through automation. The huge data sizes also mean that sentiment analysis systems must balance both accuracy and computing resource requirements. Typical NLP research maximizes accuracy, while other metrics are overlooked and thus older models are forgotten. Prior models are easily forgotten despite their possible suitability in settings where large computing resources are unavailable or relatively more costly. In this paper, we perform a broad comparative evaluation of document-level sentiment analysis models with a focus on resource costs that are important for the feasibility of model deployment and general climate consciousness. Our experiments consider different feature extraction techniques, the effect of ensembling, task-specific deep learning modeling, and domain-independent large language models (LLMs). We find that while a fine-tuned LLM achieves the best accuracy, some alternate configurations provide huge (up to 24, 283×) resource savings for a marginal (<1%) loss in accuracy. Furthermore, we find that for smaller datasets, the differences in accuracy shrink while the difference in resource consumption grows further.

Efficient Sentiment Analysis: A Resource-Aware Evaluation of Feature Extraction Techniques, Ensembling, and Deep Learning Models

Code-mixing is a common linguistic phenomenon when words or syllables from two or more languages are used within a single sentence. Several works have been conducted on building such datasets and performing downstream tasks on them. Although it is very common to observe code-mixing with multiple languages, most datasets in this domain contain code-mixed data from only two languages. In this paper, we introduce OffMix-3L, a novel dataset for the task of offensive language identification which contains code-mixed data from 3 different languages. We also provide a comprehensive experimental analysis with several models and observe that BanglishBERT outperforms all the transformer-based models and also the generative model GPT-3.5 on OffMix-3L.

Downloads

Next from IJCNLP-AACL 2023

Toward Unified Controllable Text Generation via Regular Expression Instruction

Similar lecture

Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from IJCNLP-AACL 2023

Toward Unified Controllable Text Generation via Regular Expression Instruction

Similar lecture

Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation

Downloads