Ireland

Models are increasing in size and complexity in the hunt for SOTA. But what if those 2% increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models from a multilingual perspective.

ACL 2022

Assessment of Massively Multilingual Sentiment Classifiers

sentiment datasets

multilingual text representations

sentiment classification

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

workshop paper

Masked language models (MLMs) are pre-trained with a denoising objective that is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two complementing strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. We test our models on 15 different Twitter datasets for social meaning detection. Our methods achieve 2.34% F1 over a competitive baseline, while outperforming domain-specific language models pre-trained on large datasets. Our methods also excel in few-shot learning: with only 5% of training data (severely few-shot), our methods enable an impressive 68.54% average F1. The methods are also language agnostic, as we show in a zero-shot setting involving six datasets from three different languages.

Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning

Over the years, the review helpfulness prediction task has been the subject of several works, but remains being a challenging issue in Natural Language Processing, as results vary a lot depending on the domain, on the adopted features and on the chosen classification strategy. This paper attempts to evaluate the impact of
content features and classification methods for two different domains. In particular, we run our experiments for a low resource language – Portuguese –, trying to establish a benchmark for this language. We show that simple features and classical classification methods are powerful for the task of helpfulness prediction, but are largely outperformed by a convolutional neural network-based solution.


Evaluating Content Features and Classification Methods for Helpfulness Prediction of Online Reviews: Establishing a Benchmark for Portuguese

Our system, IUCL, participated in the WASSA 2022 Shared Task on Empathy Detection and Emotion Classification. Our main goal in building this system is to investigate how the use of demographic attributes influences performance. Our (official) results show that our text-only systems perform very competitively, ranking first in the empathy detection task, reaching an average Pearson correlation of 0.54, and second in the emotion classification task, reaching a Macro-F of 0.572. Our systems that use both text and demographic data are less competitive.

IUCL at WASSA 2022 Shared Task: A Text-only Approach to Empathy and Emotion Detection

This presentation describes the continual pre-training method for the masked language model (MLM) to enhance the DeBERTa pre-trained language model. Several training strategies are designed to further improve the final downstream performance including the data augmentation with the supervised transfer, child-tuning training, and the late fusion method. Extensive experiments on the emotional classification dataset show that the proposed method outperforms other state-of-the-art methods, demonstrating our method's effectiveness. Moreover, our submission ranked Top-1 with all metrics in the evaluation phase for the Emotion Classification task.

Continuing Pre-trained Model with Multiple Training Strategies for Emotional Classification

In our paper, we describe our approach to the Emotion Classification Shared Task (Track-2). We propose a deep multi-channel hybrid model, where we combine representations from RoBERTa, The NRC emotion intensity lexicon and Emotion-Enriched word embeddings. Results attest that this methodology of combining emotion-specific features to a Transformer model (RoBERTa) provides increased performance over the Vanilla RoBERTa model.

Leveraging Emotion-Specific features to improve Transformer performance for Emotion Classification

Detecting emotions in languages is important to accomplish a complete interaction between humans and machines. This paper describes our contribution to the WASSA 2022 shared task which handles this crucial task of emotion detection. We have to identify the following emotions: sadness, surprise, neutral, anger, fear, disgust, joy based on a given essay text. We are using an ensemble of ELECTRA and
BERT models to tackle this problem achieving an F1 score of 62.76%.

Transformer based ensemble for emotion detection

Computational comprehension and identifying emotional components in language have been critical in enhancing human-computer connection in recent years. The WASSA 2022 Shared Task introduced four tracks and released a dataset of news stories: Track-1 for Empathy and Distress Prediction, Track-2 for Emotion classification, Track-3 for Personality prediction, and Track-4 for Interpersonal Reactivity Index prediction at the essay level. This paper describes our participation in the WASSA 2022 shared task on the tasks mentioned above. We developed multi-task deep learning methods to address Tracks 1 and 2 and machine learning models for Track 3 and 4. Our developed systems achieved average Pearson scores of 0.483, 0.05, and 0.08 for Track 1, 3, and 4, respectively, and a macro F1 score of 0.524 for Track 2 on the test set. We ranked 8th, 11th, 2nd, and 2nd for tracks 1, 2, 3, and 4 respectively.

Team IITP-AINLPML at WASSA 2022: Empathy Detection, Emotion Classification and Personality Detection

In this presentation, we adopt the prompt-based method with the pre-trained language model to accomplish the Personality Prediction (PER) and Reactivity Index Prediction (IRI) tasks. Specifically, the prompt is designed to provide knowledge of the extra personalized information for enhancing the pre-trained model. Data augmentation and model ensemble are adopted for obtaining better results. Extensive experiments are performed, which shows the effectiveness of the proposed method. On the final submission, our system achieves a Pearson Correlation Coefficient of 0.2301 and 0.2546 on Track 3 and Track 4 respectively. We ranked 1-st on both sub-tasks.

Prompt-based Pre-trained Model for Personality and Interpersonal Reactivity Prediction

Machine-learned models for author profiling in social media often rely on data acquired via self-reporting-based psychometric tests (ques- tionnaires) filled out by social media users. This is an expensive but accurate data collec- tion strategy. Another, less costly alternative, which leads to potentially more noisy and bi- ased data, is to rely on labels inferred from publicly available information in the profiles of the users, for instance self-reported diag- noses or test results. In this paper, we explore a third strategy, namely to directly use a corpus of items from validated psychometric tests as training data. Items from psychometric tests often consist of sentences from an I-perspective (e.g., “I make friends easily.”). Such corpora of test items constitute ‘small data’, but their avail- ability for many concepts is a rich resource. We investigate this approach for personality profil- ing, and evaluate BERT classifiers fine-tuned on such psychometric test items for the big five personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism) and analyze various augmentation strategies regard- ing their potential to address the challenges coming with such a small corpus. Our eval- uation on a publicly available Twitter corpus shows a comparable performance to in-domain training for 4/5 personality traits with T5-based data augmentation.

Items from Psychometric Tests as Training Data for Personality Profiling Models of Twitter Users

Today data-to-text systems are used as commercial solutions for automated text production. Therefore, they represent a new technology of writing. This new technology requires the author, as an act of writing, both to configure a system that then takes over the transformation into a real text, but also to maintain strategies of traditional writing. What should an environment look like, where a human guides a machine to write texts? Based on a comparison of the NLG pipeline architecture with the results of the research on the human writing process, this paper attempts to work out which tasks need to be solved and which strategies are necessary to produce good texts in this environment. From this synopsis, principles for the design of data-to-text systems as a functioning writing environment are then derived.

Assessment of Massively Multilingual Sentiment Classifiers

Downloads

Next from ACL 2022

Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning

Similar lecture

On Efficiently Acquiring Annotations for Multilingual Models

Assessment of Massively Multilingual Sentiment Classifiers

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2022

Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning

Similar lecture

On Efficiently Acquiring Annotations for Multilingual Models

Downloads