Spain

&quot;Recent progress in machine learning, on predictive tasks in particular, is usually claimed on the basis of performance comparisons.
Due to the limitation of application scenarios and computational resources, most people cannot evaluate their work on a large variety of tasks and datasets, and it has become tricky for both reviewers and readers to verify individual performance gains for a given approach.
In this paper we investigate possible approaches to achieve better efficiency on model benchmarking. 
For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model&#39;s goodness, from which to extrapolate to performance on other datasets. 
To this end, we adopt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests.
We propose certain modifications following the requirements of machine learning experiments, and present some initial results and insights in this paper.&quot;

ECAI 2020

Towards Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing

## Welcome to 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics!
This year, the conference is in Mexico City. NAACL was actually already planned for Mexico City in 2021, but due to the pandemic the entire conference was moved online. This year, finally, we get to go! So it is my sincere pleasure to welcome you to Mexico City, whether in person or virtually. Having the conference in Mexico City is a good opportunity to emphasize that NAACL is our flagship conference for ACL members not only in North America but also in Central and South America, even though NAACL has been bearing “North” in its name. At this year’s conference, we have a theme to match, with a theme track on the Languages of Latin America to showcase the linguistic diversity of the region.
 
The opportunity to present at NAACL should not depend on a researcher’s travel budget, or their family status. This is why it is so important to make virtual participation at NAACL as good an experience as possible – but we want to also provide a good experience for in-person participants. As a community, we are still working out the best way to do that. This year at NAACL, we are trying out a big virtual poster session ahead of the conference, with the hope that this will make make for a lively and interactive experience. At the same time, we are reducing virtual oral presentations, which seem to be particularly tricky to make to work well. A big thanks to the NAACL program chairs and to Luciana Benotti for all their ideas and work to improve the virtual experience. And participants, virtual as well as in-person: Please let us know what worked for you and what didn’t, so we can continue to improve hybrid conferences. 
 
I have been lucky to work with many amazing people. Without their insight, dedication and patience, and without the many hours of work they put in, NAACL would not have been possible. A huge thank you to the program chairs Helena Gomez, Kevin Duh, and Steve Bethard – you are the best! 
 
Finally, I would like to thank all authors, invited speakers and panelists, area chairs and reviewers, the volunteers organizing and chairing sessions, and all attendees, in-person and virtual. Thank you for helping us make NAACL 2024 come to life. 
 
Welcome and hope you all enjoy the conference! 
Katrin Erk 
The University of Texas at Austin 
NAACL-2024 General Chair 
*You can read the full Welcome message in the [Conference Handbook (downloadable)](https://drive.google.com/file/d/1H1NvW0VASQjkSCYgw3mr-3l4yYDSxz6B/view?usp=sharing)*

To access the event page you need to register [**here.**](http://acl.swoogo.com/naacl2024) 
Your access to the event page is limited based on your registration type. If you registered for workshops only, you will gain access to full workshops content on the day of the workshops program.

Please register!

NAACL 2024

2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

"Recent progress in machine learning, on predictive tasks in particular, is usually claimed on the basis of performance comparisons.
Due to the limitation of application scenarios and computational resources, most people cannot evaluate their work on a large variety of tasks and datasets, and it has become tricky for both reviewers and readers to verify individual performance gains for a given approach.
In this paper we investigate possible approaches to achieve better efficiency on model benchmarking. 
For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model's goodness, from which to extrapolate to performance on other datasets. 
To this end, we adopt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests.
We propose certain modifications following the requirements of machine learning experiments, and present some initial results and insights in this paper."

technical paper

ECAI2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI, including:




**An extensive programme of workshops, tutorials, and invited speakers**

**The Prestigious Applications of Intelligent Systems conference (PAIS)**

**The Frontiers in AI track sessions**

**The Lab To Market event**

**The Women in AI time**

**The EU Challenges forum**

… and many more!

ECAI 2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI.

We consider the issue of AI (Artificial Intelligence) evaluation in the light of risk management for the proper marketing of machinery. When a physical device is driven by AI, the question arises as to how to manage the risks associated with the autonomous functions. Indeed, the risk of physical damage to persons or property in the event of a ``wrong'' decision must be properly assessed. The study presents the advances needed to carry out relevant risk analyses for the deployment of safe intelligent machines, and offers good practice that leverages the knowledge in AI evaluation to optimize the safety of these machines.

Risk assessment of artificial intelligence in autonomous machines

This paper argues that at a certain point in research toward AGI, the problem may become well-enough theorized that a clear roadmap exists for achieving it, such that a Manhattan Project- like effort could greatly shorten the time to completion. If state actors perceive that this threshold has been crossed, their incentives around openness and international cooperation may shift rather suddenly, with serious implications for AI risks and the stability of international AI governance regimes. The paper characterizes how such a ‘runway’ period would be qualitatively different from preceding stages of AI research, and accordingly proposes a research program aimed at assessing how close the field of AI is to such a threshold—that is, it calls for the formulation of a ‘roadmap to the roadmap.’

Roadmap to a Roadmap: How Could We Tell When AGI is a ‘Manhattan Project’ Away?

The widespread use of experimental benchmarks in AIresearch  has  created  new  competition  and  collaboration  dynamics that are still poorly understood. In this paper we provide an innovative methodology to explore this dynamics and analyse the way different entrants in these competitions, from academia to tech giants,behave and react depending on their own or others’ achievements.We perform an analysis of over twenty popular benchmarks in AI,linking their underlying research papers. We identify links between researchers and institutions (i.e., communities) beyond the standard co-authorship relations, and we explore a series of hypotheses about their behaviour as well as some aggregated results in terms of activity, breakthroughs and efficiency. As a result, we detect and characterise the dynamics of research communities at different levels of abstraction, including organisation, affiliation, trajectories, results and activity.

The Scientometrics of AI Benchmarks: Unveiling the Underlying Mechanics of AI Research

The performance of the machine learning algorithms can be empirically evaluated via test campaigns: conventional testing approaches allow to obtain, from annotated databases, a first estimate of the operating environment of artificial intelligence systems. However, as those data are hardly I .i.d, additional scores could be useful to offer complementary performance assessment typically for intelligent systems performing critical functions. This paper proposes an approach to generate, without annotation phase, realistic corner cases to complete the exploration of the operating environment. The results of the proof of concept provided for optical character recognition highlight the interest of test data generation and automatic scenario selection methods, including GAN and adversarial approaches, to address this evaluation need.

Generating corner cases for crash-testing deep networks

"The field of Artificial Intelligence (AI) and, in particular, the Machine Learning area, counts on a wide range of performance metrics and benchmark data sets to assess the problem-solving effec- tiveness of its solutions. However, the appearance of research centres, projects or institutions addressing AI solutions from a multidisci- plinary and multi-stakeholder perspective suggests a new approach to assessment comprising ethical guidelines, reports or tools and frame- works to help both academia and business to move towards a re- sponsible conceptualisation of AI. They all highlight the relevance of three key aspects: (i) enhancing cooperation among the different stakeholders involved in the design, deployment and use of AI; (ii) promoting multidisciplinary dialogue, including different domains of expertise in this process; and (iii) fostering public engagement to maximise a trusted relation with new technologies and practitioners. In this paper, we introduce the Observatory on Society and Artifi- cial Intelligence (OSAI), an initiative grew out of the project AI4EU aimed at stimulating reflection on a broad spectrum of issues of AI (ethical, legal, social, economic and cultural). In particular, we de- scribe our work in progress around OSAI and suggest how this and similar initiatives can promote a wider appraisal of progress in AI. This will give us the opportunity to present our vision and our modus operandi to enhance the implementation of these three fundamental dimensions.
"

Progressing Towards Responsible AI

This work presents an operational definition that facilitates a common understanding of AI and its measurability, paving the way for a transparent and comparable monitoring activity in the context of AI Watch, the European Commission’s knowledge service to monitor the development, uptake and impact of artificial intelligence (AI) for Europe. The operational definition is constituted by a taxonomy and a list of related keywords. The method that we propose is iterative, considering that AI is a dynamic field, so its definition should be updated over time to capture the rapid AI evolution. The method consists of the following steps: (i) qualitative analysis of AI definitions and subdomains emanating from reports with academic, industrial and policy perspectives, (ii) selection of definition as starting point, taxonomy formation, and identification of representative keywords in AI with a natural language processing method, and (iii) taxonomy and keywords validation. This results in a unique taxonomy that represents and interconnects all the AI subdomains from political, research and industrial perspective and enables the efficient mapping of the AI landscape of economic agents across different technological areas.

Setting the Boundaries of the AI Landscape: An Operational Definition for the European Commission’s AI Watch

This short paper aims at introducing some labelling concepts developed and used by the authors. Indeed, developing a conceptual framework for interdisciplinary research in history is a much-needed strategy in order to ensure that historians use all vestiges from the past regardless their origin or support for the construction of historical discourse. Fixing the semantics of historical knowledge is the first unavoidable step to build a new scenario in which NLP tasks enable more efficient data gathering and exploitation.

The Semantics of Historical Knowledge. Labelling Strategies for Interdisciplinary and Digital Research in History

Presentation is about our chatbot creation platform and its dialog management techniques. The framework enables creating chatbots without coding using intent-slot model. It contains intent detection and dialog act classifier modules to understand user utterances more accurately. Also the framework includes and analytics tool to monitor and cluster past conversation data to discover topics and training sentences.

Domain-Independent, Task-Oriented Chatbot Creation and Conversation Policy Management Framework

"Static word embeddings, like word2vec or GloVe, are often assessed when solving syntactic and semantic analogies. Among the latter, we are interested in relations that one would find in lexical-semantic knowledge bases like WordNet, also covered in analogy test sets for English. This paper describes the creation of a new test for assessing Portuguese word embeddings, dubbed TALES, with an exclusive focus on lexical-semantic relations, acquired from lexical resources in Portuguese.
It further reports on the performance of methods previously used for solving analogies, with pre-trained Portuguese word embeddings, when applied to the created dataset, an experiment that revealed that TALES is challenging to solve.
Results achieved are briefly discussed, with conclusions that may be useful for developing new approaches for this problem, possibly new embeddings, as well as future versions of TALES."

 TALES: Test Set of Portuguese Lexical-Semantic Relations for Assessing Word Embeddings

"In this presentation we will show the application of the Thematic Theory and the Thematic Progression Theory to automatically annotate discourse features, which enable the inference of the implicit conceptual schemas of texts.
"

Downloads

Next from ECAI 2020

Risk assessment of artificial intelligence in autonomous machines

Similar lecture

ALBA: Adaptive Language-Based Assessments for Mental Health

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ECAI 2020

Risk assessment of artificial intelligence in autonomous machines

Similar lecture

ALBA: Adaptive Language-Based Assessments for Mental Health

Downloads