Spain

&quot;In this paper we introduce a methodology for identifying early warning signs of transformative progress in AI, to aid anticipatory governance and research prioritisation. We propose using expert elicitation methods to identify milestones in
AI progress, followed by collaborative causal mapping to identify key milestones which underpin several others. We call these key milestones ‘canaries’ based on the colloquial phrase ‘canary in a coal mine’ to describe advance warning of an extreme event: in this case, advance warning of transformative AI. After describing and motivating our proposed methodology, we present results from an initial implementation to identify canaries for progress towards high-level machine intelligence (HLMI). We conclude by discussing the limitations of this method, possible future improvements, and how we hope it can be used to improve monitoring of future risks from AI progress. &quot;

ECAI 2020

Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics.

**Whova App** 
Stay in touch with your fellow conference attendees via the [Whova App](https://whova.com/portal/webapp/nacon_202106/)

**Conference Structure**
https://2021.naacl.org/blog/conference-structure/

**Walkthrough video of how to NAACL 2021** 

Please take a moment to view this video explaining how to navigate the platform, attend sessions network with other attendees. 


<figure class="video_container">
  <iframe src="https://screencast-o-matic.com/watch/crhwbGVh3vx?v=6&ff=1&title=0&controls=1" width=640  height=350  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

<br>
<br>

NAACL 2021

"In this paper we introduce a methodology for identifying early warning signs of transformative progress in AI, to aid anticipatory governance and research prioritisation. We propose using expert elicitation methods to identify milestones in
AI progress, followed by collaborative causal mapping to identify key milestones which underpin several others. We call these key milestones ‘canaries’ based on the colloquial phrase ‘canary in a coal mine’ to describe advance warning of an extreme event: in this case, advance warning of transformative AI. After describing and motivating our proposed methodology, we present results from an initial implementation to identify canaries for progress towards high-level machine intelligence (HLMI). We conclude by discussing the limitations of this method, possible future improvements, and how we hope it can be used to improve monitoring of future risks from AI progress. "

technical paper

ECAI2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI, including:




**An extensive programme of workshops, tutorials, and invited speakers**

**The Prestigious Applications of Intelligent Systems conference (PAIS)**

**The Frontiers in AI track sessions**

**The Lab To Market event**

**The Women in AI time**

**The EU Challenges forum**

… and many more!

ECAI 2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI.

"Competitions are a proven and cost-effective method to quickly develop new disruptive technologies for new markets. However, artificial intelligence and robotics competitions are usually organized independently, in silos. To get the most out of competitions, evaluation of robots should be modular (tasks are designed to assess independently different technological building blocks of the robotic systems) and open (evaluation tools and data must be publicly and freely available online). This not only meets the benchmarking needs of end users (mostly interested in performance measurements of the complete system), but also those of integrators and developers of intelligent components (camera, lidar, radar, actuators, etc.).
With this objective in mind, the H2020 METRICS project (2020-2023) organizes competitions in four application areas (Healthcare, Infrastructure Inspection and Maintenance, Agri-Food, and Agile Production) relying on both physical testing facilities (field evaluation campaign) and virtual testing facilities (data-based evaluation campaign) to mobilize, in addition to the European robotics community, the artificial intelligence one.
This presentation outlines this approach and paves the way for a new robotics and artificial intelligence competition paradigm."

AI evaluation campaigns during robotics competitions: the METRICS paradigm

"The ROSE Challenge is the first robotics and artificial intelligence competition worldwide to implement a third-party performance evaluation of intra-row weeding robots in real conditions, under comparable conditions, to guarantee a credible and objective assessment of their effectiveness. This article reports on the design and
validation of testing facilities for this competition, which presents a particular complexity : the experiments take place outdoors and act on living things (crops and weeds). Moreover, it implies to guarantee repeatable experimental conditions for a comparable and equitable evaluation. The article also discusses the opportunity that these competitions represent to define testing facilities in a consensual way.
The method it proposes is very widely applicable to different fields of intelligent systems applications."

Design and validation of testing facilities for weeding robots as part of ROSE challenge

"Recent progress in machine learning, on predictive tasks in particular, is usually claimed on the basis of performance comparisons.
Due to the limitation of application scenarios and computational resources, most people cannot evaluate their work on a large variety of tasks and datasets, and it has become tricky for both reviewers and readers to verify individual performance gains for a given approach.
In this paper we investigate possible approaches to achieve better efficiency on model benchmarking. 
For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model's goodness, from which to extrapolate to performance on other datasets. 
To this end, we adopt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests.
We propose certain modifications following the requirements of machine learning experiments, and present some initial results and insights in this paper."

Towards Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing

We consider the issue of AI (Artificial Intelligence) evaluation in the light of risk management for the proper marketing of machinery. When a physical device is driven by AI, the question arises as to how to manage the risks associated with the autonomous functions. Indeed, the risk of physical damage to persons or property in the event of a ``wrong'' decision must be properly assessed. The study presents the advances needed to carry out relevant risk analyses for the deployment of safe intelligent machines, and offers good practice that leverages the knowledge in AI evaluation to optimize the safety of these machines.

Risk assessment of artificial intelligence in autonomous machines

This paper argues that at a certain point in research toward AGI, the problem may become well-enough theorized that a clear roadmap exists for achieving it, such that a Manhattan Project- like effort could greatly shorten the time to completion. If state actors perceive that this threshold has been crossed, their incentives around openness and international cooperation may shift rather suddenly, with serious implications for AI risks and the stability of international AI governance regimes. The paper characterizes how such a ‘runway’ period would be qualitatively different from preceding stages of AI research, and accordingly proposes a research program aimed at assessing how close the field of AI is to such a threshold—that is, it calls for the formulation of a ‘roadmap to the roadmap.’

Roadmap to a Roadmap: How Could We Tell When AGI is a ‘Manhattan Project’ Away?

The widespread use of experimental benchmarks in AIresearch  has  created  new  competition  and  collaboration  dynamics that are still poorly understood. In this paper we provide an innovative methodology to explore this dynamics and analyse the way different entrants in these competitions, from academia to tech giants,behave and react depending on their own or others’ achievements.We perform an analysis of over twenty popular benchmarks in AI,linking their underlying research papers. We identify links between researchers and institutions (i.e., communities) beyond the standard co-authorship relations, and we explore a series of hypotheses about their behaviour as well as some aggregated results in terms of activity, breakthroughs and efficiency. As a result, we detect and characterise the dynamics of research communities at different levels of abstraction, including organisation, affiliation, trajectories, results and activity.

The Scientometrics of AI Benchmarks: Unveiling the Underlying Mechanics of AI Research

The performance of the machine learning algorithms can be empirically evaluated via test campaigns: conventional testing approaches allow to obtain, from annotated databases, a first estimate of the operating environment of artificial intelligence systems. However, as those data are hardly I .i.d, additional scores could be useful to offer complementary performance assessment typically for intelligent systems performing critical functions. This paper proposes an approach to generate, without annotation phase, realistic corner cases to complete the exploration of the operating environment. The results of the proof of concept provided for optical character recognition highlight the interest of test data generation and automatic scenario selection methods, including GAN and adversarial approaches, to address this evaluation need.

Generating corner cases for crash-testing deep networks

"The field of Artificial Intelligence (AI) and, in particular, the Machine Learning area, counts on a wide range of performance metrics and benchmark data sets to assess the problem-solving effec- tiveness of its solutions. However, the appearance of research centres, projects or institutions addressing AI solutions from a multidisci- plinary and multi-stakeholder perspective suggests a new approach to assessment comprising ethical guidelines, reports or tools and frame- works to help both academia and business to move towards a re- sponsible conceptualisation of AI. They all highlight the relevance of three key aspects: (i) enhancing cooperation among the different stakeholders involved in the design, deployment and use of AI; (ii) promoting multidisciplinary dialogue, including different domains of expertise in this process; and (iii) fostering public engagement to maximise a trusted relation with new technologies and practitioners. In this paper, we introduce the Observatory on Society and Artifi- cial Intelligence (OSAI), an initiative grew out of the project AI4EU aimed at stimulating reflection on a broad spectrum of issues of AI (ethical, legal, social, economic and cultural). In particular, we de- scribe our work in progress around OSAI and suggest how this and similar initiatives can promote a wider appraisal of progress in AI. This will give us the opportunity to present our vision and our modus operandi to enhance the implementation of these three fundamental dimensions.
"

Progressing Towards Responsible AI

This work presents an operational definition that facilitates a common understanding of AI and its measurability, paving the way for a transparent and comparable monitoring activity in the context of AI Watch, the European Commission’s knowledge service to monitor the development, uptake and impact of artificial intelligence (AI) for Europe. The operational definition is constituted by a taxonomy and a list of related keywords. The method that we propose is iterative, considering that AI is a dynamic field, so its definition should be updated over time to capture the rapid AI evolution. The method consists of the following steps: (i) qualitative analysis of AI definitions and subdomains emanating from reports with academic, industrial and policy perspectives, (ii) selection of definition as starting point, taxonomy formation, and identification of representative keywords in AI with a natural language processing method, and (iii) taxonomy and keywords validation. This results in a unique taxonomy that represents and interconnects all the AI subdomains from political, research and industrial perspective and enables the efficient mapping of the AI landscape of economic agents across different technological areas.

Setting the Boundaries of the AI Landscape: An Operational Definition for the European Commission’s AI Watch

This short paper aims at introducing some labelling concepts developed and used by the authors. Indeed, developing a conceptual framework for interdisciplinary research in history is a much-needed strategy in order to ensure that historians use all vestiges from the past regardless their origin or support for the construction of historical discourse. Fixing the semantics of historical knowledge is the first unavoidable step to build a new scenario in which NLP tasks enable more efficient data gathering and exploitation.

Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

Downloads

Next from ECAI 2020

AI evaluation campaigns during robotics competitions: the METRICS paradigm

Similar lecture

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ECAI 2020

AI evaluation campaigns during robotics competitions: the METRICS paradigm

Similar lecture

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Downloads