Kunikazu Kobayashi, Koji Nakano, Takashi Kuremoto and Masanao Obayashi (February 1st 2010). Objective-based Reinforcement Learning System for Cooperative Behavior Acquisition, Application of Machine Learning, Yagang Zhang, IntechOpen, DOI: 10.5772/8615. 

Yoshiyuki Tajima and Takehisa Onisawa (February 1st 2010). Model-based Reinforcement Learning with Model Error and Its Application, Application of Machine Learning, Yagang Zhang, IntechOpen, DOI: 10.5772/8607. 

Lyudmila Alexeyeva (May 23rd 2018). General Functions Method in Transport Boundary Value Problems of Elasticity Theory, Differential Equations - Theory and Current Research, Terry E. Moschandreou, IntechOpen, DOI: 10.5772/intechopen.74538. 

New Zealand

Predictive auxiliary tasks have been shown to improve performance in numerous reinforcement learning works, however, this effect is still not well understood. The primary purpose of the work presented here is to investigate the impact that an auxiliary task&#39;s prediction timescale has on the agent&#39;s policy performance. We consider auxiliary tasks which learn to make on-policy predictions using temporal difference learning. We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target, which we refer to as temporal difference autoencoders (TD-AE). We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales. While we do not observe a clear relationship between the prediction timescale on performance, we make the following observations: 1) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm, 2) in some cases temporally extended TD-AE performs better than a straight autoencoder, 3) performance with auxiliary tasks is sensitive to the weight placed on the auxiliary loss, 4) despite this sensitivity, auxiliary tasks improved performance without extensive hyper-parameter tuning. Our overall conclusions are that TD-AE increases the robustness of the A2C algorithm to the trajectory length and while promising, further study is required to fully understand the relationship between auxiliary task prediction timescale and the agent&#39;s performance.


AAMAS 2020

Work in Progress: Temporally Extended Auxiliary Tasks

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-23 is the Thirty-Seventh AAAI Conference on Artificial Intelligence. The theme of this conference is to create collaborative bridges within and beyond AI. Like previous AAAI conferences, AAAI-23 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and two new activities: a Bridge Program and a Lab Program. Many of these activities are tailored to the theme of bridges and all are selected according to the highest standards, with additional programs for students and young researchers. 
AAAI is providing you with a conference planner, which you can use to help organize your itinerary of activities. This includes talks to attend in person, talks to attend remotely, breaks with colleagues and your site seeing activities. To access this conference planner, please go to [https://aaai-2023.takemobi.io](https://aaai-2023.takemobi.io).

In order to access this site, you need to register. If you haven't already, please register [here](https://aaai.org/Conferences/AAAI-23/registration/).


AAAI 2023

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines.

Predictive auxiliary tasks have been shown to improve performance in numerous reinforcement learning works, however, this effect is still not well understood. The primary purpose of the work presented here is to investigate the impact that an auxiliary task's prediction timescale has on the agent's policy performance. We consider auxiliary tasks which learn to make on-policy predictions using temporal difference learning. We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target, which we refer to as temporal difference autoencoders (TD-AE). We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales. While we do not observe a clear relationship between the prediction timescale on performance, we make the following observations: 1) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm, 2) in some cases temporally extended TD-AE performs better than a straight autoencoder, 3) performance with auxiliary tasks is sensitive to the weight placed on the auxiliary loss, 4) despite this sensitivity, auxiliary tasks improved performance without extensive hyper-parameter tuning. Our overall conclusions are that TD-AE increases the robustness of the A2C algorithm to the trajectory length and while promising, further study is required to fully understand the relationship between auxiliary task prediction timescale and the agent's performance.


technical paper

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

Describing an accurate simulation model of the driving behavior of real-world vehicles is a laborious or even impossible task, because a driver reacts to a dynamically changing environment. As multiple external factors determine driving behavior, it is usually difficult to obtain an accurate model, owing to a lack of sensors or inability to collect data. In this paper, we propose a novel technique to combine driving behavior in vehicular Internet of Things (IoT) big data with a multi-agent system. This enables correct and scalable simulation without modeling the behavior of vehicular IoT devices or the environment. We develop an extensible simulation framework, called FlowSim, that demonstrates the application of our technique for a simulation of camera-image data collection from connected cars.


Simulating Vehicular IoT Applications by Combining a Multi-agent System and Big Data

"Patients are often required to follow a medical treatment after discharge, e.g., for a chronic condition, rehabilitation after surgery, or for cancer survivor therapies. The need to adapt to new lifestyles, medication, and treatment routines, can produce an individual burden to the patient, who is often at home without the full support of healthcare professionals. Although technological solutions -in the form of mobile apps and wearables- have been proposed to mitigate these issues, it is essential to consider individual characteristics, preferences, and the context of a patient in order to oer personalized and effective support. The specific events and circumstances linked to an individual prole can be abstracted as a patient trajectory, which can contribute to a better understanding of the patient, her needs, and the most appropriate personalized support. Although patient trajectories have been studied for different illnesses and conditions, it remains challenging to effectively use them as the basis for data analytics methodologies in decentralized eHealth systems. In this work, we present a novel approach based on the multi-agent paradigm, considering patient trajectories as the cornerstone of a methodology for modelling eHealth support systems. In this design, semantic representations of individual treatment pathways are used in order to exchange patient-relevant information, potentially fed to AI systems for prediction and classification tasks. This paper describes the major challenges in this scope, as well as the design principles of the proposed agent-based architecture, including an example of its
use through a case scenario for cancer survivors support."


Agent-based modeling for Ontology-driven analysis of patient trajectories

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible.
In this work, we focus on learning to coordinate.
Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. 
We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks with Bernoulli-distributed rewards. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in applications with sparse neighbourhood structures.


Thompson Sampling for Factored Multi-Agent Bandits

"Reinforcement learning (RL) is a powerful learning paradigm in which agents can learn to maximize sparse and delayed reward signals. Although RL has had many impressive successes in complex domains, learning can take hours, days, or even years of training data. A major challenge of contemporary RL research is to discover how to learn with less data. Previous work has shown that domain information can be successfully used to shape the reward; by adding additional reward information, the agent can learn with much less data. Furthermore, if the reward is constructed from a potential function, the optimal policy is guaranteed to be unaltered. While such potential-based reward shaping (PBRS) holds promise, it is limited by the need for a well-defined potential function. Ideally, we would like to be able to take arbitrary advice from a human or other agent and improve performance without affecting the optimal policy. The recently introduced dynamic potential based advice (DPBA) method tackles this challenge by admitting arbitrary advice from a human or other agent and improves performance without affecting the optimal policy. The main contribution of this paper is to expose, theoretically and empirically, a flaw in DPBA. 
Alternatively, to achieve the ideal goals, we present a simple method called policy invariant explicit shaping (PIES) and show theoretically and empirically that PIES succeeds where DPBA fails."


Useful Policy Invariant Shaping from Arbitrary Advice

We give a brief overview of the annual Multi-Agent Programming Contest (MAPC). In this competition, we try to provide a set of challenging problems and encourage participants to solve these with agent-based frameworks and technologies. After introducing the MAPC, we elaborate on the latest scenario and provide some lessons we learned during and after the contest of 2019.


Multi-Agent Programming Contest 2019

"This presentation shows and evaluates a family of AlphaZero value targets, subsuming previous variants and introducing AlphaZero with greedy backups (A0GB).
Current state-of-the-art algorithms for playing board games use sample-based planning, such as Monte Carlo Tree Search (MCTS), combined with deep neural networks (NN) to approximate the value function. These algorithms, of which AlphaZero is a prominent example, are computationally extremely expensive to train, due to their reliance on many neural network evaluations. This limits their practical performance. 
We improve the training process of AlphaZero by using more effective training targets for the neural network. We introduce a family of training targets, covering the original AlphaZero training target as well as the soft-Z and \emph{A0C} variants from the literature. We demonstrate that A0GB, using a specific new value target from this family, is able to find the optimal policy in a small tabular domain, whereas the original AlphaZero target fails to do so.
In addition, we show that soft-Z, A0C and A0GB achieve better performance and faster training than the original AlphaZero target on two benchmark board games (Connect-Four and Breakthrough)."


Value targets in off-policy AlphaZero: a new greedy backup

Realistically modelling behaviour and interaction of heterogeneous road users (pedestrians and vehicles) in mixed-traffic zones (a.k.a. shared spaces) is challenging. The dynamic nature of the environment, heterogeneity of transport modes, and the absence of classical traffic rules make realistic microscopic traffic simulation hard problems. Existing multi-agent-based simulations of shared spaces largely use an expert-based approach, combining a symbolic (e.g. rule-based) modelling and reasoning paradigm (e.g. using BDI representations of beliefs and plans) with the hand-crafted encoding of the actual decision logic. More recently, deep learning (DL) models are largely used to derive and predict trajectories based on e.g. video data. In-depth studies comparing these two kinds of approaches are missing. In this work, we propose an expert-based model called GSFM that combines Social Force Model and Game theory and a DL model called LSTM-DBSCAN that manipulates Long Short-Term Memories and density-based clustering for multi-agent trajectory prediction. We create a common framework to run these two models in parallel to guarantee a fair comparison. Real-world mixed traffic data from shared spaces of different layout are used to calibrate/train and evaluate the models. The empirical results imply that both models can generate realistic predictions, but they differ in the way of handling collisions and mimicking heterogeneous behaviour. Via a thorough study, we draw the conclusion of their respective strengths and weaknesses. 


Trajectory Modelling in Shared Spaces: Expert-Based vs. Deep Learning Approach?

This presentation introduces a new cooperative multi-agent approach for segmenting brain Magnetic Resonance Images (MRIs). MRIs are manually processed by human radiology experts for the identification of many diseases and the monitoring of their evolution. However, such a task is time-consuming and depends on expert decision, which can be affected by many factors. Therefore, various types of research were and are still conducted to automate MRI processing, mainly MRI segmentation. The approach presented in this paper, without any parametrization or prior knowledge, uses a set of situated agents, locally interacting to segment images according to two main phases: the detection of discontinuities and the detection of similarities. An implementation of this approach was tested on phantom brain MR images to assess the results and prove its efficiency. Experimental results ensure a minimum of 89\% Dice coefficient with increasing values of the noise and the intensity non-uniformity.


A Cooperative Approach Based on Local Detection of Similarities and Discontinuities for Brain MR Images Segmentation

**Please click on the button bellow to see this lecture on SlidesLive:**

[![](https://assets.underline.io/uploads/markdown_image/1/image/08e10ad349922f32e7322b77b8df9019.png)](https://slideslive.com/38946802/boston-dynamics)

**Abstract:**

Distributed agent-based simulations often suffer from an imbalance in computational load, leading to a suboptimal use of resources. This happens when part of the computational resoures are waiting idle for another process to finish. Self-adaptive load-balancing algorithms have been developed to use these resources more optimally. These algorithms are typically implemented ad-hoc, making re-usability and maintenance difficult. In this work, we present a generic self-adaptive framework. This methodology is evaluated with the Acsim framework on two simulations: a micro-traffic simulation and a cellular automata simulation. For each of these scenarios a scalable and adaptive load-balancing algorithm is implemented, showing significant improvements in execution time of the simulation.


Adaptivity in distributed agent-based simulation

Although plenty of qualitative logical frameworks have been pro- posed to evaluate and model trust in multi-agent sittings, these ap- proaches generally ignore reasoning about quantitative aspects such as degrees of trust. In this paper, we address this limitation from the modelling and verification perspectives. We start by constructing TCTLG , a logical language to represent the quantitative aspect of trust and present a set of its reasoning postulates. Moreover, we develop and implement a new symbolic model checking algorithm and open source tool for quantifying the relationships among the interacting agents. Finally, we investigate the complexity and evaluate our approach using a case study in the health care domain.


Work in Progress: Temporally Extended Auxiliary Tasks

Downloads

Next from AAMAS 2020

Simulating Vehicular IoT Applications by Combining a Multi-agent System and Big Data

Similar lecture

Zero-Shot Assistance in Sequential Decision Problems

Work in Progress: Temporally Extended Auxiliary Tasks

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2020

Simulating Vehicular IoT Applications by Combining a Multi-agent System and Big Data

Similar lecture

Zero-Shot Assistance in Sequential Decision Problems

Downloads