New Zealand


As one of the solutions to the decentralized partially observable Markov decision process (Dec-POMDP) problems, the value decomposition method has achieved significant results recently. However, most value decomposition methods require the fully observable state of the environment during training, but this is not feasible in some scenarios where only incomplete and noisy observations can be obtained. Therefore, we propose a novel value decomposition framework, named State Inference for value DEcomposition (SIDE), which eliminates the need to know the global state by simultaneously seeking solutions to the two problems of optimal control and state inference. SIDE can be extended to any value decomposition method to tackle partially observable problems. By comparing with the performance of different algorithms in StarCraft II micromanagement tasks, we verified that though without accessible states, SIDE can infer the current state that contributes to the reinforcement learning process based on past local observations and even achieve superior results to many baselines in some complex scenarios.

AAMAS 2022

SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning

multi-agent learning

variational inference

reinforcement learning

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).
<br>
<br>



A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

technical paper


Intrinsic rewards can improve exploration in reinforcement learning, but often suffer from instability caused by non-stationary reward shaping and strong dependency on hyperparameters. In this work, we introduce Decoupled RL (DeRL) as a general framework which trains separate policies for intrinsically motivated exploration and exploitation. Such decoupling allows DeRL to leverage the benefits of intrinsic rewards for exploration while demonstrating improved robustness and sample efficiency. We evaluate DeRL algorithms in two sparse-reward environments with multiple types of intrinsic rewards. Our results show that DeRL is more robust to varying scale and speed of decay of intrinsic rewards and converges to the same evaluation returns than intrinsically motivated baselines in fewer interactions. Lastly, we discuss the challenge of distribution shift and show that divergence constraint regularisers can successfully minimise instability caused by divergence of exploration and exploitation policies.

Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration


Multi-agent reinforcement learning holds the key for solving complex tasks that demand coordination of learning agents. However, strong coordination often leads to expensive exploration over the exponentially large state-action space. A powerful approach is to decompose team works into roles, which are ideally assigned to agents with the relevant skills. Training agents to adaptively choose and play emerging roles in team thus allows the team to scale to complex tasks and quickly adapt to changing environments. These promises, however, have not been fully realized by current role-based multi-agent reinforcement learning methods as they assume either a pre-defined role structure or a fixed team size. We propose a framework to learn role assignment and transfer across team sizes. In particular, we train a role assignment network for small teams by demonstration and transfer the network to larger teams, which continue to learn through interaction with the environment. We demonstrate that re-using the role-based credit assignment structure can foster the learning process of larger reinforcement learning teams to achieve tasks requiring different roles. Our proposal outperforms competing techniques in enriched role-enforcing Prey-Predator games and in new scenarios in the StarCraft II Micro-Management benchmark.

Learning to Transfer Role Assignment Across Team Sizes


Identifying datapoints that substantially differ from normality is the task of anomaly detection (AD).
While AD has gained widespread attention in rich data domains such as images, videos, audio and text, it has has been studied less frequently in the context of reinforcement learning (RL). This is due to the additional layer of complexity that RL introduces through sequential decision making. Developing suitable anomaly detectors for RL is of particular importance in safety-critical scenarios where acting on anomalous data could result in hazardous situations. 
In this work, we address the question of what AD means in the context of RL. We found that current research trains and evaluates on overly simplistic and unrealistic scenarios which reduce to classic pattern recognition tasks. We link AD in RL to various fields in RL such as lifelong RL and generalization. We discuss their similarities, differences, and how the fields can benefit from each other. Moreover, we identify non-stationarity to be one of the key drivers for future research on AD in RL and make a first step towards a more formal treatment of the problem by framing it in terms of the recently introduced block contextual Markov decision process.
Finally, define a list of practical desiderata for future problems in the field.

Towards Anomaly Detection in Reinforcement Learning


Interactive dynamic influence diagrams (I-DIDs) are a general framework
for multiagent sequential decision making under uncertainty. Due to the model complexity, a significant amount of research has been invested into solving the model through various types of either exact or approximate algorithms. However, there is no tool that allows users to specify the algorithm parameters and visualise the model solutions. In this demo, we develop an interactive I-DID system
that implements most of the state-of-art I-DID algorithms and develops a new type of algorithms based on evolutionary computation. In particular, we propose a multi-population genetic algorithm for solving the I-DID models and automate the generation of behavioural models in the solutions. This demo will facilitate the
I-DID research development and practical applications, and elicit a new wave of I-DID solutions based on evolutionary algorithms.

Ev-IDID: Enhancing Solutions to Interactive Dynamic Influence Diagrams through Evolutionary Algorithms


Swarming behaviours are ubiquitous in nature, and an important topic to teach and learn. In the learning sciences domain of complex systems understanding, studies have shown that novices tend to assume that centralized control and leadership should exist whenever they see complex patterns. In this paper, we present an educational framework consisting of a swarm of virtual and/or physical agents (robots) as a game-based learning environment to introduce the underlying rules of swarm behaviours. 


SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning

Downloads

Next from AAMAS 2022

Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration

Similar lecture

Emergent Cooperation from Mutual Acknowledgment Exchange

SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2022

Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration

Similar lecture

Emergent Cooperation from Mutual Acknowledgment Exchange

Downloads