United Kingdom

Distributed training architectures have been shown to be effective to improve the performance of reinforcement learning algorithms. However, their performances are still poor for problems with sparse rewards, e.g., the scoring task with or without goalkeeper for robots in RoboCup soccer. It is challenging to solve these tasks in reinforcement learning, especially for those that require combining high-level actions with flexible control. To address these challenges, we introduce a distributed training framework with parallel curriculum experience replay that can collect different experiences in parallel and then automatically identify the difficulty of these subtasks. Experiments on the domain of simulated RoboCup soccer show that, the approach is effective and outperforms existing reinforcement learning methods.

AAMAS 2021

Parallel Curriculum Experience Replay in Distributed Reinforcement Learning

[![](https://assets.underline.io/uploads/markdown_image/1/image/21d13e03491d7e9acf6b8c46c7979c25.png)](https://docs.google.com/spreadsheets/d/1SCDDfkQk7es4J6RULKHVx1RzbB0Mutla--Sle5SVfb0/edit?usp=sharing)

**RoboCup 2021 will be Worldwide!**

We are excited to announce that RoboCup 2021 is a fully remote event with RoboCup competitions and activities taking place all over the world. Everyone is invited to participate in, or observe, the [competition leagues](https://2021.robocup.org/leagues) and the [symposium](https://2021.robocup.org/symposium)!

**RoboCup Symposium**

The 24th RoboCup International Symposium will be held in conjunction with RoboCup 2021 in a purely online setting. Scientific papers will be presented reporting innovative, original research with relevance to robotics and artificial intelligence. Within the described scope of topics we also encouraged submissions of high-quality overview articles, papers describing real-world research, and papers reporting theoretical results. In addition to the main track with regular research papers, there will be a special track focused on systems, data, and benchmarks featuring papers describing novel, open-source hardware and software systems, as well as datasets and benchmarks relevant to the community. In addition to research paper presentations, the RoboCup Symposium 2021 includes keynote talks by Dieter Fox, Jean-Paul Laumond, and Stefanie Tellex, and several interactive sessions.

**RoboCup Introduction Video**
<iframe width="995" height="516" src="https://www.youtube.com/embed/Cnh43cCsDKU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>


Registration is required for entrance into RoboCup 2021. <br><br>
**Fees:** 
* Major Faculty or Symposium Faculty - US$50
* Major Student, Junior Mentor, Junior Student, or Symposium Student - US$20 

***Members of qualified teams:** Please contact your league for your registration password and proceed to register with your team. *<br>
***All non-team members:** Please register as "Symposium Only".*

You need to be registered to explore this event!

RoboCup 2021

The 24th RoboCup International Symposium will be held in conjunction with RoboCup 2021 in a purely online setting. Scientific papers will be presented reporting innovative, original research with relevance to robotics and artificial intelligence. Within the described scope of topics we also encouraged submissions of high-quality overview articles, papers describing real-world research, and papers reporting theoretical results. In addition to the main track with regular research papers, there will be a special track focused on systems, data, and benchmarks featuring papers describing novel, open-source hardware and software systems, as well as datasets and benchmarks relevant to the community. In addition to research paper presentations, the RoboCup Symposium 2021 includes keynote talks by Dieter Fox, Jean-Paul Laumond, and Stefanie Tellex, and several interactive sessions.

technical paper

AAMAS is the largest and most influential conference in the area of agents and multiagent systems, bringing together researchers and practitioners in all areas of agent technology and providing and internationally renowned high-profile forum for publishing and finding out about the latest developments in the field. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).  

![](https://assets.underline.io/uploads/markdown_image/1/image/ce4604bd54468a006164d16e080204fb.jpg)

Please click on the button below to register for this event.

If you are registered and see this page please check the account email you are logged in with. It must be the same as for the registration.

If you still have trouble accessing the contact please contact support at aamas2021@underline.io 

In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from the single execution of a policy. In these settings, making decisions based on the average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. When making a decision, just the expected return -- known in reinforcement learning as the value -- cannot account for the potential range of adverse or positive outcomes a decision may have. Our key insight is that we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time.

Distributional Monte Carlo Tree Search for Risk-Aware and Multi-Objective Reinforcement Learning

In this study, we leverage multi-objective reinforcement learning to create tunable agents, i.e., agents that can adopt a range of different behaviours according to the designer's preferences, without the need for retraining. We apply this technique to sequential social dilemmas, settings where there is inherent tension between individual and collective rationality. Learning a single fixed policy in such settings leaves one at a significant disadvantage if the opponents' strategies change after learning is complete. In our work, we demonstrate empirically that the tunable agents framework allows easy adaption between cooperative and competitive behaviours in sequential social dilemmas without the need for retraining, allowing a single trained agent model to be adjusted to cater for a wide range of behaviours and opponent strategies.

Tunable Behaviours in Sequential Social Dilemmas using Multi-Objective Reinforcement Learning

In this work, we consider the problem of computing optimal actions for Reinforcement Learning (RL) agents in a co-operative setting, where the objective is to optimize a common goal. However, in many real-life applications, the agents are also required to satisfy certain constraints specified on their actions. Under this setting, the objective of the agents is to not only learn the actions that optimize the common objective but also meet the specified constraints. In recent times, the Actor-Critic algorithm with an attention mechanism has been successfully applied to obtain optimal actions for RL agents in multi-agent environments. In this work, we extend this algorithm to the constrained multi-agent RL setting.

Attention Actor-Critic Algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning

Efficient communication is a promising way to achieve cooperation among agents in many real-world scenarios. However, aimless and motiveless information sharing may not work or even degrade the cooperative performance. Typically, the multi-agent communication behaviors are motivated by extrinsic rewards from environment. We conclude the mechanism as 'Communicate what rewards you'. In this work, we present a novel communication mechanism called Intrinsic Motivated Multi-Agent Communication (IMMAC). Our key insight can be summarized as 'Communicate what surprises you'. Concretely, we use an observation-dependent intrinsic value to represent the importance of observed information. Then a gating mechanism and an attentional mechanism based on intrinsic values are designed to control communication. By encouraging agent to communicate and focus on the observations with uncertain and important information, our algorithm achieves superior communication efficiency and cooperative performance. We evaluate IMMAC on a variety of challenging tasks, and demonstrate that intrinsic values are sufficient to drive efficient communication behaviors. Moreover, we found that the combination of intrinsic values and extrinsic values can further improve the communication efficiency. Consequently, intrinsic motivation is a promising way to control communication and it is capable of being a good complement to the existing extrinsic motivated communication methods.

Intrinsic Motivated Multi-Agent Communication

This paper focuses on the multi-agent credit assignment problem. We propose a novel multi-agent reinforcement learning algorithm called 
 (MICRA) and a three-phase framework for training, adaptation, and execution of MICRA. The key features are: (1) a 
 is proposed to optimize the target agents' policy; (2) a meta-imitator is designed to infer the external agents' policies. Results show that MICRA outperforms state-of-the-art algorithms.

Fast Adaptation to External Agents via Meta Imitation Counterfactual Regret Advantage

In many distributed systems, from cloud to sensor networks, different configurations impact system performance, while strongly depending on the network topology. Hence, topological changes may entail costly reconfiguration and optimisation processes. This paper proposes a multi-agent solution for recovering a network's topology in case of node failures. The proposed approach relies on local information about the network's topology, collected and disseminated at runtime. Two strategies for distributing topological data are studied: one based on Mobile Agents (our proposal) and the other based on Trickle (a reference gossiping protocol from the literature). These two strategies were adapted for our self-healing approach, to collect topological information for network recovery; and were evaluated in terms of resource overheads. Experimental results show that both variants can recover the network topology, up to a certain node failure rate, which depends on the network topology. At the same time, Mobile Agents collect less information, focusing on local dissemination, which suffices for network recovery. This entails less bandwidth overheads than when Trickle is used. Still, Mobile Agents utilise more memory and exchange more messages during data-collection than Trickle does. These results validate the viability of the proposed self-healing solution, offering two variant implementations with diverse performance characteristics, which may suit different application domains.

A Decentralised Self-Healing Approach for Network Topology Maintenance (JAAMAS Track)

Touch is an essential sense to the social development and well-being of individuals, acting as a communicative channel for emotions and empathy. Our overall objective is to enhance the ability of embodied conversational agents (ECAs) to bond with humans. To reach such an objective, we have endowed an ECA with the capacity to touch and be touched in a social interaction with a human in an immersive environment. By drawing inspiration from literature on human-human social touch and related works on human-agent tactile interactions, we have developed a framework for a touching ECA able to perceive when and how a user touches it and decide when and how to respond accordingly. This paper focuses on going beyond a simple bi-directional touch interaction through a decision model that actually adapts its behaviour to the content of the interaction, the level of rapport and the human's touch avoidance sensibility. This enables an actual interactive loop between the agent and the human.

Decision Model for a Virtual Agent that can Touch and be Touched

Network robustness is a measure a network's ability to survive adversarial attacks. But not all parts of a network are equal. K-cores, which are dense subgraphs, are known to capture some of the key properties of many real-life networks. Therefore, previous work has attempted to model network robustness via the stability of its k-core. However, these approaches account for a single core value and thus fail to encode a global network resilience measure.

Network Robustness via Global k-cores

Green Security Games (GSGs) have been successfully used in the protection of valuable resources such as fisheries, forests, and wildlife. Real-world deployment involves both resource allocation and subsequent coordinated patrolling with communication in the presence real-time, uncertain information. Previous game models do not address both of these stages simultaneously. Furthermore, adopting existing solution strategies is difficult since they do not scale well for larger, more complex variants of the game models. We propose a novel GSG model to address these challenges. We also present a novel algorithm, CombSGPO, to compute a defender strategy for this  game model. CombSGPO performs policy search over a multidimensional, discrete action space to compute an allocation strategy that is best suited to a best-response patrolling strategy for the defender,  learnt by training a multi-agent Deep Q-Network. We show via experiments that CombSGPO converges to better strategies and is more scalable than comparable approaches. From a detailed analysis of the coordination and signaling behavior learnt by CombSGPO, we find that strategic signaling emerges in the final learnt strategy.

Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty

Among the most fundamental tools for social network analysis are centrality measures, which quantify the importance of every node in the network. This centrality analysis typically disregards the possibility that the network may have been deliberately manipulated to mislead the analysis. To solve this problem, a recent study attempted to understand how a member of a social network could rewire the connections therein to avoid being identified as a leader of that network. However, the study was based on the assumption that the network analyzer-the seeker-is oblivious to any evasion attempts by the evader. In this paper, we relax this assumption by modelling the seeker and evader as strategic players in a Bayesian Stackelberg game. In this context, we study the complexity of various optimization problems, and analyze the equilibria of the game under different assumptions, thereby drawing the first conclusions in the literature regarding which centralities the seeker should use to maximize the chances of detecting a strategic evader.

Downloads

Next from AAMAS 2021

Distributional Monte Carlo Tree Search for Risk-Aware and Multi-Objective Reinforcement Learning

Similar lecture

rSoccer: A Framework for Studying Reinforcement Learning in Small and VerySmall Size Robot Soccer

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2021

Distributional Monte Carlo Tree Search for Risk-Aware and Multi-Objective Reinforcement Learning

Similar lecture

rSoccer: A Framework for Studying Reinforcement Learning in Small and VerySmall Size Robot Soccer

Downloads