United Kingdom

In this study, we leverage multi-objective reinforcement learning to create tunable agents, i.e., agents that can adopt a range of different behaviours according to the designer&#39;s preferences, without the need for retraining. We apply this technique to sequential social dilemmas, settings where there is inherent tension between individual and collective rationality. Learning a single fixed policy in such settings leaves one at a significant disadvantage if the opponents&#39; strategies change after learning is complete. In our work, we demonstrate empirically that the tunable agents framework allows easy adaption between cooperative and competitive behaviours in sequential social dilemmas without the need for retraining, allowing a single trained agent model to be adjusted to cater for a wide range of behaviours and opponent strategies.

AAMAS 2021

Tunable Behaviours in Sequential Social Dilemmas using Multi-Objective Reinforcement Learning

AAMAS is the largest and most influential conference in the area of agents and multiagent systems, bringing together researchers and practitioners in all areas of agent technology and providing and internationally renowned high-profile forum for publishing and finding out about the latest developments in the field. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).  

![](https://assets.underline.io/uploads/markdown_image/1/image/ce4604bd54468a006164d16e080204fb.jpg)

Please click on the button below to register for this event.

If you are registered and see this page please check the account email you are logged in with. It must be the same as for the registration.

If you still have trouble accessing the contact please contact support at aamas2021@underline.io 

In this study, we leverage multi-objective reinforcement learning to create tunable agents, i.e., agents that can adopt a range of different behaviours according to the designer's preferences, without the need for retraining. We apply this technique to sequential social dilemmas, settings where there is inherent tension between individual and collective rationality. Learning a single fixed policy in such settings leaves one at a significant disadvantage if the opponents' strategies change after learning is complete. In our work, we demonstrate empirically that the tunable agents framework allows easy adaption between cooperative and competitive behaviours in sequential social dilemmas without the need for retraining, allowing a single trained agent model to be adjusted to cater for a wide range of behaviours and opponent strategies.

poster

Modeling vessel movement in a maritime environment is an extremely challenging task given the complex nature of vessel behavior. Several existing multiagent maritime decision making frameworks require access to an accurate traffic simulator. We develop a system using electronic navigation charts to generate realistic and high fidelity vessel traffic data using Generative Adversarial Networks (GANs). Our proposed Ship-GAN uses a conditional Wasserstein GAN to model a vessel's behavior. The generator can simulate the travel time of vessels across different maritime zones conditioned on vessels' speeds and traffic intensity. Furthermore, it can be used as an accurate simulator for prior decision making approaches for maritime traffic coordination, which used less accurate model than our approach. Experiments performed on the historical data from heavily trafficked Singapore strait show that our Ship-GAN system generates data whose statistical distribution is close to the real data distribution, and better fit than prior methods.

Ship-GAN: Generative Modeling Based Maritime Traffic Simulator

In recent years, machine learning (ML) models have been successfully applied in a variety of real-world applications. However, they are often complex and incomprehensible to human users. This can decrease trust in their outputs and render their usage in critical settings ethically problematic. As a result, several methods for explaining such ML models have been proposed recently, in particular for black-box models such as deep neural networks (NNs). Nevertheless, these methods predominantly explain model outputs in terms of inputs, disregarding the inner workings of the ML model computing those outputs. We present Argflow, a toolkit enabling the generation of a variety of ‘deep’ argumentative explanations (DAXs) for outputs of NNs on classification tasks.

Argflow: A Toolkit for Deep Argumentative Explanations for Neural Networks

In this paper, we demonstrate a Symbolic Reinforcement Learning (SRL) architecture for safe control in Radio Access Network (RAN) applications. In our automated tool, a user can select a high-level safety specifications expressed in Linear Temporal Logic (LTL) to shield an RL agent running in a given cellular network with aim of optimizing network performance, as measured through certain Key Performance Indicators (KPIs). In the proposed architecture, network safety shielding is ensured through model-checking techniques over combined discrete system models (automata) that are abstracted through reinforcement learning. We demonstrate the user interface (UI) helping the user set intent specifications to the architecture and inspect the difference in allowed and blocked actions.

Symbolic Reinforcement Learning for Safe RAN Control

In this paper we consider a particular class of problems called
 (MAGB) which constitutes a modified version of the Bernoulli MAB problem where two new elements must be taken into account:the
 and the
 The agent has an initial budget that evolves in time following the received rewards, which can be either +1 after a
 or -1 after a
 The problem can also be seen as a MAB version of the classic
 game. The contribution of this paper is a preliminary analysis on the probability of being ruined given the current budget and observations,   and the proposition of an alternative regret formulation, combining the classic regret notion with the expected loss due to the probability of being ruined. Finally, standard state-of-the-art methods are experimentally compared using the proposed metric.

Gambler Bandits and the Regret of Being Ruined

Agents that interact with other agents often do not know a priori what the other agents' strategies are, but have to maximise their own online return while interacting with and learning about others. The optimal adaptive behaviour under uncertainty over the other agents' strategies w.r.t. some prior can in principle be computed using the Interactive Bayesian Reinforcement Learning framework. Unfortunately, doing so is intractable in most settings, and existing approximation methods are restricted to small tasks. To overcome this, we propose to meta-learn (alongside the policy) approximate belief inference by combining sequential and hierarchical VAEs. We show empirically that our approach can learn a factorised belief model that separates the other agent's permanent and temporal structure, and outperforms methods that sample from the approximate posterior or do not have this hierarchical structure. A full version of this work can be found in Zintgraf et al. (2021).

Deep Interactive Bayesian Reinforcement Learning via Meta-Learning

Touch is an essential sense to the social development and well-being of individuals, acting as a communicative channel for emotions and empathy. Our overall objective is to enhance the ability of embodied conversational agents (ECAs) to bond with humans. To reach such an objective, we have endowed an ECA with the capacity to touch and be touched in a social interaction with a human in an immersive environment. By drawing inspiration from literature on human-human social touch and related works on human-agent tactile interactions, we have developed a framework for a touching ECA able to perceive when and how a user touches it and decide when and how to respond accordingly. This paper focuses on going beyond a simple bi-directional touch interaction through a decision model that actually adapts its behaviour to the content of the interaction, the level of rapport and the human's touch avoidance sensibility. This enables an actual interactive loop between the agent and the human.

Decision Model for a Virtual Agent that can Touch and be Touched

The envy-free, Pareto-efficient allocation of indivisible goods leads to computationally hard problems. There is a big variety of modeling issues, such as agent-specific utility functions or (high numbers of) different types of goods. In recent work, Bredereck et al. [ACM EC~2019] addressed this topic by showing (theoretical) fixed-parameter tractability results for "high-multiplicity fair allocation", exploiting parameters such as number of agents or maximum absolute   utility values. To this end, they used a number of tools from (theoretical) integer linear programming. We "engineer" their work towards practical usefulness, thereby being able to solve all real-world instances from the state-of-art online platform "spliddit.org for provably fair solutions". Besides providing the foundations for a fast tool for fair allocations, we also offer a flexible framework with the possibility to relax fairness or efficiency demands so to, e.g., allow tradeoffs between fairness and social welfare. Moreover, our framework provides ways to interpret and explain "solution paths" which makes it possible to perform further explorations in cases when no envy-free and efficient allocations exist.

High-Multiplicity Fair Allocation Made More Practical

In this work, we define the consensus-prevention problem, which examines the canonical swarm robotic consensus problem from an adversarial point of view: how (if at all) is it possible to lead a swarm into a disagreement, that is, prevent them from reaching an agreement. We focus on consensus-prevention in physically grounded tasks, concentrating on influencing the direction of movement of a flocking swarm and guaranteeing that the swarm will never converge to the same direction by the use of external, predefined agents, referred to as diverting agents.

Spatial Consensus-Prevention in Robotic Swarms

The pursuit of values drives human behavior and promotes cooperation. Existing research is focused on general (e.g., Schwartz) values that transcend contexts. However, context-specific values are necessary to (1) understand human decisions, and (2) engineer intelligent agents that can elicit human values and take value-aligned actions.

Axies: Identifying and Evaluating Context-Specific Values

Multi-agent reinforcement learning (MARL) has been increasingly explored to learn the cooperative policy towards maximizing a certain global reward. Many existing studies take advantage of graph neural networks (GNN) in MARL to propagate critical collaborative information over the interaction graph, built upon inter-connected agents. Nevertheless, the vanilla GNN approach yields substantial defects in dealing with complex real-world scenarios since the generic message passing mechanism is ineffective between heterogeneous vertices and, moreover, simple message aggregation functions are incapable of accurately modeling the combinational interactions from multiple neighbors. While adopting complex GNN models with more informative message passing and aggregation mechanisms can obviously benefit heterogeneous vertex representations and cooperative policy learning, it could, on the other hand, increase the training difficulty of MARL and demand more intense and direct reward signals compared to the original global reward. To address these challenges, we propose a new cooperative learning framework with pre-trained heterogeneous observation representations. Particularly, we employ an encoder-decoder based graph attention to learn the intricate interactions and heterogeneous representations that can be more easily leveraged by MARL. Moreover, we design a pre-training with local actor-critic algorithm to ease the difficulty in cooperative policy learning. Extensive experiments over real-world scenarios demonstrate that our new approach can significantly outperform existing MARL baselines as well as operational research solutions that are widely-used in industry.

Tunable Behaviours in Sequential Social Dilemmas using Multi-Objective Reinforcement Learning

Next from AAMAS 2021

Ship-GAN: Generative Modeling Based Maritime Traffic Simulator

Similar lecture

Approximate Difference Rewards for Scalable Multigent Reinforcement Learning