United Kingdom

Continual learning aims to provide intelligent agents capable of learning multiple tasks sequentially with neural networks. In most settings of the current approaches, the agent starts from randomly initialized parameters and is optimized to master the current task regardless of the usefulness of the learned representation for future tasks. Moreover, each of the future tasks uses all the previously learned knowledge although parts of this knowledge might not be helpful for its learning. These cause interference among tasks, especially when the data of previous tasks is not accessible. In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM), which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting. SAM incorporates an attention mechanism that learns to select the particular relevant representation for each future task. We empirically show that we can achieve a better performance than several state-of-the-art methods for continual learning by building on top of the selected representation learned by SAM. We also show the role of the meta-attention mechanism in boosting informative features corresponding to the input task and identifying the correct target in the task agnostic inference. Finally, we demonstrate that popular existing continual learning methods gain a performance boost when they adopt SAM as a starting point.

AAMAS 2021

Self-Attention Meta-Learner for Continual Learning

AAMAS is the largest and most influential conference in the area of agents and multiagent systems, bringing together researchers and practitioners in all areas of agent technology and providing and internationally renowned high-profile forum for publishing and finding out about the latest developments in the field. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).  

![](https://assets.underline.io/uploads/markdown_image/1/image/ce4604bd54468a006164d16e080204fb.jpg)

Please click on the button below to register for this event.

If you are registered and see this page please check the account email you are logged in with. It must be the same as for the registration.

If you still have trouble accessing the contact please contact support at aamas2021@underline.io 

poster

Human-agent interactions present people guide an object or agent to act as human intentions. This demonstration work develops an online human-agent interaction system, particularly targeting the brain-computer interface (BCI), which uses real-time brain cortex signals: electroencephalogram (EEG) to control the agent in Unity3D game platform. The developed system also provides the online visualisation of EEG signals, including pre-processed temporal data and power spectral in three frequency bands (theta, alpha, and beta). To build this systematic work, we firstly collect wireless EEG signals via the Bluetooth transmission from a commercially available 14-channel brainware headset (Emotiv). EEG signals are then pre-processed and fed into a trained deep learning model to predict the human intentions, which will be sent to Unity3D platform to control an agent’s movements in game playing, such as a karting game scenario. The online testing results show the feasibility of our systematic work that will benefit for human-agent interaction community. The demonstration video can be viewed at the following link: https://youtu.be/9AWKHeatc6I

An Online Human-Agent Interaction System: A Brain-controlled Agent Playing Games in Unity

Pool block withholding attack, which reduces the effective mining power in the system and leads to potential systemic instability in the blockchain, can be modeled as a non-cooperative game called "the miner's dilemma". However, existing literature on the game-theoretic properties of this attack only gives a preliminary analysis. In this paper, we establish the existence and uniqueness of pure Nash equilibrium for the two-player miner's dilemma. Then we give a tight upper bound 2 for PPoA, which measures how much mining power is wasted in the game. Moreover, we show the uniqueness and the tight bound holds in a more general setting with betrayal assumption. Inspired by the experiments on the games among three mining pools, we conjecture that similar results should hold for the N-player miner's dilemma game (N ≥ 2).

The Tight Bound for Pure Price of Anarchy in an Extended Miner's Dilemma Game

We design a prediction market to recover a complete and fully general probability distribution over a random variable. Traders buy and sell interval securities that pay $1 if the outcome falls into an interval and $0 otherwise. Our market takes the form of a central automated market maker  and allows traders to express interval endpoints of arbitrary precision. We present two designs in both of which market operations take time logarithmic in the number of intervals (that traders distinguish), providing the first computationally efficient market for a continuous variable. Our first design replicates the popular logarithmic market scoring rule  (LMSR), but operates exponentially faster than a standard LMSR by exploiting its modularity properties to construct a balanced binary tree and decompose computations along the tree nodes. The second design consists of two or more parallel LMSR market makers that mediate submarkets of increasingly fine-grained outcome partitions.  This design remains computationally efficient for all operations, including arbitrage removal across submarkets. It adds two additional benefits for the market designer: (1)~the ability to express utility for information at various resolutions by assigning different liquidity values, and  (2)~the ability to guarantee a true constant bounded loss by appropriately decreasing the liquidity in each submarket.

Log-time Prediction Markets for Interval Securities

In modular reinforcement learning (MRL), a complex decision making problem is decomposed into multiple simpler subproblems each solved by a separate module. Often, these subproblems have conflicting goals, and incomparable reward scales. A composable decision making architecture requires that even the modules authored separately with possibly misaligned reward scales can be combined coherently. An arbitrator should consider different module's action preferences to learn effective global action selection. We present a novel framework called GRACIAS that assigns fine-grained importance to the different modules based on their relevance in a given state, and enables composable decision making based on modern deep RL methods such as deep deterministic policy gradient (DDPG) and deep Q-learning. We provide insights into the convergence properties of GRACIAS and also show that previous MRL algorithms reduce to special cases of our framework. We experimentally demonstrate on several standard MRL domains that our approach works significantly better than the previous MRL methods, and is highly robust to incomparable reward scales. Our framework extends MRL to complex Atari games such as Qbert, and has a better learning curve than the conventional RL algorithms.

Action Selection for Composable Modular Deep Reinforcement Learning

Ensemble learning is a very prevalent method employed in machine learning. The relative success of ensemble methods is attributed to their ability to tackle a wide range of instances and complex problems that require different low-level approaches. However, ensemble methods are relatively less popular in reinforcement learning owing to the high sample complexity and computational expense involved in obtaining a diverse ensemble. We present a novel training and model selection framework for model-free reinforcement algorithms that use ensembles of policies obtained from a single training run. These policies are diverse in nature and are learned through directed perturbation of the model parameters at regular intervals. We show that learning and selecting an adequately diverse set of policies is required for a good ensemble while extreme diversity can prove detrimental to overall performance. Selection of an adequately diverse set of policies is done through our novel policy selection framework. We evaluate our approach on challenging discrete and continuous control tasks and also discuss various ensembling strategies. Our framework is substantially sample efficient, computationally inexpensive and is seen to outperform state-of-the-art (SOTA) scores in Atari 2600 and Mujoco.

SEERL: Sample Efficient Ensemble Reinforcement Learning

Symbolic planning models allow decision-making agents to sequence actions in arbitrary ways to achieve a variety of goals in dynamic domains. However, they are typically handcrafted and tend to require precise formulations that are not robust to human error. Reinforcement learning (RL) approaches do not require such models, and instead learn domain dynamics by exploring the environment and collecting rewards. However, RL approaches tend to require millions of episodes of experience and often learn policies that are not easily transferable to other tasks. In this paper, we address one aspect of the open problem of integrating these approaches: how can decision-making agents resolve discrepancies in their symbolic planning models while attempting to accomplish goals? We propose an integrated framework named SPOTTER that uses RL to augment and support ("spot") a planning agent by discovering new operators needed by the agent to accomplish goals that are initially unreachable for the agent. SPOTTER outperforms pure-RL approaches while also discovering transferable symbolic knowledge and does not require supervision, successful plan traces or any a priori  knowledge about the missing planning operator.

SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

People often encounter high-stress situations. Modeling and being able to predict people’s behavior in such situations, how they cope, is a critical research topic. To that end, we propose a computational model of coping that casts Lazarus' theory of coping into a Partial Observable Markov Decision Process (POMDP) framework. This includes an appraisal process that models the factors that lead to stress by assessing a person's relation to the environment and a coping process that models people’s behavior in the face of such stress. This coping process includes problem-focused coping, whereby people seek to alter the external environment, and emotion-focused coping, whereby people alter their internal beliefs, goals, and intentions in the face of stress. We evaluate the model’s assumptions and predictions in the context of a high-stress situation that is increasingly common, the extreme conditions of a hurricane. We collected human survey data from the last several years of major U.S. hurricanes to evaluate the features in the models used for appraisal calculation. Additionally, we conducted a controlled human-subject experiment simulating a hurricane experience to investigate the prediction of the model on how people change their beliefs and goals to cope with the situation. The results show that, as predicted by the model, the proposed model features are significantly associated with the evacuation decisions and post-decision people also change their beliefs and goals in the directions that align with their prior decisions. Lastly, we conduct a simulation study showing that the proposed model is qualitatively closer to the experiment data than the baseline models that do not incorporate coping effects.

A Computational Model of Coping for Simulating Human Behavior in High-Stress Situations

Blockchain-based Distributed Ledgers (DLs) promise to transform the existing financial system by making it truly democratic. In the past decade, blockchain technology has seen many novel applications ranging from the banking industry to real estate. However, in order to be adopted universally, blockchain systems must be scalable to support a high volume of transactions. As we increase the throughput of the DL system, the underlying peer-to-peer network might face multiple levels of challenges to keep up with the requirements. Due to varying network capacities, the slower nodes would be at a relative disadvantage compared to the faster ones, which could negatively impact their revenue. In order to quantify their relative advantage or disadvantage, we introduce two measures of network fairness, $p_f$, the probability of frontrunning and α_f$, the publishing fairness. We show that as we scale the blockchain, both these measures deteriorate, implying that the slower nodes face a disadvantage at higher throughputs. It results in the faster nodes getting more than their fair share of the reward while the slower nodes (slow in terms of network quality) get less. Thus, fairness and scalability in blockchain systems do not go hand in hand.

We might walk together, but I run faster: Network Fairness and Scalability in Blockchains

Agents operating in the open world often produce negative side effects (NSE), which are difficult to identify at design time. We examine how a human can assist an agent, beyond providing feedback, and exploit their broader scope of knowledge to mitigate the impacts of NSE. We formulate this problem as a human-agent team with decoupled objectives. The agent optimizes its assigned task, during which its actions may produce NSE. The human shapes the environment through minor reconfiguration actions so as to mitigate the impacts of agent's side effects, without significantly degrading agent performance. We present an algorithm to solve this problem. Empirical evaluation shows that the proposed framework can successfully mitigate NSE, without affecting the agent's ability to complete its assigned task.

Mitigating Negative Side Effects via Environment Shaping

Complex networks and reputation systems are fundamental mechanisms to sustain cooperation in populations of self-regarding agents. These mechanisms are typically studied in isolation. In online social platforms, however, behavioral dynamics are likely to result from their combination. Here we investigate the relationship between social networks and reputation-based cooperation (in a Prisoner’s Dilemma setting) in large populations. We develop a new evolutionary game-theoretical model and study dynamics in networks with varying degrees of community structure. We show that networks exhibiting modular structures hamper global cooperation: reputation-based group identities emerge in different communities and strategies that uniquely cooperate with in-group members fixate, sustaining polarization and group bias. Global cooperation is recovered provided that inter-community edges are added.

Self-Attention Meta-Learner for Continual Learning

Next from AAMAS 2021

An Online Human-Agent Interaction System: A Brain-controlled Agent Playing Games in Unity

Similar lecture

Self-Attention Meta-Learner for Continual Learning