New Zealand


In many applications, the desire is to change the behavior of users over a period of repeated episodes of similar decision making. For example, in a nutritionist scenario, one may try to encourage the user to adopt better food consumption habits. In this paper, we propose to model this recommendation scenario with a long-term goal, that we call coaching, as an iterative two-player game. At each decision time, the user makes a proposal (e.g., a meal) based on her/his preferences, the coach can then suggest a change in this proposal that the user may or may not accept. After each episode, the user updates her/his preferences and the coach adapts its model of the user.

We propose a formalization of the coaching problem and discuss several possible criteria to measure the performance of a coach. 
Different coaching strategies are described in the paper. They are then tested and compared using a real-world dataset in the field of nutrition, where a user is simulated using a simple, but general, model of decision making and adaptation. Results show that it pays to adapt to user characteristics and use non-myopic strategies, which aim for long-term gains, when the number of interactions becomes large. 

Although illustrated on choice sequences for food consumption, the scope of the proposed method goes far beyond this use case, as in sports, health, entertainment or tourist activity choices, etc.

AAMAS 2022

Coaching agent: making recommendations for behavior change. A case study on improving eating habits

teacher-student scenario

long-term recommendation

automated assistant

reinforcement learning

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2025

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

technical paper

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).

A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).


Cooperation is an overarching aspiration of artificial intelligence (AI) research. Recent studies demonstrate that AI agents trained with deep reinforcement learning are capable of interacting and collaborating with humans. These studies have primarily evaluated human compatibility and intelligence through "objective" metrics like task performance, obscuring potential variation in the levels of trust and subjective preference that different agents generate. To better understand the factors shaping subjective preferences in human-agent cooperation, we train deep reinforcement learning agents in Coins, a two-player social dilemma. We recruit participants for a human-agent cooperation study, and measure their social perception of the agents they encounter. Participants' perceptions of warmth and competence predict their preferences for different agents, above and beyond objective performance metrics. Drawing inspiration from social science and biology research, we subsequently implement a new "partner choice" framework to elicit revealed preferences: after playing an episode with an agent, participants are asked whether they would like to play the next round with the same agent or to play alone. As with stated preferences, social perception better predicts participants' revealed preferences than does objective performance. Given these findings, we recommend human-agent interaction researchers routinely incorporate the measurement of social perception and subjective preferences into their studies.


Warmth and Competence in Human-Agent Cooperation


In sequential decision making -- whether it's realized with or without the benefit of a model -- objectives are often underspecified or incomplete. This gives discretion to the acting agent to realize the stated objective in ways that may result in undesirable outcomes, including inadvertently creating an unsafe environment or indirectly impacting the agency of humans or other agents that typically operate in the environment. In this paper, we explore how to build a reinforcement learning (RL) agent that contemplates the impact of its actions on the wellbeing and agency of others in the environment, most notably humans. We endow RL agents with the ability to contemplate such impact by augmenting their reward based on expectation of future return by others in the environment, providing different criteria for characterizing impact. We further endow these agents with the ability to differentially factor this impact into their decision making, manifesting behaviour that ranges from self-centred to self-less, as demonstrated by experiments in gridworld environments.

Be Considerate: Avoiding Negative Side Effects in Reinforcement Learning


Confronted with patients suffering from Alzheimer's disease, professional caregivers must be aware of their own non-verbal communication as well as the patient's non-verbal communication in order to respond to the behavioural disorders which are frequent in this disease (e.g. apathy, aggression, anxiety, etc...). Virtual patients, which are interactive animated characters that simulate the behaviors of a patient, are increasingly used to train caregivers to interact with patients. The virtual patient needs to dynamically generate multimodal expressions of pathological behaviors consistent with the pedagogical goals. In this article, we introduce COPALZ (COmputational model of Pathological appraisal biases for a virtual ALZheimer's patient), a new model that supports the generation of pathological behaviors identified as pedagogically relevant for caregivers interacting with Alzheimer's patients. This model is based on the theory of appraisal biases and on interaction data that we collected using a partially simulated version of our virtual patient with 31 caregivers. We describe initial evaluations and explain how we intend to evaluate the fully automatic virtual patient during training sessions.

COPALZ: A Computational Model of Pathological Appraisal Biases for an Interactive Virtual Alzheimer Patient


Training a multi-agent reinforcement learning (MARL) model with a sparse reward is generally difficult because numerous combinations of interactions among agents induce a certain outcome (i.e., success or failure). Earlier studies have tried to resolve this issue by employing an intrinsic reward to induce interactions that are helpful for learning an effective policy. However, this approach requires extensive prior knowledge for designing an intrinsic reward. To train the MARL model effectively without designing the intrinsic reward, we propose a learning-based exploration strategy to generate the initial states of a game. The proposed method adopts a variational graph autoencoder to represent a game state such that (1) the state can be compactly encoded to a latent representation by considering relationships among agents, and (2) the latent representation can be used as an effective input for a coupled surrogate model to predict an exploration score. The proposed method then finds new latent representations that maximize the exploration scores and decodes these representations to generate initial states from which the MARL model starts training in the game and thus experiences novel and rewardable states. We demonstrate that our method improves the training and performance of the MARL model more than the existing exploration methods.

REMAX: Relational Representation for Multi-Agent Exploration


Critical sectors of human society are progressing toward the adoption of powerful artificial intelligence (AI) agents, which are trained individually on behalf of self-interested principals but deployed in a shared environment. Short of direct centralized regulation of AI, which is as difficult an issue as regulation of human actions, one must design institutional mechanisms that indirectly guide agents' behaviors to safeguard and improve social welfare in the shared environment. Our paper focuses on one important class of such mechanisms: the problem of adaptive incentive design, whereby a central planner intervenes on the payoffs of an agent population via incentives in order to optimize a system objective. To tackle this problem in high-dimensional environments whose dynamics may be unknown or too complex to model, we propose a model-free meta-gradient method to learn an adaptive incentive function in the context of multi-agent reinforcement learning. Via the principle of online cross-validation, the incentive designer explicitly accounts for its impact on agents' learning and, through them, the impact on future social welfare. Experiments on didactic benchmark problems show that the proposed method can induce selfish agents to learn near-optimal cooperative behavior and significantly outperform learning-oblivious baselines. When applied to a complex simulated economy, the proposed method finds tax policies that achieve better trade-off between economic productivity and equality than baselines, a result that we interpret via a detailed behavioral analysis.

Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learning


Machine learning of Theory of Mind (ToM) is essential to build social agents that co-live with humans and other agents. This capacity, once acquired, will help machine infer the mental states of others from observed contextual action trajectories, enabling future prediction of goals, intention, actions and successor representations. The underlying mechanism for such a prediction remains unclear, however. Inspired by the observation that humans often infer the character traits of others, then use it to explain behaviour, we propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories. This trait vector then multiplicatively modulates the prediction mechanism via the so-called ‘fast weights’ scheme in the prediction neural network, which reads the current context and predicts the future. We empirically show that the fast weights provide a good inductive bias to model the character traits of agents and hence improves mind reading ability. On the indirect assessment of false-belief understanding, the new ToM model enables more efficient helping behaviours.

Learning Theory of Mind via Dynamic Traits Attribution


Reinforcement learning (RL) in partially observable, fully cooperative multi-agent settings (Dec-POMDPs) can in principle be used to address many real-world challenges such as controlling a swarm of rescue robots or a synchronous team of quadcopters. However, Dec-POMDPs are significantly harder to solve than single-agent problems, with the former being NEXP-complete and the latter, MDPs, being just P-complete. Hence, current RL algorithms for Dec-POMDPs suffer from poor sample complexity, thereby reducing their applicability to practical problems where environment interaction is costly. Our key insight is that using just a polynomial number of samples, one can learn a centralized model that generalizes across different policies. We can then optimize the policy within the learned model instead of the true system, reducing the number of environment interactions. We also learn a centralized exploration policy within our model that learns to collect additional data in state-action regions with high model uncertainty. We empirically evaluate the proposed model-based algorithm, MARCO, in three cooperative communication tasks, where it improves sample efficiency by up to 20x. Finally, to investigate the theoretical sample complexity, we propose a tabular model-based method for Dec-POMDPs and show that it achieves sample complexity in polynomial samples.

Centralized Model and Exploration Policy for Multi-Agent RL


We propose a novel formulation of group fairness in the contextual multi-armed bandit (CMAB) setting. In the CMAB setting a sequential decision maker must at each time step choose an arm to pull from a finite set of arms after observing some context for each of the potential arm pulls. In our model arms are partitioned into two or more sensitive groups based on some protected feature (e.g., age, race, or socio-economic status). Despite the fact that there may be differences in expected payout between the groups, we may wish to ensure some form of fairness between picking arms from the various groups. In this work we explore two definitions of fairness: equal group probability, wherein the probability of pulling an arm from any of the protected groups is the same; and proportional parity, wherein the probability of choosing an arm from a particular group is proportional to the size of that group. We provide a novel algorithm that can accommodate these notions of fairness for an arbitrary number of groups, and provide bounds on the regret for our algorithm. We then validate our algorithm using synthetic data as well as two real-world datasets for intervention settings wherein we want to allocate resources fairly across protected groups.

Group fairness in bandit arm selection


Autonomous agents may encounter the transportability problem when they suffer performance deficits from training in an environment that differs in key respects from that in which they are deployed. Although a causal treatment of transportability has been studied in the data sciences, the present work expands its utility into online, multi-agent, reinforcement learning systems in which agents are capable of both experimenting within their own environments and observing the choices of agents in separate, potentially different ones. In order to accelerate learning, agents in these MAT problems face the unique challenge of determining which agents are acting in similar environments, and if so, how to incorporate these observations into their policy. We propose and compare several agent policies that exploit local similarities between environments using causal selection diagrams, demonstrating that optimal policies are learned more quickly than in baseline agents that do not. Simulation results support the efficacy of these new agents in a novel variant of the Multi-Armed Bandit problem with MAT environments.

Exploiting Causal Structure for Transportability in Online, Multi-Agent Environments



In kidney exchange programs, multiple patient-donor pairs each
of whom are otherwise incompatible, exchange their donors to
receive compatible kidneys. The Kidney Exchange problem is
typically modelled as a directed graph where every vertex is either
an altruistic donor or a pair of patient and donor; directed edges are
added from a donor to its compatible patients. The computational
task is to find if there exists a collection of disjoint cycles and paths
starting from altruistic donor vertices of length at most ℓ_𝑐 and ℓ_𝑝
respectively that covers at least some specific number 𝑡 of nonaltruistic vertices (patients). We study parameterized algorithms for
the kidney exchange problem in this paper. Specifically, we design
FPT algorithms parameterized by each of the following parameters:
(1) the number of patients who receive kidney, (2) treewidth of the
input graph + max{ℓ_𝑝, ℓ_𝑐 }, and (3) the number of vertex types in the
input graph when ℓ_𝑝 ≤ ℓ_𝑐 . We also present interesting algorithmic
and hardness results on the kernelization complexity of the problem.
Finally, we present an approximation algorithm for an important
special case of Kidney Exchange.

Downloads

Next from AAMAS 2022

Warmth and Competence in Human-Agent Cooperation

Similar lecture

Developing an AI-based Leadership Coach for the US Naval Academy

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2022

Warmth and Competence in Human-Agent Cooperation

Similar lecture

Developing an AI-based Leadership Coach for the US Naval Academy

Downloads