Bernt Johan Leira, Arifian Agusta and Sebastian Thöns (September 24th 2019). Utilization of Digital Twins and Other Numerical Relatives for Efficient Monte Carlo Simulation in Structural Analysis, Theory, Application, and Implementation of Monte Carlo Method in Science and Technology, Pooneh Saidi Bidokhti, IntechOpen, DOI: 10.5772/intechopen.89144. 

Zongyan Wang (March 18th 2020). Digital Twin Technology, Industry 4.0 - Impact on Intelligent Logistics and Manufacturing, Tamás Bányai, Antonella Petrilloand Fabio De Felice, IntechOpen, DOI: 10.5772/intechopen.80974. 

Po-Hsun Cheng, Sheau-Ling Hsieh, Tsan-Nan Chien, Ying-Pei Chen, Mei-Ju Su, Yung-Chien Weng, Sao-Jie Chen, Feipei Lai and Jin-Shin Lai (April 1st 2010). Understanding Discrepancy: A Conceptual Persistent Healthcare Quality Improvement Process for Software Development Management, Process Management, Maria Pomffyova, IntechOpen, DOI: 10.5772/8453. 

New Zealand

A digital twin is a digital representation of a physical asset reproducing its data model, its behaviour and its communication with other physical assets. Although most of their concrete applications can be found mainly in the industrial context, healthcare represents another relevant area where digital twins can have a disruptive impact. The main research question tackled by this paper is about the integration of digital twins with agents and Multi-Agent Systems (MAS) technologies in healthcare, discussing what the benefits are and how such an integration could be realised.


AAMAS 2020

On the Integration of Agents and Digital Twins in Healthcare

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

technical paper

In many multi-agent interactions in the real world, agents receive payoffs over multiple distinct criteria; i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with non-linear utilities. Specifically, we consider multi-objective normal form games with non-linear utility functions under the scalarised expected returns optimisation criterion. We contribute a novel actor-critic formulation to allow reinforcement learning of mixed strategies in this setting, along with an extension that incorporates opponent policy reconstruction using conditional action frequencies. Empirical results in five different MONFGs demonstrate that opponent modelling can drastically alter the learning dynamics in this setting. When equilibria are present opponent modelling can confer significant benefits on agents that implement it. However, when there are no Nash equilibria, opponent modelling can have adverse effects on utility, and has a neutral effect at best (after extensive hyper-parameter optimisation).


Opponent Modelling using Policy Reconstruction for Multi-Objective Normal Form Games

We compare synthetic population-based travel demand modeling with the state of the art
travel demand models used by metropolitan planning offices in the United States. Our 
comparison of the models for three US cities shows that synthetic population-based
models match the state of the art models closely for the temporal trip distributions and
the spatial distribution of destinations. The advantages of the synthetic population-based 
method are that it provides greater spatial resolution, can be generalized to any region,
and can be used for studying correlations with demographics and activity types, which are
useful for modeling the effects of policy changes.

Improved Travel Demand Modeling with Synthetic Populations

We study the problem of two competing camps aiming to maximize the adoption of their respective opinions, by optimally investing in nodes of a social network in two phases. The final opinion of a node in phase 1 acts as its bias in phase 2, and this bias determines the effectiveness of a camp's investment on the node. Using an extension of Friedkin-Johnsen model of opinion dynamics, we formulate the camps' utility functions. We show the existence and polynomial time computability of Nash equilibrium under reasonable assumptions. Using simulations, we quantify the effects of the nodes' biases and the weightage attributed to them, as well as that of a camp deviating from its equilibrium strategy.


A Two Phase Investment Game for Competitive Opinion Dynamics in Social Networks

We define noisy combinatorial markets, a specific class of combinatorial markets and introduce a model to study them. We present a learning methodology we developed to learn competitive equilibria of noisy combinatorial markets. We summarize preliminary experimental results to evaluate the performance of our learning algorithms.


Learning Competitive Equilibria  in Noisy Combinatorial Markets

This talk describes a policy search approach for railway scheduling using the covariance matrix adaptation evolution strategy (CMA-ES). The goal is to define arrival/departure times and track allocations for all trains such that the resource and operating constraints of the railway line are satisfied, and priority-weighted train departure delay is minimised. The proposed approach is scalable in the sense that (i) the optimised policy can be applied to an arbitrarily long railway line, independent of the number of trains, tracks, and stations, and (ii) the on-line implementation is computationally light enough to be applied in real-time. Our experiments show that policies computed with CMA-ES are able to produce solutions with lower priorityweighted delay than heuristics and reinforcement learning (RL) algorithms reported in literature, on synthetic examples as well as actual railway line data from portions of the Indian Railway network.

Optimising a Real-Time Scheduler for Railway Lines using Policy Search

Robustness is an important property of software systems, and the availability of proper feedback is seen as crucial to obtain it, especially in the case of systems of distributed and interconnected components.
Multiagent Systems (MAS) are valuable for conceptualizing and implementing distributed systems, but the current design methodologies for MAS fall short in addressing robustness in a systematic way at design time.
In this paper we outline our vision of how robustness in MAS can be granted as a design property. To this end, we exploit the notion of accountability as a mechanism for building reporting frameworks and, then, we describe how robustness is gained.
We exemplify our vision on the JaCaMo agent platform.

Fragility and Robustness in Multiagent Systems

Reinforcement learning requires exploration, leading to repeated execution of sub-optimal actions. Naive exploration techniques address this problem by changing gradually from exploration to exploitation. This approach employs a wide search resulting in exhaustive exploration and low sample-efficiency. More advanced search methods explore optimistically based on an upper bound estimate of expected rewards. These methods employ deep search, aiming to reach states not previously visited. Another deep search strategy is found in action-elimination methods, which aim to discover and eliminate sub-optimal actions. Despite the effectiveness of advanced deep search strategies, some problems are better suited to naive exploration. We devise a new method, called Interval Q-Learning, that finds a balance between wide and deep search. It assigns a small probability to taking sub-optimal actions and combines both greedy and optimistic exploration. This allows for fast convergence to a near-optimal policy, and then exploration around it. We demonstrate the performance of tabular and deep Q-network versions of Interval Q-Learning, showing that it offers convergence speed-up both in problems that favor wide exploration methods and those that favor deep search strategies.


Interval Q-Learning: Balancing Deep and Wide Exploration

Epidemics of infectious diseases are an important threat to public health and global economies. 
Yet, the development of prevention strategies remains a challenging process.
For this reason, we investigate a deep reinforcement learning approach to automatically learn prevention strategies in an epidemiological model, in the context of pandemic influenza.
To this end, we construct a new epidemiological meta-population model, with 379 patches, that balances between model complexity and computational efficiency such that the use of reinforcement learning techniques becomes attainable.
First, we set up a ground truth such that we can evaluate the performance of the ""Proximal Policy Optimization"" algorithm to learn in a single district of this epidemiological model.
Next, we consider a larger scale problem, by conducting an experiment where we aim to learn a joint policy to control the districts in a community of 11 tightly coupled districts, for which no ground truth can be established. 
This experiment shows that deep reinforcement learning can be used to learn mitigation policies in complex epidemiological models with a large state space.
Moreover, through this experiment, we demonstrate that there can be an advantage to consider collaboration between districts when designing prevention strategies.


Deep reinforcement learning for large-scale epidemic control

Reward Redistribution Mechanism in Multi-Agent Reinforcement Learning

Option discovery and skill acquisition frameworks are integral to the functioning of a hierarchically organized Reinforcement learning agent. However, such techniques often yield a large number of options or skills, which can be represented succinctly by filtering out any redundant information. Such a reduction can decrease the required computation while also improving the performance on a target task. To compress an array of option policies, we attempt to find a policy basis that accurately captures the set of all options. In this work, we propose Option Encoder, an auto-encoder based framework with intelligently constrained weights, that helps discover a collection of basis policies. The policy basis can be used as a proxy for the original set of skills in a suitable hierarchically organized framework. We demonstrate the efficacy of our method on a collection of grid-worlds and Deepmind-lab by evaluating the obtained policy basis on downstream tasks.


On the Integration of Agents and Digital Twins in Healthcare

Downloads

Next from AAMAS 2020

Opponent Modelling using Policy Reconstruction for Multi-Objective Normal Form Games

Similar lecture

Deep reinforcement learning for large-scale epidemic control

On the Integration of Agents and Digital Twins in Healthcare

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2020

Opponent Modelling using Policy Reconstruction for Multi-Objective Normal Form Games

Similar lecture

Deep reinforcement learning for large-scale epidemic control

Downloads