United States

The headline successes of reinforcement learning on video games like StarCraft and DOTA 2 beg questions about the future role of the technology in the Department of Defense. Military simulations seemingly have much in common with video games, and the lack of a strong human behavior model is arguably a limiting factor for their use in the current environment. Training exercises require that opposing forces and adjoining allied forces act intelligently, and human pucksters with appropriate backgrounds and the funding to pay for them is not always forthcoming. The scope of analytical simulations must currently be limited due to the lack of current behavior models that react flexibly and appropriately over a wide range of circumstances. And more aggressive use cases for simulation in operational decision support might become practical with better AI to serve as a sparring partner. It is this last use case that we focus on here.

AI can potentially provide operational decision support by being a competent opponent of the human planner in simulation. The human planner would control his own forces according to his proposed plan, and the AI would do its best to defeat the plan. The planner could then choose to alter the plan and fight the AI again until he is satisfied. But is it actually practical to develop strong players using reinforcement learning?

Previously, it has been shown that reinforcement learning is capable of learning optimal attacking behavior in simple ground combat scenarios [Boron, J and Darken, C. &quot;Developing Combat Behavior through Reinforcement Learning in Wargames and Simulations&quot;, IEEE Conference on Games (CoG) 2020]. The primary examples studied were quite small (e.g. three units attacking two on a featureless plain, with all units being the same type, defenders stationary), but the Proximal Policy Optimization algorithm was able to learn provably optimal attacking behavior, and even find qualitatively different optimal solutions for the cases of an urgent attack (highly discounted rewards) and a more cautious one (less discounting). Still, this scenario is far short of the complexity that would be required for operational use.

In the proposed presentation, we will describe our ongoing and planned efforts to investigate the ability of reinforcement learning to learn strong attacking behavior in more complex scenarios, i.e. with more units of multiple types maneuvering through heterogeneous terrain, and with an actively maneuvering opponent. Our new training simulation for the AI, while still relatively simple, supports four different unit types and five different types of terrain. Maps are hexagonal grids of any size and the number of units per side is arbitrary. The simulation supports an AI Gym standard interface and is typically trained using the Stable Baselines 3 suite of PyTorch-based reinforcement learning algorithms.

The new training simulation represents the input features of the neural network in an image like manner, permitting the use of convolutional neural networks (CNNs). The spatially invariant nature of CNN kernels allows trained policies to learn optimal attacking behaviors against a retreating adversarial force, regardless of where the units are initially placed in the simulation. This feature promotes a more dynamically trained agent and increases the likelihood of achieving optimal behaviors in highly complex scenarios.

NAML 2021

Reinforcement Learning for Tactical Decision AI

constructive simulation

neural network

reinforcement learning

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

The headline successes of reinforcement learning on video games like StarCraft and DOTA 2 beg questions about the future role of the technology in the Department of Defense. Military simulations seemingly have much in common with video games, and the lack of a strong human behavior model is arguably a limiting factor for their use in the current environment. Training exercises require that opposing forces and adjoining allied forces act intelligently, and human pucksters with appropriate backgrounds and the funding to pay for them is not always forthcoming. The scope of analytical simulations must currently be limited due to the lack of current behavior models that react flexibly and appropriately over a wide range of circumstances. And more aggressive use cases for simulation in operational decision support might become practical with better AI to serve as a sparring partner. It is this last use case that we focus on here.

AI can potentially provide operational decision support by being a competent opponent of the human planner in simulation. The human planner would control his own forces according to his proposed plan, and the AI would do its best to defeat the plan. The planner could then choose to alter the plan and fight the AI again until he is satisfied. But is it actually practical to develop strong players using reinforcement learning?

Previously, it has been shown that reinforcement learning is capable of learning optimal attacking behavior in simple ground combat scenarios [Boron, J and Darken, C. "Developing Combat Behavior through Reinforcement Learning in Wargames and Simulations", IEEE Conference on Games (CoG) 2020]. The primary examples studied were quite small (e.g. three units attacking two on a featureless plain, with all units being the same type, defenders stationary), but the Proximal Policy Optimization algorithm was able to learn provably optimal attacking behavior, and even find qualitatively different optimal solutions for the cases of an urgent attack (highly discounted rewards) and a more cautious one (less discounting). Still, this scenario is far short of the complexity that would be required for operational use.

In the proposed presentation, we will describe our ongoing and planned efforts to investigate the ability of reinforcement learning to learn strong attacking behavior in more complex scenarios, i.e. with more units of multiple types maneuvering through heterogeneous terrain, and with an actively maneuvering opponent. Our new training simulation for the AI, while still relatively simple, supports four different unit types and five different types of terrain. Maps are hexagonal grids of any size and the number of units per side is arbitrary. The simulation supports an AI Gym standard interface and is typically trained using the Stable Baselines 3 suite of PyTorch-based reinforcement learning algorithms.

The new training simulation represents the input features of the neural network in an image like manner, permitting the use of convolutional neural networks (CNNs). The spatially invariant nature of CNN kernels allows trained policies to learn optimal attacking behaviors against a retreating adversarial force, regardless of where the units are initially placed in the simulation. This feature promotes a more dynamically trained agent and increases the likelihood of achieving optimal behaviors in highly complex scenarios.

poster

Mindful™ software provides human-understandable competency statements in the DARPA Competency Aware Machine Learning program. MindfuL™ software models environments across ML applications compactly with hierarchical Dirichlet processes allowing it to provide environment based-competency assessments through deep learning and deep explainer models as well as competency of similar historical data.

Mindful™: Competency Aware Machine Learning

We present an ongoing investigation into the learning of the input to state relationship of a dynamical system by using nonlinear Adaptive Control Theory combined with Spiking Neural Networks (SNNs). In particular, we use a specific type of SNN named Efficient Balanced Networks (EBNs) initially proposed by Deneve, Machens et. al. that provides desirable properties such as distributed information representation, low firing rates, and high encoding precision. Neuronal spikes are fired based on a threshold that is generated by optimizing over a global cost function that penalizes output tracking error, spike frequency, and spike distribution across neurons. Our objective is to implement these algorithms on neuromorphic hardware and apply them to complex robotic control applications. Applications include online system identification, generation of oracles for policy training, adaptive model predictive control, and model-based reinforcement learning. This presentation is effectively a status update on these investigations showing results obtained to date.

Learning Dynamical Systems with Neuromorphic Efficiently Balanced Spiking Neural Networks

When learning to perform motor tasks in a simulated environment, neural networks must be allowed to explore their action space to discover new potentially viable solutions. However, in an online learning scenario with physical hardware, this exploration must be constrained by relevant safety considerations in order to avoid damage to the agent’s hardware and environment. Our aim is to address this problem by training a neural network, which we will refer to as the “safety network,” to estimate the region of attraction (ROA) of a dynamical system controlled by an “acting network.” This safety network can thereby be used to quantify the relative safety of control actions proposed by the acting network and prevent the selection of damaging actions. Here we present our preliminary development of the safety network by training a non-spiking neural network to represent the ROA of several dynamical system benchmark problems. The training of this network is predicated upon both Lyapunov theory and neural solutions to partial differential equations (PDEs). By learning to approximate the viscosity solution to a specially chosen PDE, the safety network learns to approximate a Lyapunov function whose unity level set is the boundary of the ROA. We train our safety network to solve these PDEs in a semi-supervised manner, utilizing a loss function that penalizes disagreement with the PDE’s initial and boundary conditions, as well as non-zero residual and variational terms. In future work, we intend to extend this technique to spiking neural networks to be run on neuromorphic hardware for real-time online learning.

Learning to Estimate Stability Margins in Dynamical Systems

Our NLP predictive algorithm can accurately assign and provide solutions to NAVY 311 support request tickets, reducing contact center labor and time to resolution.

Next Generation NLP and AI Methods in the Navy Contact Center

A heterogeneous, nonoverlapping domain-decomposition, explicit, finite volume method (HT-NODDE-FVM) is developed and applied to a real-time, hybrid, process-state estimator of 3D unsteady, advection-diffusion plume fields produced by moving aerial intruders. The hybrid Luenberger-naïve estimator uses field measurements taken onboard a data-driven self-guided UAV and computes the plume field in real-time while guiding the UAV to locations that provide optimal information to the estimator.

Real-Time Estimation of Threat Fields With an Autodidactic Data-Driven Sensing Aerial Vehicle

The search for neural architecture is producing many of the most exciting results in artificial intelligence. It has increasingly become apparent that task-specific neural architecture plays a crucial role for effectively solving problems. This paper presents a simple method for learning neural architecture through random mutation. This method demonstrates 1) neural architecture may be learned during the agent’s lifetime, 2) neural architecture may be constructed over a single lifetime without any initial connections or neurons, and 3) architectural modifications enable rapid adaptation to dynamic and novel task scenarios. Starting without any neurons or connections, this method constructs a neural architecture capable of high-performance on several tasks. The lifelong learning capabilities of this method are demonstrated in an environment without episodic resets, even learning with constantly changing morphology, limb disablement, and changing task goals all without losing locomotion capabilities.

Self-Constructing Neural Networks Through Random Mutation

Object proposal algorithms are needed to find bounding boxes for salient class-agnostic objects. It is an important pre-processing step for object detection in the wild. While most state-of-the-art object detection methods adopt an end-to-end deep neural networks, here we aim at an independent object proposal unit that has low complexity and high performance. The proposed light-weight object proposal can be combined with any classification process to reduce model and computation complexity.
Our object proposal method is built upon the PixelHop framework, it is called OPPHop (Object Proposal PixelHop). The term “hop” denotes the neighborhood of a pixel. The high-level principle of our method is described below. OPPHop conducts spectral analysis on neighborhoods of different sizes centered on a pixel through a sequence of cascaded dimension reduction units. The spectral analysis tool is called the Saab (Subspace approximation with adjusted bias) transform. The neighborhoods of an object contain salient contours and, as a result, they have distinctive spectral signatures at a certain scale that matches the object size. The distinctive regions can be predicted based on supervised learning with Saab coefficients as the input.
The proposed OPPHop method consists of the following three steps: 
1. From fine-to-coarse spatial resolutions, conduct successive Saab transforms for spectral representation of neighborhoods of various sizes.
2. Predict distinctive spatial regions in various spatial resolutions via supervised learning.
3. Fuse distinctive regions of different spatial resolutions via image priors.
We evaluate OPPHop framework on the popular PASCAL VOC 2007 dataset using conventional accuracy metrics such as the Intersection over Union (IoU), the area under the curve (AUC), and the proposal recall under various threshold values. We conduct performance benchmarking with the well-known Selective Search and Edge Box methods. Furthermore, we compare their requirements in terms of the number of proposals, computational complexity and model sizes to achieve such accuracy. Finally, we demonstrate by experiments that OPPHop outperforms Selective Search and Edge Box in both efficiency and accuracy.

OPPHop: A Light-Weight High-Performance Object Proposal Method

In the pursuit of Artificial General Intelligence (AGI), Machine Learning (ML) has achieved immense progress by exploiting parallel classical computing. Because of this success, our reliance on these techniques in autonomous systems has exponentially grown. However, throughout the scientific community there is growing concern of our increasingly rapid approach towards the theoretical limits of classical computation, better known as the end of Moore's Law. To continue these advancements, the adaptation and exploration of novel computational platforms is absolutely necessary. Biologically accurate, energy efficient neuromorphic systems that mimic the incredible efficiency of the human mind and the theoretically powerful optimization potential of fully entangled quantum systems are particularly promising arenas for enabling future advances. In this work, we compare these two flavors of novel substrate in detail by applying them to an identical challenge, and benchmark their performances in power consumption, time to solution, and reconstruction error against a classical CPU approach.
Our focus is on a subset of Hopfield networks, specifically, sparse attractor networks in which a uniform applied field is used to globally suppress activity. This encourages solutions consisting of a minimum number of active elements. We solve this sparse coding problem using the biologically inspired Locally Competitive Algorithm (LCA) on the D-Wave quantum annealer and the Intel Loihi neuromorphic spiking processor in two distinct ways, and compare the solutions to those found classically using Orthogonal Matching Pursuit (OMP). The Fashion-MNIST data set was chosen and dimensionally-reduced by sparse Principal Component Analysis (sPCA) under the constraint that a graph-based clustering metric (Henze-Penrose) remain unchanged. This enables the problem to be mapped identically to both devices. A sign flipped second data set was appended to the original to give each class a mean zero distribution, effectively creating an environment where the data could not be linearly separated. An early-in-time normalization technique for Loihi is presented along with analysis of optimal parameter selection and unsupervised dictionary learning using a local Hebbian rule for all variations. Studies are ongoing, but preliminary results suggest each computational substrate requires casting the NP-Hard optimization problem in a slightly different manner to best capture the individual strengths of the substrates, and the new Loihi method allows for a more realistic comparison with the D-Wave, while also drastically reducing the time needed to compute the solution with statistically improved reconstructions. Both substrates trail the classical approach to this problem in terms of reconstruction error, but D-Wave is the only one performing true binary sparse coding and Loihi provides immense energy savings along with viable solutions for both the full simulation and, especially, the early in time normalization variation.
The development of these substrates, and their use in expanding the machine learning parameter space, has multiple applications to Naval operations. For NP-Hard problems, our development of practical applications for quantum annealers lends itself to centralized decision making. For processing at the tactical edge, our progress in characterizing rapid neuromorphic solutions at low power are important in determining performance envelopes.


Brain Inspired Quantum and Neuromorphic Machine Learning

MetaTagger is an accredited GOTS ML test and evaluation toolset designed to streamline ground-truth data labelling and performance testing of imagery and Full Motion Video (FMV) based models. MetaTagger accelerates model testing to increase performance transparency, user trust, and model adoption. A smart collection of tools enable fast image labeling for training data purposes. Our custom-built tool called AutoTagger uses a Single Object Tracking (SOT) algorithm to automate object detection, labeling, and creation of ground truth datasets in less than half the time of manual approaches. Once Ground truth is created, FVM inference can be evaluated in minutes by simply uploading inference to the application and letting the MetaTagger algorithm work in the background. Once evaluation is completed, a filterable reporting tool allows custom reports to be generated and exported for future analysis. MetaTagger is the one-stop-shop for testing and evaluation of object detection and object classification algorithms.

MetaTagger - Computer Vision Testing & Evaluation Toolset

We approach the problem of image captioning with Cycle-Consistent Generative Adversarial Networks (CycleGANs). Due to CycleGANs' ability to learn functions to map between multiple domains and use duality to strengthen each individual mapping with the usage of a cycle consistency loss, these models show great promise in their ability to learn both image captioning and image synthesis and to create a better image captioning framework. Historically, cycle consistency loss was based on the premise that the input should undergo little to no change when mapped to another domain and then back to its original; however, image captioning presents a unique challenge to this concept due to the many-to-many nature of the mapping from images to captions and vice-versa. We have combined the state-of-the-art image synthesis model from Hao Zhang, StackGAN, with a modified version of Kelvin Xu's Show, Attend, and Tell image captioning model in a CycleGAN architecture to create a robust image captioning framework that is able to create diverse captions for each input image. Due to the duality of CycleGAN, we will also explore our model's capability to successfully generate relevant images based on an image description to test its ability to perform the opposite task and observe CycleGAN's ability at learning mappings between two domains from different mediums. We will be discussing our model's performance on the following datasets: Oxford Flowers, Caltech-UCSD Birds, and Microsoft Common Objects in Context (MSCOCO) datasets.

Premium content

Reinforcement Learning for Tactical Decision AI

Next from NAML 2021

Mindful™: Competency Aware Machine Learning

Similar lecture

Leverage Artificial Intelligence to Learn, Optimize, and Win (LAILOW) for the Marine Maintenance and Supply Complex System