United States

Recognizing that “garbage in” leads to “garbage out,” many organizations that employ machine learning and other types of data analytics strive for a zero-defect data ingestion process. While these efforts are critical, they are not infallible. Therefore, programs should adopt a multi-layered approach to risk management that includes mitigation plans for addressing issues that occur after faulty data has corrupted data analytic efforts and potentially influenced program-level decisions.

The impetus for what would become the Quality Risk Management Framework was a planned change to an analytics environment’s data ingestion process. The authors realized that the upcoming changes would likely increase the risks relating to data management. The authors also realized that these risks might not be discovered until after that faulty data had been used in subsequent analysis and decision support efforts, including those that may impact the safety of personnel. With these potential issues in mind, the authors performed a literature review in the hopes of finding an existing framework that addressed their concerns. While the review did not uncover a suitable framework, it did uncover elements that the authors would later incorporate into their Quality Risk Management Framework.

The authors’ resulting framework takes the form of a matrix where the columns relate to six quality dimensions and the rows relate to the six phases of the intelligence lifecycle. Using this matrix, the authors were able to identify 36 unique areas where quality risks could arise. Then, the authors developed mitigation plans for each area of potential danger. With these mitigation plans established, the authors are confident their program will be able to rapidly respond to the discovery of a quality issue regardless of when the program discovers it.

NAML 2021

What if we were wrong? – Creating and Applying a Quality Risk Management Framework

intelligence lifecycle

quality dimensions

quality risk management framework

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

poster

Over the years, the use of data throughout the V-22 Osprey program steadily grew. Initially, automated software analyzed flight data to inform maintenance decisions. A few years later, analysts mined maintenance data to inform program-level decisions. Then, a few years after that, analysts leveraged artificial intelligence and machine learning on dozens of different V-22 data sets to gain deep insights on aircraft readiness. While each of the three program areas described above was successful, their roadmaps and maturation plans were independent until now. This presentation describes how overlaying a decision framework on the “big picture” enabled system-level optimization of the formerly siloed initiatives.

In the 1950s, John Boyd, an instructor at the Air Force’s Fighter Weapons School, realized that the pilot who made decisions faster than his opponent would win the aerial dogfight. Studying this further, Boyd realized that decision-making was a continual progression through four activities: observe, orient, decide, and act. Boyd subsequently named this decision-making process the OODA Loop. When the authors of this presentation took a system-level view of the V-22 maintenance support infrastructure, they discovered those same four elements were also critical factors in their program's success. They also realized, as did Boyd, that their goal should be to get through the decision-making cycle as quickly as possible.

With this system-level view and a goal of optimizing the OODA Loop, the authors examined the existing technologies and capabilities supporting each of the four decision-making activities. Next, they identified what was necessary to accelerate the program’s progression through the OODA Loop. Then, they set out to fill those gaps. As these new capabilities are introduced, the program’s decision-making system will become increasingly optimized. As a result, the V-22 program will gain maximum benefit from its machine learning and other data analytic efforts.

Applying a Decision Framework to Increase Machine Learning Impact on the V-22 Osprey Program

Naval Officers at the Office of Naval Research Reserve Component (ONR-RC) have developed a prototype taxonomy for Artificial Intelligence (AI), focused on Naval and DoD audiences and applications. Our taxonomy establishes 4 pillars of AI: Data, Mechanism, Function, and Naval Task. Sub-taxonomies within these pillars are based on authoritative resources from organizations such as NASA and the Association for Computing Machinery (ACM). We posit that any AI system can be decomposed and classified according our structure, which mitigates problems of mutual exclusivity that other taxonomies suffer. Organizations like ONR can use taxonomy to organize and shape their AI research portfolio. Furthermore, we hypothesize that concrete proposals for future AI systems can be developed by constructing their Data, Mechanism, Function, and Task components. This hypothesis is currently being tested by students at the Naval War College, who will write such proposals for the ongoing course AI for Strategic Leaders. We offer this prototype taxonomy for feedback and criticism, so that we may continue to improve its utility for wider DoD audiences.

Taxonomy of Artificial Intelligence

The objective of Self-Localization and Intelligent Mapping (SLIM) research is to expand the Navy's AUV capabilities through artificial intelligence (AI) to static undersea sensors and/or dynamic undersea groups, improve autonomous perception and data fusion, and generate world models from individual sensing while localizing the sensors and UUV swarm within the model. In the current study, we present the performance of the proposed method in a networked robotic scenario using robot operating system (ROS) Gazebo for simulation of physical environment.

A Novel Hybrid Method for Self-Localization and Intelligent Mapping of AUVs

NSWC Corona is developing the Perceptive Engine with Natural Language Processing (peNLP) application with the vision of enabling the digital transformation of documentation and processes, and bringing advanced data analytic capabilities to analysts’ fingertips. Its intuitive interface will include: displaying all documentation in one searchable wiki-like location with insightful metadata, tracing of concepts through related files, comparing and contrasting within the corpus to reduce redundancy and discover conflicts, and a better overall understanding of the documents through in-depth analyses and pattern recognition algorithms (i.e. clustering, changing word meaning over time, metrics) for future integration across different data sources (i.e. logistics, specifications, maintenance, RMA, funding). 

The peNLP application has made rapid advancements in its user interface in integrating NLP capabilities with automated Python scripts. With both “Document Search” and “Document Knowledge Graph” views, peNLP utilizes in-house, Corona-developed Python scripts to derive relevant metadata for documentation (i.e. top keywords, action items, date, section). Use of the BM25 search algorithm allows in-depth search capabilities of not only the documentation metadata but also of sections of interest within the documents. Additional information for users is provided by advanced NLP capabilities such as abstractive and extractive summarization as a quick read for end users, topic modeling, hypothesis testing between requirements and documents, Named Entity Recognition (NER) and syntax parsing for identifying roles and responsibilities, and textual entailment using AllenNLP for understanding overlap of similar language between different documents. All of peNLP’s information is integrated together through its extensive Document Knowledge Graph, which not only behaves as a visual aid in finding connections between documents but also serves as the backing database for tracing concepts and requirements through corpora.


Perceptive Engine with NLP (peNLP)

The task of classifying objects at range in low resolution imagery is challenging due to a lack of clarity of the object and spatial information surrounding the object. Deep learning methods, while state-of-the-art when using high-volume, high-resolution data, often fail in this case. Additionally, it has been shown that these deep learning methods are highly susceptible to variations to the input data (e.g. illumination) and therefore can fail to generalize to new data.

In this presentation, we use the tracks of objects derived from video data to study the dynamics of the objects motion to distinguish between targets and confusers. Using a topological approach of embedding the motion tracks using a time-delay embedding, and computing persistent homology, we can identify targets and confusers with high accuracy. For comparison, we utilize deep learning approaches on images of the objects as well as the motion track data. We find the topological approaches are better able to differentiate the different types of objects at a variety of track lengths. In particular we see the most significant performance gains for the shortest track lengths. Moreover, we show that the topological approaches presented are better able to generalize to new data from tracks collected separately, and under different conditions, from those used to train the model. These results build on work showcased at the 2020 NAML and will also describe our first foray into applying these techniques to maritime activity classification from AIS data.

I Spy in the Sea and Sky: Classification of Maritime and Aerial Targets
from Tracking Data

The process of unsupervised anomaly detection, which aims to identify outliers in data sets without the use of labeled training data, is critically important across a variety of domains including medicine, security, finance, and defense. Here we describe a new learning paradigm that combines batch stochastic gradient descent algorithms and loss functions tailored for the anomaly detection problem to reliably identify outliers regardless of the underlying data set or modality. We describe data models that can leverage manifold learning techniques, autoencoders, and generative adversarial networks and show how the training paradigm may be used to learn the unknown parameters that govern the underlying models such that outliers may be reliably identified without relying on previously labeled data. We show here that our framework demonstrates a clear improvement over state-of-the-art techniques on established benchmark data sets for unsupervised multivariate anomaly detection and further demonstrate value within Navy-relevant scenarios.

A New Learning Paradigm for Reliable, Unsupervised Anomaly Detection in Navy-Relevant Scenarios

Traffic Management (TM) is a key functionality in Navy tactical network that prioritizes and shapes traffic to meet the required network Quality-of-Service (QoS). Due to the highly disruptive, dynamic, and constrained nature of tactical networks that are not present in commercial or enterprise networks, any solution tailored to tactical networks must understand the state of the network and adapt to network condition changes. In a bandwidth constrained network, it is also imperative that a solution limits or reduces overhead. Because the packets traversing the network hold a wealth of knowledge regarding the state of the network, collecting and monitoring packets allows network devices to glean valuable information regarding the state of the network, reducing the need for overhead traffic and allowing limited resources to be more efficiently used for application traffic. This project investigates using Deep Packet Inspection (DPI) in combination with Machine Learning (ML) techniques to yield timely understanding of network conditions that can be used to tune traffic management, while minimizing overhead. 
An example of using DPI to glean network condition is leveraging the application traffic’s Transmission Control Protocol (TCP) Sequence Number Acknowledgments (ACKs) to determine bi-directional link liveness that routing protocols such as Open Shortest Path First (OSPF) also detects. This approach is successful because there are well-defined protocol data in different protocols that are used to detect the same network condition, and there is redundancy when the different protocols are used simultaneously. However, the space of DPI is extremely broad, both in terms of techniques, which extend beyond leveraging well-defined protocol data such as TCP ACKs, and the sheer volume of raw data. Even in a low bandwidth environment, control traffic, data traffic, and the meta-data embedded in all traffic that is available for analysis can be at a level that is difficult to comprehend. An accomplishment of this research is the use ML algorithms to extrapolate link quality. Experiments evaluating these data using different methodologies showed that ML algorithms performed the best at extrapolating link quality given the same raw packets. Using ML to monitor network conditions for TM provides the end user with a tool for autonomous traffic optimization under changing field conditions and network resources.
The meta-data extracted through DPI holds key information about network conditions that can be utilized to dynamically react to a change in environment and traffic in additional ways beyond pre-defined static configurations. Future work includes improving the confidence level of the predicted network conditions as well as incorporating more intelligence in the data plane with the aim of ensuring that TM meets the required network QoS reducing routing protocol overhead.

Deep Packet Inspection with Machine Learning Enabling Traffic Management in Constrained Environments

In recent years, machine learning has made strides forward in the domain of computer vision. Some teams have attempted to use existing neural networks to analyze images generated by Particle Image Velocimetry (PIV). PIV measures dynamic flow by seeding a fluid system with particles visible to a camera but too small to disrupt the flow itself. Recently some researchers have used a neural network trained in a supervised fashion on synthetic data and been able to obtain results as accurate as current state-of-the-art methods. Another group has used a similar neural network but with an unsupervised training method resulting in a model that performs better than the supervised model for less ideal input data. Machine learning’s main advantage for PIV over the current state-of-the
art is its speed. Both conventional and current machine learning methods for analyzing PIV images can struggle to accurately analyze imagery from field measurements, where resolution may be lower, or the image may be noisy. In this work, we build a robust model for analyzing noisy and low-resolution PIV data using a semi-supervised learning approach. We benchmark accuracy and computational performance against the current state of the art on real and synthetic data and explore whether such a method will perform better for noisier input and more challenging domains.

Machine Learning For Robust Particle Image Velocimetry

Commercial satellite data collection enables vessel tracking via Automatic Identification System (AIS) observations with global coverage and high revisit rates at relatively low cost. USG maritime analysts can use such data for monitoring and analysis via maritime domain awareness (MDA) tools. While the coverage and revisit rates make these commercial data relevant for addressing a wide range of maritime domain problems, the resulting large data volume and velocity hamper human analysis from fully exploiting these data. This presents an opportunity for application of machine learning (ML) analytics to automate low-level detection of activity of interest from the data streams. This would help analysts more quickly answer time-sensitive questions and focus on decision-making instead of manual data analysis.

For DARPA’s Geospatial Cloud Analytics (GCA) program, we developed flexible ML analytics for recognizing maritime behaviors from AIS data. These analytics extend our Multi-INT Analytics for Pattern Learning and Exploitation (MAPLE) family of ML capabilities. We deployed these analytics as a cloud-based service (MAPLE as a Service – MaaS) to perform automated detection of potential illegal, unreported, and unregulated (IUU) fishing activity at tactical time scales. We validated our algorithms via a quantitative performance evaluation. We demonstrated that our approach achieves comparable performance to published results for state-of-the-art fishing detection algorithms. The baseline state-of the-art approaches are fishing gear type specific (i.e., one approach for longliner, another for trawler, and a third for purse seiner gear). In contrast, MaaS uses a single approach for all of these fishing gear types. This evaluation process used an anonymized AIS data set with fishing behavior ground truth provided by Global Fishing Watch for model training and validation. We also employed commercial AIS data from two different regions (U.S. Pacific Islands & coast of Gabon) for additional performance evaluation.

Since MaaS learns behavior models from relevant examples, we can apply MaaS to other maritime behavior recognition problems (e.g., mine laying, piracy, and research vessel activity, as well as multi-vessel behavior like transshipment or aggressive vessel formations) without having to design new algorithms for each problem. The cloud-based analytics as a service approach enables rapid integration with multiple data sources and consumers. This approach makes use of open standards that enable machine-to-machine interaction for data access, analytics processing, and analytic product publication. It admits ready extension of existing MDA tools to enhance human-machine interactions. We are currently working to integrate a prototype MaaS capability for automatically detecting fishing behaviors from AIS data with the USG web-based SeaVision MDA tool.

Machine Learning Provides SeaVision with Maritime Behavior Recognition

Distinguish a lake from a flood seems easy, but when you’re looking at an aerial photograph, factors like angle, altitude, cloud cover, and context can make the task more difficult. And when you need to identify 100,000 aerial images in order to give first responders the information they need to accelerate disaster response efforts, you need to combine the speed and accuracy of machine learning (ML) with the precision of human judgement. Join to learn more about how building an end-to-end pipeline that ingests LADI and other related data using a classifier that incorporates Amazon Augmented AI can speed access to data and improve the outputs of the model for responders and the command center to extend situational awareness for planning and decision making.


Premium content

What if we were wrong? – Creating and Applying a Quality Risk Management Framework

Next from NAML 2021

Applying a Decision Framework to Increase Machine Learning Impact on the V-22 Osprey Program

Similar lecture

Leveraging Jupiter for Decision Advantage Across the Department of the NavyDepartment of the Navy