United States

Despite existence of large amounts of unclassified documentation, describing military engagements such as TTPs, historic books etc there is a shortage of formal event data that can be used for M&amp;S purposes. We propose a hybrid AI/ML methodology to extract low-level event data from documents and aggregate them to higher level of abstraction that M&amp;S systems can operate on. The same exact methodology can be used to extract low-level event data in real-world situations from text messages that soldiers exchange on the battlefield to “light the simulation decision tree”, thus linking simulation and actual events together. 


NAML 2021

Acquiring realistic training data for modeling and simulation(M&amp;S)

extracting training data;nlp; event aggregation

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

Despite existence of large amounts of unclassified documentation, describing military engagements such as TTPs, historic books etc there is a shortage of formal event data that can be used for M&S purposes. We propose a hybrid AI/ML methodology to extract low-level event data from documents and aggregate them to higher level of abstraction that M&S systems can operate on. The same exact methodology can be used to extract low-level event data in real-world situations from text messages that soldiers exchange on the battlefield to “light the simulation decision tree”, thus linking simulation and actual events together. 


Acquiring realistic training data for modeling and simulation(M&S)

poster

In order to achieve large-scale acceptance of ML/AI across the Navy and DoD, algorithms must be efficient and work well with available hardware. Currently, a large portion of the ML/AI workload is dominated by cloud-based server farms (Google, Amazon, etc.), where massive datasets and computational resources can be leveraged. However, many DoD applications cannot use these services due to limitations in memory and connectivity. Recent growth in the Internet of Things has led to the commercial development of low-SWaP hardware platforms specifically designed for many flavors of ML, including GPUs, neural network accelerators, and neuromorphic processing cores. However, the underlying operation of ML/AI algorithms on these platforms are not well understood. 
Warfighting of the future will be waged not only with bombs and bullets, but also with smart systems through information warfare at the tactical edge. The use of ML/AI as part of information warfare is an area of projected growth for the Navy and DoD. While there is a strong growing workforce with experience in the software and computer science aspects of ML/AI, implementation of this technology into DoD systems will require expertise in the implementation of ML/AI hardware as well. An understanding of the hardware platforms currently available, along with the current trajectory of S&T efforts, will allow scientists and engineers to implement systems with ML/AI technology currently being developed improved SWaP.
The decision of which hardware platform to deploy and implement specific ML/AI algorithms is not always clear. They each have their own advantages and limitations that allow for their maximum use and performance for training and inference but there are relevant metrics such as performance, power efficiency, and usability to aid in a successful implementation of research for real world applications. This paper will explore some of the unique capabilities and advantages of familiar platforms including the NVIDIA Jetson Nano, Raspberry Pi with the Intel Neural Compute Stick, and Google Coral Dev Board. 


Evaluating ML/AI Edge Computing Devices for Real World Applications

The Predictive Cost of Readiness Engine (PCORE) expands on the Digital Aviation Readiness Technology Engine (DARTE) by combining financial information with traditional readiness data (manning, training, and equipment) to focus on predicting an innovative new aircraft-level readiness metric for the Naval FA-18 fleet: return on investment (ROI). For this metric, the investment into an aircraft is defined as the cost of people (maintenance man hours) and parts, while the return from an aircraft is flight hours. A temporal pattern attention mechanism enhanced long short-term memory (LSTMA) network is used to predict future aircraft ROI. Predicting ROI allows maintainers and leadership to more accurately make decisions regarding part priority, whether to perform costly maintenance actions, and how best to allocate finite resources such as material, labor, and funding.

Improving the Efficiency of the Naval FA-18 Fleet Using The Predictive Cost of Readiness Engine (PCORE)

Machine Learning Approach to Corrosion Maintenance Identification

Underreporting fleet corrosion maintenance records results in the data consumers, analyst, and decision makers with inaccurate information to pursue corrosion mitigations in order to maintain equipment mission readiness. The NAVAIR methodology of recording equipment maintenance entails completing a Visual Information Display System/Maintenance Action Form (VIDS/MAF). The VIDS/MAF includes underutilized fleet narratives that describe the equipment discrepancy and the corrective maintenance action. This research investigates whether an unsupervised machine-learning algorithm can process fleet text narratives and identify the corrosion maintenance records.

The research proceeded by employing Natural Language Processing (NLP) to the VIDS/MAF unstructured text data. The researchers acquired data comprising of a single piece of NAVAIR equipment that contains a generous quantity of both corrosion and non-corrosion related maintenance records. The unstructured maintenance text was preprocessed utilizing lemmatization, special character removal, lowercasing characters, removing punctuations and numbers, and stripping white space. The researchers applied a Latent Dirichlet Allocation (LDA) topic algorithm to the processed unstructured text iterated across a sequence of topic quantity hyper-parameters. A harmonic mean function identified the optimal number of hyper-parameter topics. The researchers applied the LDA algorithm with the optimal number of hyper-parameter topics against the dataset. This LDA model identified additional potential corrosion maintenance records by a factor of five.

The objective of this project consisted of determining whether NLP techniques can identify corrosion maintenance records beyond the capability of the structured fields found in the VIDS/MAF. The research was successful in identifying additional probable corrosion maintenance records. Furthermore, this research has the potential for utilization in analogous applications, such as identifying underreported wiring issues that is applicable across the Navy enterprise.


Machine Learning Approach to Corrosion Maintenance Identification

This approach shows the use of robotics operating system (ROS) framework to improve the processing time performance of stochastic optimization systems as they are applied to large data sets for robotics applications. As a case study, we implement the genetic algorithm using ROS to optimize its processing time performance. Genetic algorithms are commonly used to find optimal solutions to optimization problems with different levels of complexity, however, they require lengthy data processing time when applied to large data sets. By making use of ROS action messages we can perform parallel processing in asynchronous mode which allows us to breakdown the processing of large datasets into smaller ones which can then be computed in parallel at separate servers therefore greatly reducing computation time. Another advantage of using ROS action messages, is their asynchronous aspect in which they communicate results among clients and servers therefore allowing the client to perform other tasks while waiting for results from other servers. Once these data sets have been evaluated and processed, they can be put back together into a large data set, and fed back into the genetic algorithm for further optimization therefore improving the processing speed of the evaluation module for the genetic algorithm while also avoiding constraints in memory resources due to a single server handling all the data processing. While past research to improve computation time of Genetics Algorithms have made use of parallel processing, using the ROS framework allows us to have the advantage of having feedback about the status of each separate processing task in a dynamic manner without having to add much complexity to code development.

Use of ROS framework for parallel computation of Stochastic Optimization Systems

The Georgia Tech Research Institute has been supporting the JAIC Joint Logistics virtual sensing National Mission Intititive to develop the VSTAR digital twin capability. VSTAR is a digital twin MBSE for the H-60 4G accelerometer targeted at data reconstruction for vertical acceleration to model, simulation, reconstruct, and data stitch an AI/ML synthetic signal to replace corrupted flight data. This data stream is critical due to post-flight analysis and maintenance planning (RUL, CRTs, etc) being directly based off damage calculations using the flight regimes experienced by the H-60 determined from the 4G accelerometer measurements. The poster discusses the hybrid multi-physics and AI/ML MBSE approach leveraged to develop the VSTAR capability.

Digital Twin Virtual Sensor Development for H-60 Accelerometer Data Reconstruction

The maritime transportation industry faces a number of logistical challenges, including navigating safely under bridges, reducing costs due to intermodal transfers, and handling various water and weather conditions. In November 2020
alone, maritime safety incidents occurred at the Centreville Turnpike Bridge near Chesapeake, Virginia and the Barataria Bridge near Lafitte, Louisiana. Such incidents incur major economic losses due to the negative impact of the closure of roads and maritime passages in their respective areas. Trabus Technologies (TRABUS) has created the River Information Systems Enterprise (RISE) framework to address many of the logistical challenges faced by mariners along the U.S. inland waterways. At the heart of RISE is an artificial intelligence (AI) based river level prediction system known as the River Prediction Analytics Toolkit (RPAT), that forecasts hourly river heights out to seven days in the future. These predictions are made using deep neural network models by processing historical hydrological readings at over 300 different river gage sites within the U.S. inland waterways and weather observations from nearly 100 nearby Automated Surface Observing System (ASOS) sensor network stations at U.S. airports. River level forecasts provide mariners with predictive analytics that enable them to plan their trips, make optimal decisions on how much cargo to load on barges, account for varying climatic conditions such as drought and early spring snow melt, and safely navigate under bridges. Predictions generated by RPAT are useful in additional maritime decision support tools such as the Bridge Air Gap (BAG) tool. US Army Corps of Engineers (USACE) provides metadata for many bridges along the United States Inland Waterway system. BAG tool calculations are made on the fly by subtracting water level elevation from the bridge low steel elevation. The water level elevation incorporates both a gage datum elevation and the RPAT projected river height. The predictions improve on existing river level forecast methods such as the National Oceanic and Atmospheric Administration (NOAA) river predictions and the National Water Model (NWM) by increasing the reporting frequency and prediction accuracy. Due to the lack of a publicly available repository of bridge air gap recorded data, prediction errors on bridge clearance must be estimated. Bridge low steel elevation and gauge datum elevation are both theoretically constant, so errors in clearance calculations depend solely on RPAT errors. For many sites in the U.S. inland waterways, 7-day RPAT projections have a root mean squared error (RMSE) of 1.5 feet or less and an accuracy range between 75-98%. This means that a typical BAG tool prediction also should have a RMSE of 1.5 feet or less and an accuracy between 75-98%. This presentation discusses the generation of the RPAT models, highlights issues during the development of the BAG tool, and offers suggestions for future work involving bridge navigation, river currents and cargo load optimization.

River Level and Bridge Air Gap Predictions for United States Inland Waterways

A common focus across the DoD is: How can our tools and information help improve fleet readiness? It is evident that Data Science plays an immense role when it comes to decision making and the analysis of issues. The increase of data digitization only furthers these efforts. One avenue of interest is Natural Language Processing (NLP), which is a subset of Artificial Intelligence (AI). NLP focuses on developing models to understand human language. This work focuses on a specific area of NLP called Word Embeddings, which maps words to vectors of real numbers. These numerical representations of words helps enhance modeling tasks such as multiclass classification. Our use-case demonstrates how Word Embeddings were able to enhance classification results of maintenance narratives.

The maintenance narratives are free-form text that the sailors input when completing maintenance actions. The dataset of about 60,000 records is labeled and has six different categories of maintenance. The process is then broken down into two approaches, the classical approach and then the word embedding approach. The classical approach involves the textual data being broken down into tokens and forming a Document Frequency Matrix (DFM). This DFM is then used to build supervised classifiers such as Naive Bayes and Gradient Boosting Machines. This is then compared to the Word Embedding approach, which also tokenizes the textual data, but then uses it to form word vector representations of each token. These vectors are then used to build a supervised classifier that yields better results than the classical approach.

Word Embedding models allows for a deeper understanding of human language as opposed to classical approaches. It is able to absorb more context and syntactic similarity from the text. The corpora of data that is used to train it determines how the language model will behave. In our case, the model is centric to naval maintenance data. However, such work could be applied in various fields where multiclass classification would be beneficial.


Maintenance Classification using Word Vectors

Recent budget constraints require the Naval Aviation Enterprise to maximize readiness while minimizing cost. As such, it is important to know which aircraft are the most expensive to keep in a mission capable (MC) state and what factors affect this cost. An AI/ML model is developed as an extension to the Digital Aviation Readiness Technology Engine (DARTE) family of models to determine the factors that lead to a high average cost per day of MC for an aircraft in the Naval FA-18 fleet. This model then predicts if the average cost per day of MC will be above a fixed cost threshold in the future. Various model architectures are explored, including tree-based ensemble methods and long short-term memory (LSTM) networks. These models are trained on a combination of traditional readiness datasets (manning, training, equipping) and financial datasets. The resulting models are then examined with explainable artificial intelligence (XAI) techniques to obtain a rigorous and comprehensive feature importance which may be used to determine past and predict future aircraft status.

Understanding and Predicting the Future Cost of Readiness for an Aging Naval FA-18 Fleet

The Strike Force Interoperability Office (SFIO) assembles and filters large quantities of data from a number of sources to produce Task Group readiness briefs, providing guidance in assisting mission planners with their recommendations. This process is historically manually intensive and significantly impacts personnel workload and time better spent conducting analysis.

PRIOR, a cloud native application, provides:

Secure and reliable semi-automated data driven analysis which leads to time and cost savings
Flexible microservice architecture and use of Natural Language Processing (NLP) techniques which support the transformation of large quantities of disparate data to assist with mission readiness assessment.


Planning Resources for Interoperability Options and Readiness (PRIOR)

The current state of machine learning and artificial intelligence mirrors where digital technology was 20 years ago; systems are being developed without rigorous testing and with limited (if any) security mechanisms. This lack has created challenges across both the DoD and industry with respect to the amount of trust in the safety, robustness, and performance of intelligent systems, resulting in degraded capabilities and technological utility for the warfighter. We argue for early, often, and specific consideration of security and robustness as part of a comprehensive DevSecOps process – one that incorporates tooling, methodology, and changes to acquisition strategy.

Premium content

Acquiring realistic training data for modeling and simulation(M&S)

Next from NAML 2021

Evaluating ML/AI Edge Computing Devices for Real World Applications

Similar lecture

Leveraging Jupiter for Decision Advantage Across the Department of the NavyDepartment of the Navy