United States

If a feature in a dataset can be identified as a potential causal factor through an objective process, this data can then be used to feed any number of conceptual approaches to prevention of safety mishaps. One of the many ways in which this analysis can be attempted is via longitudinal analysis of specific traits attributed to personnel, either individually or in aggregate. That is to say, by examining specific traits as they change over time for their predictive value with respect to a specific outcome. Each moment that a variable is measured results in a feature that is highly correlated with the feature derived from the same variable in the previous or next moment. This presents unique challenges for fitting regression models since fitting highly correlated features will muddle the attribution of causal factors and result in a reduced ability to distinguish the relative importance of features. Additional challenges with regard to analyzing safety mishaps are that the data is frequently imbalanced, meaning that there are very few mishaps when compared to the number of instances in which a mishap did not occur, and that it is more important to be correct about the potential causal factors extracted than it is to be quick. This paper will walk through the process of comprehensively sampling the feature space, applying a progressive framework of winnowing tests designed to reduce the number of models considered to a computationally manageable number while removing irrelevant features, and then developing the results into a workable set of models. Case studies for this process using specific examples are available in the appendices.

NAML 2021

Longitudinal Analysis of Imbalanced Data Sets Using Logistic Regression for Safety Mishaps

multicorrelation

imbalanced

longitudinal

logistic

regression

safety

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


ICPR 2020

technical paper

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

Naval Tactical Messages, a centralized form of text communication capturing essentially every critical element of the Navy at large, are sent rapidly across the Fleet at a high volume on a daily basis. Each command gathers insights from Tactical Messages in a myriad of ways, with the ultimate goal of providing a formalized mean of disseminating action taskers and providing relevant information to necessary stakeholders. Project Hermes, a DON CDO project, seeks to pipeline all Naval Tactical Message traffic into the DON CIO Jupiter Analytics environment, automating the extract, transform, and load process to enable broad Artificial Intelligence (AI), Machine Learning (ML), and Natural Language Processing (NLP) capabilities and applications. Long term vision of Project Hermes seeks to provide Program and Project specific data analytics and visualizations in a rapidly developing tactical and operational situation, aiding Watch Floor situational awareness, enabling real-time analysis, correlating data and features to be utilized in model development of Operational Risk indicators at the unit level, and to apply these broad AI capabilities across the entire catalog of Naval Tactical Messages to be leveraged by the Navy as a whole.

Project Hermes – Naval Messaging

An initiative at NIWC Pacific called Cloud Based Data Analyics Cells (C-DAC) was designed and architected for the pursose of lowering the barrier of entry to world of data analytics. The goal of the project was to create an “Off the Shelf” analytics platform that will be available to any NIWC project. The C-DAC was architected for the cloud (NRD&E AWS) leveraging cloud native services and offerings. The C-DAC will leverage the latest cloud technologies that facilitate on-demand stand-up of unique C-DAC instantiations as well as simplify the maintenance and complience efforts needed to support an operational big data analytics environment. Project deliverables were cloud formation scripts to provision a C-DAC as well as playbooks for using cloud native services to perform data curation, data analysis and machine learning.

Lowering The Barrier Of Entry For Artificial Intelligence And Machine Learning

DISTRIBUTION A. Approved for public releaseas of 24February2021: distribution unlimited.

Cyber threats constantly evolve, making detection of novel attacks difficult. These threats and attacks are often anomalous, especially for large scale networks that generate a significant amount of normal traffic. The ratio of normal to anomalous network traffic can cause extreme dataset imbalances, making them poorly suited for traditional Machine Learning (ML) algorithms. Anomaly detection algorithms provide avenues for identifying these types of cyber threats. Evaluation of features contributing to the identification of anomalous cyber events is critical in determining the type of cyber threat that the anomalous event may represent. 

We evaluate feature contribution using two well-known anomaly detection algorithms, Local Outlier Factor (LOF) and Isolation Forest (IF). Each of these unsupervised learning algorithms identifies anomalous data differently; where LOF uses regional density methods, IF uses decision trees. We evaluated a Euclidean-based measurement for LOF and a localized Depth-based Isolation Forest Feature Importance (DIFFI) method for IF to extract local feature importance. We demonstrate the extent to which feature evaluation methods are dependent on the anomaly detection algorithm used, and provide insight to features important for anomaly detection in cyber data using synthetic and real world data. 

Finally, we discuss the integration of anomaly scoring explainability into a novel cyber platform called the Pacific Ecosystem for Cyber (PEcoC, pronounced “peacock”). PEcoC leverages local explainability to accelerate analyst response, increase the body of knowledge available to analysts, and to lower barriers of entry for trainees and junior analysts. Ultimately, integrating local explainability of anomaly detection provides useful and important context for expert cyber analysts responsible for network defense. Anomaly detection within PEcoC complements existing defensive-cyber TTPs, signature-based detection systems, and augments analysts’ workflows and analytical options.

DISTRIBUTION A. Approved for public releaseas of 24February2021: distribution unlimited.

Feature Evaluation of Anomaly Detection Models for Cyber Data

We discuss issues in trying to apply AI to procurement tasks. We conducted surveys with procurement professionals who had implemented AI methods in procurement to determine their goals with the AI, the obstacles they encountered, and their assessment of the success of their projects.

A survey of adoption of commercial-style artificial intelligence for procurement

Mr. Sasala provides a compelling arugment on why data readiness drives warfighting readiness, and how the Department of the Navy will be using to drive organizational performance across the DON, including the USN and USMC.

DON Data Transformation

The Department of Defense (DoD) needs methods to be able to quickly ingest and extract relevant information from large quantities of unstructured documents that may vary greatly in length, content, variety, theme or topic, formality, and language. For example, Special Operations Forces (SOF) are often called in to participate in so-called “smash-and-grab” missions, whereby large quantities of previously unviewed and undocumented Captured Enemy Material (CEM) are obtained and must be examined for relevant intelligence information; however, searching and sifting through the potentially wide range and quantity of content obtained as CEM can be difficult and time consuming via manual inspection. Supervised methods of searching for standard labels are insufficient, since new threats may surface at any time and retraining supervised models introduces extra overhead for the pipeline. Automated methods are required for quickly parsing, separating, and identifying relevant or irrelevant sources of information. We outline an ongoing research and development effort aimed at building an automated, modular text pipeline that would ingest documents, translate them into English, perform unsupervised topic modeling to generate relevant topics and themes in messy data, and pass results on to the analyst so that they can avoid the ‘noise’ and more immediately inform their missions. The pipeline utilizes a tri-gram based naïve Bayes classifier for language detection, an NSA language pair translator, hierarchical non-negative matrix factorization rank 2 (HierNMF2) for topic modeling, and a salience-based two-dimensional (2D) rendering algorithm to help visualize the results. This unique workflow is aimed at addressing the “first-pass” problem with examining raw, messy, unstructured, text-based intelligence such as CEM.

Disparate Data Distillation

There is a great deal of interest in Naval applications of reinforcement learning (RL) for decision making and autonomy. RL models typically need full information about an environment to make good decisions and successfully maximize the decision making policy. Some work has been done on partially observable RL, particularly through the study of partially observable Markov decision processes (POMDPs), but POMDPs assume that the part of the environment that can be observed is pre-determined. This work focuses on developing models that can excel at tasks and also choose which part of the environment to view in order to accomplish their task. The model is only rewarded for its success in the task but it also learns to choose which part of the environment to view. We demonstrate that RL agents learn where to look and are able to reach similar performance in Atari games such as Pong and Seaquest as when the models are trained in fully observable environments.

Learning to Both Observe and Act

Machine learning is a key application of artificial intelligence that can be used to solve complex tasks for optimizing workload, improving safety, and increasing the efficiency of our sailors onboard aircraft carriers in the Navy. Identification of aircraft, people, and equipment for awareness on the flight deck is currently performed manually through visual inspection by several sailors. This task is labor intensive, requires focus for extended periods of time, and requires multiple personnel stationed during all flight operations. PATRIOT is a research endeavor to apply computer vision techniques to aircraft launch and recovery (ALRE) to reduce data delays and errors, and increase situational awareness. PATRIOT tracks aircraft, personnel, and vehicles through existing camera feeds, and integrates the information with the Ouija board, the interface sailors currently use to view and modify the status of aircraft for flight planning. This work includes an overview and comparison of the current methods and preliminary results for the overall pipeline for PATRIOT, as well as the components for data collection and processing, image classification, pose regression, and logical analysis

Panoramic Asset Tracking of Real-time Information for the Ouija Tabletop
(PATRIOT)

Marines in combat require a rapid and flexible logistics capability responsive to the 21st century battlefield. The USMC’s supply and maintenance chain receives data feed from a long chain, including: maintenance, requisition, transportation, and finance processes, activities, and decisions.
Such a complex enterprise needs trusted deep analytics, including machine learning (ML) and artificial intelligence (AI) methods to achieve automation, improve readiness, and win in unexpected environment. 

In this paper, we first review an overall framework of leveraging artificial Intelligence to learn, optimize, and win (LAILOW) for a complex enterprise, and then show how the LAILOW framework is applied to the USMC supply and maintenance chain as a test case. We first show a collection of supervised machine-learning algorithms, from an open-source based on the python ML library scikit-learn and the Soar reinforcement learning algorithm, to predict business metrics such as predict probability of fail (POF). We then show an unsupervised machine learning method lexical link analysis (LLA) to discover associative and sequential patterns, combined with predictive models in a game-theoretic set up using the coevolutionary algorithms in the LAILOW framework, where demand perturbations introduced by high-impact and low-occurrence items propagate to the whole system. We show the LAILOW framework as predictive and simulation tool using data sets extracted from operational databases to improve total readiness, agility, and resilience for the USMC logistics enterprise.

To show the feasibility of the LAILOW, we have sampled data for a specific Table of Authorized Material Control Number (TAMCN, 2020) from the database Global Combat Support System-Marine Corps (GCSS-MC), including service ticket number level attributes related to maintenance data, supply data, and equipment usage data. The sample data set contains 2065 service numbers/tickets, where 599 (29%) have more than 65 days (65 days is the mean value of the days between the open and close dates for the data set), between their opened and closed date, which is the target variable or measure of performance (MOP) for prediction. 

We found Soar-RL results comparable in predictive accuracy for predicting the MOP. Since Soar-RL is also rule-based and explainable, it was selected and used in a simulation phase integrated with the coevolutionary algorithms. The simulation shows that the logistics solutions, on average, worsens in evolution (in terms of the fitness value), while the opponent, representing logistics tests, on average, improves in evolution in terms of the MOP. The algorithms systematically simulate and discover possible new tests or “vulnerability”, and evolved solutions or “resiliency” are also discovered. Therefore, the LAILOW framework provides a holistic predictive and simulation platform to improve total readiness of a resilient and agile USMC logistics enterprise.



Leverage Artificial Intelligence to Learn, Optimize, and Win (LAILOW) for the Marine Maintenance and Supply Complex System

Accountability is widely understood as a goal for well governed computer systems, and is a sought-after value in many governance contexts. But how can it be achieved? Recent work on standards for governable artificial intelligence systems offers a related principle: traceability. Traceability requires establishing not only how a system worked but how it was created and for what purpose, in a way that explains why a system has particular dynamics or behaviors. It connects records of how the system was constructed and what the system did mechanically to the broader goals of governance, in a way that highlights human understanding of that mechanical operation and the decision processes underlying it. We examine the various ways in which the principle of traceability has been articulated in AI principles and other policy documents from around the world, distill from these a set of requirements on software systems driven by the principle, and systematize the technologies available to meet those requirements. From our map of requirements to supporting tools, techniques, and procedures, we identify gaps and needs separating what traceability requires from the toolbox available for practitioners. This map reframes existing discussions around accountability and transparency, using the principle of traceability to show how, when, and why trans- parency can be deployed to serve accountability goals and thereby improve the normative fidelity of systems and their development processes.

This work was recently published at the 2021 ACM Conference on Fairness, Accountability, and Transparency.

Premium content

Longitudinal Analysis of Imbalanced Data Sets Using Logistic Regression for Safety Mishaps

Downloads

Next from NAML 2021

Project Hermes – Naval Messaging

Similar lecture

Using Machine Learning to Refer Patients with Chronic Kidney Disease to Secondary Care