United States

Introduction: Approximately 1 out of 4 Navy recruits will not complete their obligated service. Identifying factors influencing Navy attrition versus retention is pivotal for maintaining a skilled and motivated force. This study utilizes machine learning and variable importance measures to identify risk factors that contribute to Navy attrition versus retention. Variable importance measures are extracted from random forest and extreme gradient boosting decision trees among newly recruited Navy accessions. These classification tree methods are more robust and sensitive when dealing with several hundreds of predictor variables, offering superior performance relative to traditional multivariable analyses. Leveraging advanced analytics is crucial to better understand these factors for military organizations. Navy retention refers to the rate at which military personnel choose to continue their service beyond their initial enlistment period.

Methods: The inclusion criterion was Navy personnel with an accession date in 2016 and the outcome of interest was a discharge date before the end of obligated service (i.e., attrition). The analysis focused on 545 independent variables branched from demographics, medical outpatient visits, and outpatient pharmacy medications. Accession, discharge, and demographic information were obtained from the Career History Archival and Medical Personnel System. The medical and pharmacy outpatient data were obtained from the Military Health System Medical Data Repository. All research and study materials were conducted under IRB protocol NRHC.2022.2020 and followed all HIPAA and human subject research guideline and protocols.
.
Results: Random forest (RF), extreme gradient boosting (XGBoost), and logistic regression (LR) models were trained with different subsets of features to determine which set of variables provided the best prediction for attrition. Model performance metrics were generated describing the sensitivity, specificity, positive predictive value and negative predictive value, area under the curve, and classification accuracy for the validation data. Variable importance measures (VIM) were then assessed including mean decrease accuracy, mean decrease Gini impurity, gain, and cover. The resulting VIM identified 5 groups: mental health, alcohol related problems, occupations, gender, musculoskeletal and medical appointment/prescription compliance. The top 3 VIM and their overall relative risk on attrition were adjustment disorders (RR=1.39 higher), Seaman Specialists (RR=3.01 higher) and Electronic Equipment Repairers (RR=.44 lower). 
Among Navy accessions in 2016 (n=39,866), 28.15% (n=11,177) did not complete their obligated service. The best overall accuracy for predicting attrition and AUC was 81.7% and 90.0% from the RF model respectively. Both the RF and XGBoost models outperformed the LR.

Conclusion: This study demonstrates the effectiveness of variable importance measures as a valuable tool in decision-making processes within the context of the military personnel management and retention system. It also demonstrates the utility of machine learning models and the benefits when compared to traditional regression analyses. Ensemble approaches improve overall predictive performance and makes the model more robust and resistant to overfitting compared to a single decision tree.


NAML 2024

A Comparison of Variable Importance Measures in Predicting Navy Retention Among Navy Recruits in 2016

extreme gradient boosting

random forests

variable importance

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2025

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

technical paper

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

Please log in to explore this workshop.

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Opponent surprise should be considered a common occurrence in war. This research makes recommendations to ensure an AI provides realistic solutions. Consider a non-combinatorial version of chess where unknowns are introduced sometime during the game in favor of the opponent. For example, the opposing player’s pawn can move two squares forward multiple times. Adaptive algorithms, adjusting tactics during non-stationary game play, have been developed as a solution when time invariant novelty or emergent scenarios occur. Now consider a worst case where every several moves, the opponent is allowed to use his game pieces in surprising ways, where change is now time variant. The research offers an approach to maintain user trust through XAI techniques and describes a mathematical foundation to avoid poor recommendations and uncover emergent solutions to uncover confounder. Use of game trees and payoff tables, Event-Verb-Event chains of causal sequences, and Directed Acyclic Graph models introduce a do-calculus inspired backdoor weight adjustment and open world error contributions theorem explaining confounder bias and bounding the algorithm’s recommendation error. The research offers dataset design constructs that provide the algorithm with the ability to provide limitations to algorithm recommendations due to confounder bias and explain confounder error contributions within the recommendation. The design constructs result in a larger dataset size specific to encountering and more effectively handling novelty during game play, allowing for more realistic explanations to better maintain user trust and compensate for confounders in the algorithm’s recommended solutions.

AI dataset design recommendations to deal with unknowns

One of the biggest Navy challenges in fleet management is the security and assurance of software and hardware components being loaded or updated when ships come to port. Ship components that may otherwise be inaccessible or out of band may temporarily be connected through the boarding of maintenance staff that may have both physical and network access of critical components and/or through wireless updates and transmissions. Security and attestation ML tools that perform detection of anomalies are limited by the availability and quality of data.

Oracle will present topics in two areas that are critical in developing effective ML tools in response to outlined Navy challenges. The first is in establishing a digital twin environment in which complex engineering assets are represented by a “Golden System” based on thoroughly tested components. A digital twin is a virtual copy of a real-world asset, augmented by real-time data and analytics. The Golden System (i.e., digital twin) is used to enable the training of ML algorithms that perform critical functions such as anomaly detection by generating a baseline of telemetry, time series, and signatures. The second topic pertains to data quality through improved data sampling and preprocessing.

Oracle has developed Intelligent Data Preprocessing (IDP) innovations to optimize ML performance for a variety of Machine Learning use cases and data mining techniques. Use cases include prognostics, streaming analytics, prognostic cyber security, and EMI fingerprints for detection of counterfeit components in enterprise data center IT assets. The IDP innovations include Analytical Resampling Process (ARP), Unquantize (Unquantization of Quantized Sensor Signals), and Missing Value Imputation (MVI). The IDP algorithms have been integrated into a workflow referred to as a Data Preprocessing Pipeline (DPP). The use of the DPP has resulted in higher prognostic accuracy, and lower false-alarm and missed alarm probabilities in a variety of ML algorithm-based models. DPP packages are implemented in Python and MATLAB.

Oracle will present an extension of the concept of digital twins as applied to EMI fingerprinting to passively detect counterfeit components in electronic systems (in reference to hardware assurance and attestation challenges). A golden system is used to create a trusted signature. This signature is based on CPU activity from which EMI telemetry is derived and stored. Oracle’s Multivariate State Estimation Technique (MSET) 2 is used to detect counterfeit components by referencing the golden system derived signature. A research and development pilot is underway with an Oracle Government Defense and Intelligence (GDI) customer to demonstrate MSET2 EMI Anomaly detection above current innovation standard capabilities.

Benefits of Digital Twins and Improved Data Processing to Fleet Management Assurance

Summarization of Aviation Data for Maintenance Analysis

The future success of the warfighter will depend, in part, on artificial intelligence. In practice, many artificially intelligent algorithms, and in particular deep learning (DL) models, will have uncertain performance when encountering new scenarios, especially algorithms that rely on continual learning to maintain performance in dynamic environments. Under this effort, we developed a framework and accompanying experiments to determine if the system under test was not meeting performance requirements and to take appropriate actions such as warning the user or terminating the system under test. This safety framework is intended to integrate with machine learning software to allow them to more safely and easily transition to their target domain.

The framework includes several distinct components which can operate independently and can be replaced with new state of the art methods as they become available. The main categories of components include out of domain detection and monitoring, system runtime diagnostics and monitoring, world state prediction and data generation, probabilistic certification, and continual learning. A variety of modules have been implemented for each component which include both open-source modules as well as custom, in-house modules.

Experiments were performed to evaluate the framework’s performance using a variety of custom data sets ranging from classifying rotating digits to simulations of helicopters landing on air-capable ships. Experiments are ongoing to improve and evaluate the system’s performance in higher fidelity scenarios with an eventual integration with machine learning software used to track aircraft and support equipment.

Safety Framework for Deploying Learning Systems

Commanders and policymakers often require human-machine teaming on subjective decisions, often in a specific setting, a region or a specific population, and may call on engineers to rapidly redeploy a system to another setting. Subjective tests often lack ground truth, or even a reference standard, to calculate performance metrics such as accuracy and precision. Extensions to multiclass problems are even more challenging. We present a biologically-inspired resampling method to reframe these problems as a treatment effect analysis a specific result from the original method validation which will support rapid revalidation in a new setting. Military use cases are proposed, and specific analyses from medicine are presented, along with a software implementation for reference.

Sample size estimation for multi-class, multi-rater human-machine teaming problems without ground truth, applications to military medicine and military affairs

Discussions on building a framework for the development of Robust and Safe AI systems with extensions to Naval and Defense  applications

DoD operations increasingly seek to use AI-enhanced control systems to make rapid decisions at mission time, for both autonomy and more general command and control operations. This is motivated by a need for both automated and human control systems that can make decisions significantly faster than the dynamics of the real-world environments in which they operate. Despite the advances offered by AI-enhanced systems in terms of reaction time and their ability to integrate large amounts of available data into their decisions, these systems are often mistrusted, as a consequence of their decisions being both opaque and prone to making errors that a human operator would not. Reliability in fast-paced environments requires that decisions can be quickly substantiated in real-time to prevent such errors and build trust in these systems. We are developing methods for using AI-enhanced control systems as intelligent decision tools, helping operators better understand the context of any AI-derived recommendation and empowering operators to fully explore the supporting evidence for that recommendation. To accomplish this, we have extended a methodology for adversarial explanations to state-of-the-art reinforcement learning frameworks, including MuZero. Adversarial explanations are a technique for generating counterfactuals that call attention to salient portions of data that contributed to a given decision. While related work focuses on highlighting salient data, such highlights say nothing about the relationships amongst that data that led to a decision. In contrast, adversarial explanations result in a “texture” of data that can be intuitively understood by an operator who is familiar with the kinds of data used to make a decision, even if they are not familiar with the patterns in the data that justify one decision versus another. In our work, we have shown how these textures can be used to diagnose either if an AI agent is missing an important detail in its decision process, or if an operator is not considering a vital detail. Additionally, we provide mechanisms for evaluating the criticality of any given decision – that is, evaluating when the consequences of a wrong decision are potentially catastrophic to mission success. This criticality is tied to the timeframe in which a good decision must be made to avoid catastrophe; that is, it helps operators decide if a decision is not only important, but also urgent, allowing important but non-urgent decisions to receive additional analysis and consideration. Together, these techniques allow for the more trustworthy exploitation of AI capabilities in rapid environments, giving better insights into recommended decisions and empowering human operators to override those decisions when necessary.

Making AI a Better Partner for Trusted C2 Applications

Data Analytics Needs Psychology

The Scalable Asymmetric Lifecycle Engagement (SCALE) Embedded System Security (ESS) Trusted Artificial Intelligence (TAI) program is a workforce development effort within the Trusted and Assured Microelectronics (T&AM) program. The goal of TAI is to provide a novel workforce development model that engages current DOD/DIB/GOV scientists and engineers with ongoing research projects at partner universities. This exposes DOD/DIB/GOV employees to state-of-the-art research and helps them direct it towards DOD/DIB/GOV-relevant problems. It also allows students at the undergraduate and graduate level to interact with DOD/DIB/GOV employees with the goal of fostering interest in DOD/DIB/GOV employment, increasing the likelihood of recruiting a ready and flexible workforce cleared for work for the DOD/DIB/GOV. This effort provides interesting and challenging continuing education to government employees while forging close relationships between principal investigators and their labs, increasing opportunities for future collaborations and recruitment.

Trusted AI and Naval Workforce Development

As AI systems take on critical roles in high-stakes decisions, the ability to explain their decisions becomes crucial for assigning appropriate trust. Limitations for explaining black-box systems make interpretable AI methods an appealing alternative. Case-based reasoning (CBR) solves new problems by retrieving solutions to similar prior problems and adapting their solutions to fit the new problem circumstances, and is “naturally interpretable”, in that decisions can be justified by presenting the cases from which they were derived. However, there has been little study of when the benefits of CBR apply and how specifics of case presentation affect explanatory effectiveness. To assess factors affecting the usefulness of cases to explain decision-making, we conducted a human subjects study (N=110) to investigate how alternative types of explanations of a CBR system’s conclusions impact user trust in the CBR system. Our goal was to inform future design of case-based decision support systems by advancing understanding of how different system types and explanations affect the factors of goodness, convincingness and trust in the system.

Our experiment tested how two different types of interactions with a CBR system—one-shot, or interactive “conversational case-based reasoning”—and four types of explanation presentation affected subjects’ assessment of explanations of decisions of a case-based classification system. In our tests, augmenting presented cases with explicitly highlighted similarities and differences was helpful, but augmenting cases with nearby counterfactual cases was not. Whether the system interacted in a conversational manner was not found to impact scores, suggesting that the case itself is more important to users than how it was retrieved. To assess effects when erroneous conclusions were explained, we provided the user with an intentionally incorrect conclusion, without the user being made aware that the conclusion was incorrect, and found impact on scores for goodness, convincingness and trust of case-based explanations. This suggested that case-based explanations were not masking system errors. We also studied the effects of similarity of the presented case to the new problem, and found that it impacted convincingness and trust, underlining the importance of sufficient past cases.

The study illustrates that even for interpretable methods, specifics of explanation presentation matter. Results can be used by system designers to improve the quality of the system explanations. Also, by underlining that the key factor in goodness, convincingness, and trust is the quality of the cases themselves, the study provides support for case presentation as a natural vehicle for explanation.

This work was funded by the US Department of Defense (Contract W52P1J2093009), and by the Department of the Navy, Office of Naval Research (Award N00014-19-1-2655).


Premium content

Downloads

Next from NAML 2024

AI dataset design recommendations to deal with unknowns

Similar lecture

Comparing Pre-service and In-service Predictive Factors that Lead to Attrition Among Active Duty Marine Recruits