United States

The focus of this research is to develop and assess reinforcement learning (RL) based control schemes for control of unmanned underwater vehicles (UUVs) in multi-agent dynamic docking scenarios. UUVs are essential in military, scientific, and commercial applications. Applications include intelligence, surveillance, reconnaissance, bathymetric surveying, inspection and identification, biological sensing, and more. Modern underwater vehicle guidance and control often relies upon predefined expert-designed plans or controllers. However, RL offers a more flexible rapid learning approach. RL agents learn from interactions and experiences within their environment, whether simulated or real, enhancing decision-making and optimizing actions. This process allows them to develop behaviors and strategies given operationally complex objectives. Recent successes of reinforcement learning has seen it applied in diverse fields-from board games to aerial combat. Which demonstrates its versatility and potential for enhanced decision making. Applying these proven RL strategies could revolutionize the performance of UUVs, enabling them to navigate and operate in complex underwater environments with increased precision and maneuverability. The Proximal Policy Optimization (PPO) algorithm is employed to train RL-based controllers for 6-DOF REMUS 100 UUV simulation models in two scenarios: dock-pursuit and dock-loitering. Dock-pursuit focuses on intercepting a moving platform while avoiding collisions and minimizing control effort, whereas dock-loitering prioritizes maintaining optimal field-of-view coverage and strategic positioning relative to the docking platform. The simulation integrates a collision detection system, an alpha-beta filter for state estimation and imperfect observations, environment/state randomness, and a field-of-view sensor. Results from numerous trainings and simulations for various different scenarios is presented including high- and low-level control. Finally, the work acknowledges limitations, including the absence of sim-to-real validation, and outlines potential avenues for future research. By demonstrating RL-based UUV control, this work aims to demonstrate RL&#39;s potential in advancing performance in operationally complex underwater domains.

NAML 2025

Multi-Agent Reinforcement Learning for Control of Unmanned Underwater Vehicles in Dynamic Docking Scenarios

guidance navigation and control

uuvs

reinforcement learning

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this workshop.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2024

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


The focus of this research is to develop and assess reinforcement learning (RL) based control schemes for control of unmanned underwater vehicles (UUVs) in multi-agent dynamic docking scenarios. UUVs are essential in military, scientific, and commercial applications. Applications include intelligence, surveillance, reconnaissance, bathymetric surveying, inspection and identification, biological sensing, and more. Modern underwater vehicle guidance and control often relies upon predefined expert-designed plans or controllers. However, RL offers a more flexible rapid learning approach. RL agents learn from interactions and experiences within their environment, whether simulated or real, enhancing decision-making and optimizing actions. This process allows them to develop behaviors and strategies given operationally complex objectives. Recent successes of reinforcement learning has seen it applied in diverse fields-from board games to aerial combat. Which demonstrates its versatility and potential for enhanced decision making. Applying these proven RL strategies could revolutionize the performance of UUVs, enabling them to navigate and operate in complex underwater environments with increased precision and maneuverability. The Proximal Policy Optimization (PPO) algorithm is employed to train RL-based controllers for 6-DOF REMUS 100 UUV simulation models in two scenarios: dock-pursuit and dock-loitering. Dock-pursuit focuses on intercepting a moving platform while avoiding collisions and minimizing control effort, whereas dock-loitering prioritizes maintaining optimal field-of-view coverage and strategic positioning relative to the docking platform. The simulation integrates a collision detection system, an alpha-beta filter for state estimation and imperfect observations, environment/state randomness, and a field-of-view sensor. Results from numerous trainings and simulations for various different scenarios is presented including high- and low-level control. Finally, the work acknowledges limitations, including the absence of sim-to-real validation, and outlines potential avenues for future research. By demonstrating RL-based UUV control, this work aims to demonstrate RL's potential in advancing performance in operationally complex underwater domains.

technical paper

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event.

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

In the past five years, advancements in Artificial Intelligence (AI) have significantly impacted Naval operations, enhancing real-time data processing, predictive analytics, and autonomous decision-making. The integration of AI at the edge—where data is processed locally—has become crucial for improving operational efficiency, reducing latency, and bolstering security. This is particularly vital in the context of the current geopolitical landscape, characterized by peer competition and emerging contingencies. This white paper outlines the necessity and strategic advantages of deploying and integrating AI at the edge today to address the evolving challenges of modern Naval operations.

Integrating AI at the Edge Today:  Transforming Naval Strategies and Operations

Bottlenose dolphins often conceal behavioral signs of illness until it reaches an advanced stage. Motivated by the efficacy of vocal biomarkers in human health diagnostics, we utilized supervised machine learning methods to assess various model architectures' effectiveness in classifying dolphin health status from the acoustic features of their whistles. A gradient boosting classifier achieved an 72.3% accuracy in distinguishing between normal and abnormal health states—a significant improvement over chance (permutation test; 1000 iterations, p < 0.001). The model was trained on 30,693 whistles from 15 dolphins and the test set (15%), totalled 3612 ‘normal’ and 1775 ‘abnormal’ whistles. The classifier identified the health status of the dolphin from the whistles features with 72.3% accuracy, 73.2% recall, 56.1% precision, and an 63.5% F1 score. These findings suggest the encoding of internal health information within dolphin whistle features, with indications that the severity of illness correlates with classification accuracy, notably in its success for identifying 'critical' cases (94.2%). The successful development of this diagnostic tool holds promise for furnishing a passive, non-invasive, and cost-effective means for early disease detection in the Navy’s bottlenose dolphins.

Navy Dolphin Health Classification from Whistle Features

We will present our cutting-edge research on securing AI models against adversarial threats, focusing on trustworthy AI in defense and operational environments. Our work explores techniques to detect, assess, and remediate AI security vulnerabilities, even in low-compute and low-connectivity environments critical to naval and military applications. We will also present our approach to evaluating AI trustworthiness, safeguarding models from adversarial attacks, and enhancing resilience in real-world deployments. Attendees will gain insights into our methodologies, along with a preview of our prototype tools designed to improve AI security in constrained operational settings. This session will highlight how our research enables more secure AI adoption in mission-critical scenarios, empowering decision-makers to deploy AI confidently in high-stakes environments.

Mission-Critical AI: Automating Trust, Security, and Performance at Scale

Machine learning has become increasingly useful for predicting quantities that previously required computationally expensive physics-based models. For example, optimizing the route of a ship to minimize fuel burned would typically require a digital twin or a physics-based ship model, which are both computationally expensive and require information about the ship that is not readily available. This study investigates using machine learning to predict calm water resistance and added resistance in waves using limited information about the sea state and ship as part of an approach known as Machine Learning Surface ship Resistance estimation Framework (ML-SURF). This portion of the study describes the exploratory data analysis (EDA), model training and tuning, and error analysis that has been performed on a training dataset consisting of approximately 2,000 hull forms. The Carderock Analytics and Machine Learning Environment (CAMLE) was used to automate EDA and optimize model training. We performed analysis of the training dataset, evaluated multiple machine learning models, including neural networks and tree-based models and performed analysis on the models to improve model accuracy. The resulting model is highly accurate and runs much faster than traditional physics-based models. We will review each step in this process, discuss lessons learned and potential future work.

Ship Resistance Estimation Framework with Machine Learning: Data Analysis and Model Training 

Agents for Navy Applications – A Practical Guide
Mark Tabladillo, Ph.D.
Cloud Solution Architect, Microsoft
Support Technical Advisor, Flank Speed
Abstract
This presentation explores the application of artificial intelligence (AI) agents in enhancing naval operations through the integration of Autogen and Azure AI Foundry. With the increasing complexity of naval missions and the rising threat of cyber-attacks, these tools provide a scalable and efficient solution for real-time decision-making and operational resilience.
The session highlights how hierarchical reinforcement learning (HRL) frameworks and modular agent architectures can be used to address the multi-level challenges of naval coordination. Autogen’s capabilities for creating and managing AI agents are combined with Azure AI Foundry’s robust infrastructure to demonstrate practical applications in ship coordination and cyber-attack mitigation.
A live demo showcases a simulated naval scenario where a fleet must navigate through a high-threat zone while responding to cyber-attacks. The demonstration illustrates how individual ship agents implement defensive measures, a command agent oversees fleet coordination, and large language models (LLMs) analyze intercepted communications for situational insights. These interactions highlight the agility and adaptability of AI systems in dynamic operational contexts.
The presentation concludes with a discussion on the scalability and future potential of Autogen and Azure AI Foundry in multi-domain naval operations, emphasizing their role in maintaining decision-making superiority and mission continuity in complex environments.
Citations:
Black, S. E. (2024). Mastering the digital art of war: Developing intelligent combat simulation agents for wargaming using hierarchical reinforcement learning (Doctoral dissertation). Naval Postgraduate School. Retrieved from https://hdl.handle.net/10945/73072
Black, S., & Darken, C. (2023). Scaling intelligent agents in combat simulations for wargaming. Paper presented at the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando, Florida.
Black, S., & Darken, C. (2024). Scaling artificial intelligence for digital wargaming in support of decision-making. Naval Postgraduate School. Retrieved from arXiv:2402.06075
Chen, Y., & Chu, S. (2024). Large language models in wargaming: Methodology, application, and robustness. Proceedings of the CVPR Workshop 2024.
Weller, D., Meltschack, M., & Schwindling, D. (2024). Leveraging large language models for enhanced wargaming in multi-domain operations

Agents for Navy Applications – A Practical Guide

TorchSig is a radio frequency (RF) machine learning (ML) software library that performs dataset generation and model training for signal detection and modulation recognition. This tutorial will be in two parts, beginning with a TorchSig overview followed by live demonstrations of the software. The TorchSig overview will include a brief history, future plans and then cover details such as datasets, signal generation, transforms and models. The overview leads into the demonstration portion which will walk through a series of publicly available Jupyter notebooks. The tutorial will show how to generate both wideband and narrowband datasets, how to train YOLO models with TorchSig data and how to use the pre-trained models.

TorchSig: Open-source Software for RF Machine Learning

MAGTF (Marine Air Ground Task Force) is a mission where different Program of Records collaborate. The MCTSSA/MCSC Test & Evaluation team creates test plans from the Requirement PDF documents. NPS-CS team used Computer Science, AI/ML concepts to explore automation and analysis to assist in the human intensive process.

System Requirements Clustering with Machine Learning and Architecture Design

Our presentation will describe and demonstrate the OLIVE platform and its machine learning capabilities. OLIVE is a Government off the Shelf (GOTS)/Unlimited Government Rights multi-media processing system in wide use in the DoD and Intel Community. OLIVE is a multi-media software development and deployment platform designed to be 1) readily deployable and transition technology from basic research to software customers can use, 2) flexible and easy to integrate into a wide range of customer systems, 3) modular, with the ability to update core components separately from the base system/platform, 4) efficient, facilitate the reuse of capabilities with a shared architecture for audio, text, imagery and video, 5) inexpensive, such that new innovations can be built off the existing base easy as possible so that the technology we’ve already developed is free and remains USG Unlimited Rights, 6) supported by a team of experienced researchers, software engineers and quality assurance personnel to provide training, documentation, maintenance and facilitate implementation and testing of new features. 
OLIVE is a transition platform, and as technology matures in our core research programs, we port the successful and robust components as “plugins” within the OLIVE framework to make them readily accessible to new and existing USG end-users and mission sites. Plugins include approximately thirty different capabilities, including voice biometrics (speaker recognition), language recognition, translation, transcription, face detection and recognition, topic detection and others. We have integrated technology from over twenty distinct projects from DARPA, IARPA, SOCOM, FBI, IC, DoD, etc. OLIVE runs natively on Windows, Android and Linux operating systems.
OLIVE is a software server using TCP/IP-based API with various client interfaces. OLIVE has a high-level API for both Java and Python and a lower-level API using Google ProtoBuffers and ZMQ that are compatible with almost all modern programming languages. The OLIVE framework is composed of several key parts: 1) Idento, SRI’s Core Deep Learning/AI and signal processing library, 2) the core OLIVE Server, which connects plug-ins with end user systems through an API. The enterprise API exposes capabilities to end-user interfaces and the core services API enable developers to integrate new capabilities into OLIVE., 3) Plugins, the independent modules that define recipes for specific capabilities (e.g. language ID). Each plugin has domains, that target a specific set of conditions or language. Each end user may use different plugins and there are often mission-specific plugins or domains. 4) GUIs, OLIVE has four GUIs developed by SRI and is integrated into many other end user GUIs. The OLIVE GUIs may be used directly by customers, though many use the OLIVE APIs directly to expose plug-ins to end users.
OLIVE is powered by multiple artificial intelligence models capable of running off-line in USG systems. Deployments have ranged from enterprise cloud scenarios for the IC to Android deployments for special operations. We will provide details on our approaches to deploying high-end, and high-accuracy AI capabilities in reduced resource compute environments and customizing models and algorithms to the military environment with high-noise acoustics, uncooperative targets and specialized language use.

OLIVE: a government-rights multi-media processing platform for the deployment of Artificial Intelligence

How organizations can manage the transformation driven by technological change.

Innovation and Organizational Transformation

The DoD Digital OnRamp Platform is a cutting-edge solution, designed to connect innovative private sector organizations to DOD opportunities, while also making private sector capabilities discoverable by Department of Defense (DOD) users.  The Platform will bridge the knowledge gap and provide a demand signal from the DOD to commercial/academic technology communities.  Leveraging the latest advancements in AI technologies (AI/ML/LLM/etc.), this platform aims to simplify and enhance the process of matching private sector capabilities with DoD opportunities, and vice versa.
 
The DOD Digital OnRamp Platform will provide an AI-enabled, secure, scalable, and efficient environment to facilitate:
•	Private Sector Matching to DOD Opportunities: Utilizing multiple, publicly-available data sources such as FPDS.gov, SBIR.gov, etc., the platform will provide users with: 1) matches  to relevant opportunities and technologies, 2) links to online training to help the user respond to the specific matched opportunities and 3) links to physical government offices within 100 miles of their address that can provide additional assistance, such as a Naval Tech Bridge, Air Force Spark Cell, DOD Innovation OnRamp Hub, APEX Accelerator, Small Business Administration office, etc.  
•	DOD Mission-Related Technology Queries:  Utilizing DOD and commercial datasets, the platform will provide DOD users with responses to their queries regarding technology, products and capabilities available from commercial companies and academia.  
•	Optimized Engagement Pathways: The platform will identify pathways to engage further for DOD users, i.e., the availability on certain DOD contracts or commercial options, as well as a collaboration layer to communicate with private sector organizations which have an account on the platform. 


Premium content

Downloads

Next from NAML 2025

Integrating AI at the Edge Today: Transforming Naval Strategies and Operations

Similar lecture

A Novel Reinforcement Learning Framework for Improved Guidance and Power Management of Unmanned Underwater Vehicles