United States

he SeaVision maritime awareness tool provides an unclassified common operating picture for the U.S. Navy and many partner nations. It was developed at Naval Information Warfare Center (NIWC) Pacific and the U.S. Department of Transportation, Volpe Center. SeaVision has a complex interface, and training materials span many hundreds of pages. This makes learning to use SeaVision a difficult task, especially for our foreign partners. 

We used open-source LLMs to develop a chatbot that can answer questions about how to use the SeaVision tool. With this chatbot, new users can ask a question and quickly learn how to use any of SeaVision&#39;s many features. We provide the LLM context by inserting text excerpts from training documents and transcripts of training videos. Our system&#39;s architecture combines both a retrieval augmented generation (RAG) system and a custom memory module. Our RAG system uses a FAISS vector database to store the training files as embedded passages; cosine similarity was used as our metric for retrieval. The custom memory module keeps track of past questions, answers and contexts. When memory is limited, the module favors preserving the newest question and answer pairs while discarding the oldest context available. The memory module also acts as a context gatekeeper by stopping duplicate contexts from being fed into the prompt. 

Although question answering is important to being able to operate SeaVision, it is only a small part of the larger picture. We plan to expand the functionality of this chatbot to perform atomic tasks carried out by maritime domain analysts such as getting the current locations of ships, the paths ships have followed through time, port call histories, inspection histories, warnings about potentially unsafe or insecure shipping, and much more with the goal of combining multiple atomic tasks into single common tasks. We plan to demonstrate how this type of tool can automate tasks and greatly improve Warfighter performance. Many other U.S. Navy systems have similarly complicated interfaces, and chatbots could improve Warfighter performance for these systems as well.

NAML 2024

Training SeaVision Operators using Large Languages Models

retrieval augmentation generation

open-source models

maritime domain awareness

large language models

chatbots

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

NAML 2025

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

he SeaVision maritime awareness tool provides an unclassified common operating picture for the U.S. Navy and many partner nations. It was developed at Naval Information Warfare Center (NIWC) Pacific and the U.S. Department of Transportation, Volpe Center. SeaVision has a complex interface, and training materials span many hundreds of pages. This makes learning to use SeaVision a difficult task, especially for our foreign partners. 

We used open-source LLMs to develop a chatbot that can answer questions about how to use the SeaVision tool. With this chatbot, new users can ask a question and quickly learn how to use any of SeaVision's many features. We provide the LLM context by inserting text excerpts from training documents and transcripts of training videos. Our system's architecture combines both a retrieval augmented generation (RAG) system and a custom memory module. Our RAG system uses a FAISS vector database to store the training files as embedded passages; cosine similarity was used as our metric for retrieval. The custom memory module keeps track of past questions, answers and contexts. When memory is limited, the module favors preserving the newest question and answer pairs while discarding the oldest context available. The memory module also acts as a context gatekeeper by stopping duplicate contexts from being fed into the prompt. 

Although question answering is important to being able to operate SeaVision, it is only a small part of the larger picture. We plan to expand the functionality of this chatbot to perform atomic tasks carried out by maritime domain analysts such as getting the current locations of ships, the paths ships have followed through time, port call histories, inspection histories, warnings about potentially unsafe or insecure shipping, and much more with the goal of combining multiple atomic tasks into single common tasks. We plan to demonstrate how this type of tool can automate tasks and greatly improve Warfighter performance. Many other U.S. Navy systems have similarly complicated interfaces, and chatbots could improve Warfighter performance for these systems as well.

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

Please log in to explore this workshop.

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Typically, effective object detection in images requires thousands or tens of thousands of labeled instances. Crowd-sourced data has been used to train object detectors for commercial purposes, but the DoD cannot use this same approach to train neural networks with supervised learning. Either there is an insufficient number of instances of DOD-relevant data, or curating and labeling that many images is too cost and time prohibitive to be of use. ​

For this reason, few-shot object detection can be used to limit training data to tens or hundreds of instances rather than thousands upon thousands. Specifically, our experiments adapt few-shot object detection to the Naval domain. We set up two experiments where in the first experiment, we use a large, common dataset to train a base network followed by training with a small sub-group of novel classes of data; and, in the second experiment, we test another few-shot network, DE-Vit, which can use few-shot methods to find new objects in an image. ​

There are many methods that can be used to train neural networks with fewer data. These experiments are conducted as a part of the SAVANT project—which uses an ensemble of semi-supervised methods to build upon datasets. It will serve as a test bed for semi-supervised methods while creating some much-needed labeled Naval data.

Adapting Few-Shot Learning To The Naval Domain

Naval aviation assets on aircraft carriers such as aircraft, catapults and arresting gear are tracked through information systems that provide situational awareness to sailors. The current concept of operations requires highly skilled workstation operators to manage everything from aircraft status to flight schedules to keep information up to date. Operators receive information over radio communication channels from flight deck personnel. This process requires the operator to be dedicated to listening to updates and manually entering what they hear into workstations. This project developed an automated speech recognition prototype that can automatically listen into the audio communication pipeline and add information into computer systems.  
A speech recognition pipeline was created to recognize the standard phrases spoken by aircraft carrier flight deck personnel. Outputs can be used to correctly identify the actions that need to be performed to update computer systems automatically. The pipeline includes speech to text Automatic Speech Recognition (ASR) models and speech enhancement followed by post processing and similarity matching techniques to increase accuracy. Recursion was implemented to re-run the pipeline with a larger speech model if the result does not match a standard flight deck phrase. The team made use of Whisper, a transformer-based model with various model sizes ranging from 39 million to 1.56 billion trained parameters. The pipeline was tested in a reverberation chamber using flight deck phrases under various noise levels and model sizes. 
The team is currently working on fine-tuning the model and creating synthetic data. Data generation used by the team include the ability to fabricate semi-realistic speech in 108 varying voices and dialects. The team has tested fine-tuning efforts on a hold-out percentage of the dataset but has plans to continue fine-tuning the whisper model on both real-world and synthetic data to note results on increased model performance. While high-decibel noisy recordings were obtained from a laboratory, the researchers hope to obtain more real-world data and ultimately test the system either in a simulated environment or the flight deck environment. 


Aircraft Carrier Flight Deck Information Updates using Speech Recognition 

we discuss the development and deployment of a robust autonomous system capable of performing various tasks in the maritime domain under unknown dynamic conditions. We investigate a data-driven approach based on modular design for ease of transfer of autonomy across different maritime surface vessel platforms. The data-driven approach alleviates issues related to a priori identification of mechanis- tic system models that may become deficient under evolving system behaviors and/or shifting, unanticipated, environmental influences. Our proposed learning-based platform comprises a deep Koopman system model and a change point detector that provides guidance on domain shifts prompting relearning under severe exogenous/endogenous perturbations. Motion control of the autonomous system is achieved via an optimal controller design. The Koopman linearized model naturally lends itself to a linear–quadratic regulator (LQR) control design. We propose the C3DKL control architecture “Cascade Control with Change Point Detection and Deep Koopman Learning”. The framework is verified in both simulation and real experiments.

C3DKL: Cascade Control with Change Point Detection and Deep Koopman Learning for Time-varying Systems

Federated learning (FL) is a distributed machine learning paradigm that enables clients to collaborate without accessing, infringing upon, or leaking user raw data by sharing only model parameters. Multimodal automatic target recognition (ATR) in FL aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each client will be diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality models to the server. In this study, we propose multimodal fusion Federated learning with joint Modality and Client selection (mmFedMC), a new FL framework for multimodal ATR that can tackle the above-mentioned challenges. The joint selection algorithm incorporates two main components: (a) A modality selection criterion for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality model size as a gauge of communication overhead, against (iii) the frequency of modality model updates, denoted recency, to enhance generalizability. This allows mmFedMC to flexibly balance performance against communication costs, depending on resource constraints and application requirements. (b) A client selection for the server is performed on the basis of the cross-entropy of prediction as a metric. This involves selecting a subset of clients for uploading and aggregating models, which further reduces communication overhead and optimizes the overall process. Experiments on a real-world dataset with six modalities demonstrate that mmFedMC achieves 98.19\% accuracy with a communication overhead of only 5MB per client. This represents an approximate 10\% increase in accuracy compared to random modality or client selection. Moreover, in comparison to other state-of-the-art methods that do not utilize a selection mechanism, mmFedMC enhances accuracy by a factor of 2x to 4x. Alternatively, to achieve comparable levels of accuracy, other methods require approximately 20x more communication overhead, i.e., about 100MB per client. Beyond the trade-off between performance and communication overhead, mmFedMC also offers an interpretable representation of the modality impact during the FL process. In the initial stages of FL, most modalities exhibit similar impacts. However, as the communication rounds progress, modalities with larger feature sets and complex models play a subordinate role in modality selection due to their higher communication costs. With the advancement of FL, more straightforward modalities that continue to convey substantial information emerge as primary contributors.

Communication-Efficient Federated Learning for Multimodal Automatic Target Recognition

Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

The objective of this line of research is to use Convolutional Neural
Networks (CNNs) to automatically identify corrosion in photographs.
This approach provides a reliable alternative to traditional
approaches. Corrosion, a widespread issue in several sectors,
necessitates prompt detection to avoid structural harm and guarantee
operational security. The use of deep learning, particularly
Convolutional Neural Networks (CNNs), offers an advanced method for
detecting corrosion. This is achieved by using the neural networks'
capability to identify complex patterns in picture data. This research
investigates the creation and assessment of a Convolutional Neural
Network (CNN) model that was trained using a dataset consisting of
photos depicting both corrosion and non-corrosion. The model's
effectiveness is assessed by analyzing key performance indicators such
as True Positive Rate, False Positive Rate, accuracy, recall, and area
under the ROC curve. The study illuminates the capabilities and
constraints of the CNN-based technique, providing useful insights for
businesses in need of effective and precise corrosion detection
solutions.


Corrosion Detection Using Deep Learning

We consider the problem faced by an intelligent attacker that wants to discover new attack paths to compromise a network in a cyber-security network intrusion scenario. This problem is challenging because the number of possible actions for the attacker could increase exponentially with the number of nodes in the network and the number of services running on those nodes. We formulate the attacker's problem as a reinforcement learning task and use a decision transformer network to tackle the complexity of the attacker’s action space. We have evaluated our techniques within a gamified network intrusion detection scenario called Cyber AI Gym Environment (CAGE). Our results show that the proposed approach can enable the attacker to learn successful attack strategies against two difficulty levels of defender strategies.

Decision Transformer for Learning Intelligent Attacker Strategies in a Network Intrusion Detection Game

The diverse range of naval vessels, ranging from aircraft carriers to unmanned surface vessels, and infrastructure, such as smart naval mines or buoys, presents a challenge for traditional machine learning (ML) techniques in naval defense. Traditional ML, which centralizes network data, struggles with the vast distances and unstable or insecure communication links typical in naval defense scenarios. 

Federated learning (FL) offers a solution by keeping the data and ML model training local to the naval vessels and only performing occasional ML model combinations, or "global aggregations" at a central server, or "mothership" for naval applications. FL does address some of the heterogeneity inherent to the naval setting, for example, it minimizes the communication burden during deployments through sporadic global ML model aggregations. "Statistical heterogeneity" here refers to data-related differences among network elements, such as various dataset sizes. "Systems heterogeneity" relates to physical differences between naval objects, such as varying reliability of communication links. 

However, existing literature in FL typically assumes the presence of fully labeled datasets across all naval elements, an unrealistic scenario in practical naval deployments where a diverse mixture of both \textbf{labeled} and \textbf{unlabeled} data is common. Current FL-based methodologies tend to produce a uniform global ML model for all naval elements, neglecting the information contained with the unlabeled data, potentially leading to suboptimal global ML models for diverse naval resources. 

To address this, we introduce a decentralized federated domain adaptation methodology which leverages both labeled and unlabeled data to identify the underlying statistical divergences among naval vessels. In doing so, we are able to optimize the transfer of ML models from naval elements with high quality labeled data (called sources) to naval resources with low quality or unlabeled data (called targets), thereby personalizing ML models to better fit the real-time needs of different naval vessels in a deployment. Our methodology, called Source-Target Determination and Link Formation (ST-LF), optimizes both (i) classification of naval resources into sources and targets and (ii) source-target link formation, in a manner that considers the trade-off between ML model accuracy and communication energy expenditures. To obtain a concrete objective function, we derive a measurable generalization error bound that accounts for estimates of source-target hypothesis deviations and divergences between data distributions. The resulting optimization problem is a mixed-integer signomial program, a class of NP-hard problems, for which we develop an algorithm based on successive convex approximations to solve it tractably. Subsequent numerical evaluations of ST-LF demonstrate that it improves classification accuracy and energy efficiency over state-of-the-art baselines.

Envisioning Multi-Source to Multi-Target Federated Domain Adaptation for Naval Defense Scenarios

In many Navy-relevant applications, dedicated datasets for machine learning are noisy in nature, small in quantity, and challenging to obtain. This work uses applications of transfer learning to exploit small noisy datasets for machine learning tasks. Our focus is on electroencephalogram (EEG) neural datasets for human in the loop optimization of warfighter performance. Specifically, we apply transfer learning to classification tasks on small noisy datasets which leverage larger, publicly available laboratory setting EEG datasets on similar but not identical task domains. We find that transfer learning from a large publicly available motor imagery dataset improves performance and reduces the amount of data required for cognitive state classification of participants learning to walk with an exoskeleton dataset (data source: Army Research Lab). Further, we are exploring the impact of transfer learning on real-time EEG state classification for user interface (UI) development. Following a similar approach as above, we will first build a transfer-learning model by harnessing laboratory-collected EEG data during cognitive workload tasks, and then apply it beta-testers as they interact with UI prototypes. Together, these examples highlight the potential impact that transfer learning can have on Navy-relevant applications by taking advantage of data sources collected in privileged non-Naval environments.

Transfer learning for human in the loop optimization of warfighter performance

You don't need to understand how to fly a plane or design a plane to understand what aviation can do for war. The same thing with artificial intelligence.
This talk walks you through strategic alignment, how the Marine Corps is thinking about implementing AI across the Corps and in support of both Marine Corps operations in the broader naval mission. 

Marine Corps is very good at fighting. But if you look at how we fought for the last 50 years compared to where we could be in the future, we certainly can fight smart. The ability of MC to pull in data, to pull from different sources, from different venues and things, apply analytics, and help us to decision advantage across a range of what we do. Whether that is at the pointy end of the stick, whether that's recruiting, whether that's our internal talent management, whether that's logistics, whether that's supply system,the ability for us to leverage some of these tools is not near infinite, but we have an enormous amount to do and we want to overall be able to fight smart. But right now we're not and this is our path to how we're gonna get to that to take into account this the emerging technology where we think this not only where AI is today, but where we think AI is going over the next few years and then in the beyond into the 2030.

Premium content

Next from NAML 2024

Adapting Few-Shot Learning To The Naval Domain

Similar lecture

MUSES: Multimodal Sensemaking in SeaVision