United States

This project explores lessons learned in designing, prototyping, and evaluating AI agents for intelligence analysis for a cyber defense use case. Despite growing interest, there remains little practical guidance on how to design and build AI agents, particularly guidance grounded in real-world implementations rather than vendor marketing. This project investigates the unique challenges of workflow design and agent evaluation, especially for novel or evolving processes.

Agentic AI systems including AI agents leverage augmented Large Language Models (LLMs) to autonomously complete tasks, workflows, or make decisions. Augmented LLMs are LLMs equipped with tools, access to memory, and retrieval capabilities. Using the capabilities of an augmented LLM, agents can autonomously search for information by generating their own queries, can select and use tools to complete specific actions, and can store and recall information to build context and improve performance.

Overall, AI agents have the potential to increase speed and scale, which may mean freeing up humans to handle more challenging or creative tasks. However, AI agents also present unique challenges, in that they are non-deterministic systems, meaning they can produce stochastic, and in some cases, incorrect, results. Furthermore, AI agents can rely on many Machine Learning (ML) components, increasing the chance of cascading errors.

This work presents lessons learned about the opportunities and challenges related to building AI agents. This work centers around a case study in which the team designed, prototyped, and evaluated two AI agents, built from scratch in LangGraph, for a cyber operations use case. The team worked closely with a Subject Matter Expert (SME) to design and refine the agents.

The first agent defines Priority Intelligence Requirements (PIRs) based on inputs such as the ACARE-I threat actors, a terrain map, and commanders&#39; intent. The second agent then takes the same inputs, along with the newly created PIRs, to identify associated MITRE ATT&amp;CK Tactics, Techniques, and Data Sources that a cyber team will need to be aware of for their upcoming mission.

Through this work, the team developed a set of practical recommendations for teams looking to build AI agents for mission critical use cases. These recommendations range from guidance on incorporating traceability into agents during early design, to implementing agents modularly throughout development and experimentation. These lessons learned are intended to help guide future agentic AI projects in developing agents iteratively with stakeholders, designing agents in a manner that is useful and user-centered, and identifying where traditional software engineering and ML practices are still required. This work presents actionable guidance informed by operator experience to support effective adoption by government organizations such as the Department of War (DoW).

NAML 2026

Agentic AI Unpacked: Design and Engineering Lessons from a Mission-Focused Case Study

cyber operations

agentic ai

co-design

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

AAAI 2026

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

This project explores lessons learned in designing, prototyping, and evaluating AI agents for intelligence analysis for a cyber defense use case. Despite growing interest, there remains little practical guidance on how to design and build AI agents, particularly guidance grounded in real-world implementations rather than vendor marketing. This project investigates the unique challenges of workflow design and agent evaluation, especially for novel or evolving processes.

Agentic AI systems including AI agents leverage augmented Large Language Models (LLMs) to autonomously complete tasks, workflows, or make decisions. Augmented LLMs are LLMs equipped with tools, access to memory, and retrieval capabilities. Using the capabilities of an augmented LLM, agents can autonomously search for information by generating their own queries, can select and use tools to complete specific actions, and can store and recall information to build context and improve performance.

Overall, AI agents have the potential to increase speed and scale, which may mean freeing up humans to handle more challenging or creative tasks. However, AI agents also present unique challenges, in that they are non-deterministic systems, meaning they can produce stochastic, and in some cases, incorrect, results. Furthermore, AI agents can rely on many Machine Learning (ML) components, increasing the chance of cascading errors.

This work presents lessons learned about the opportunities and challenges related to building AI agents. This work centers around a case study in which the team designed, prototyped, and evaluated two AI agents, built from scratch in LangGraph, for a cyber operations use case. The team worked closely with a Subject Matter Expert (SME) to design and refine the agents.

The first agent defines Priority Intelligence Requirements (PIRs) based on inputs such as the ACARE-I threat actors, a terrain map, and commanders' intent. The second agent then takes the same inputs, along with the newly created PIRs, to identify associated MITRE ATT&CK Tactics, Techniques, and Data Sources that a cyber team will need to be aware of for their upcoming mission.

Through this work, the team developed a set of practical recommendations for teams looking to build AI agents for mission critical use cases. These recommendations range from guidance on incorporating traceability into agents during early design, to implementing agents modularly throughout development and experimentation. These lessons learned are intended to help guide future agentic AI projects in developing agents iteratively with stakeholders, designing agents in a manner that is useful and user-centered, and identifying where traditional software engineering and ML practices are still required. This work presents actionable guidance informed by operator experience to support effective adoption by government organizations such as the Department of War (DoW).

poster

### Welcome to the 10th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.
#### On-Demand Access

* This is an on-demand site and all the recorded content from the workshop is available here. 
* Access to the recordings is exclusively available to registered participants.
* Each registrant has received an email notification, with instructions on how to access the content.

[![](https://assets.underline.io/markdown_image/1/image/60cd12aa376c191d3467ea2ee60944ea.png)](https://naml2026.org/)

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

This is an on-demand site. All recorded content from NAML 2026 is available here and access to it is exclusively available to registered participants.

**Please log in using the email address you registered with.** If you have any issues logging in, please contact us at <naml2026@underline.io>.

Let’s celebrate the past decade by investing in the next. 🚀

Please log in to explore this event.

This is an on-demand site. All recorded content from NAML 2026 is available here and access to it is exclusively available to registered participants.

**Explore the latest advancements in machine learning tailored to Naval applications 🔎**

With on-demand access to NAML 2026, you can watch all lectures & browse poster presentations at your convenience, while also engaging directly with presenters via the Underline platform - ask questions, exchange ideas, and stay connected.

**If you still haven’t registered, you can do so now.**
👇
**Take advantage of a special one-time offer: full access for just $99, with complimentary registration for students, government, and military employees.**

**Get access here: https://naml2026-ondemand.paperform.co/**

Please check your spam folder, or contact our support team at **<naml2026@underline.io>** if you registered, but still haven't received an access email.

Let’s celebrate the past decade by investing in the next. 🚀

Please register!

The 10th Annual Workshop on Naval Applications of Machine Learning (NAML 2026) was held in San Diego, California, March 2-5, 2026. NAML is a forum for interdisciplinary experts from government, industry, and academia to discuss results of ongoing research on theory and applications of machine learning to address some of the most challenging problems faced by our naval forces. 



Autonomous systems supporting naval operations must coordinate diverse subsystems, each with distinct sensing, communication, and control capabilities. In contested and uncertain environments, communication constraints make fully centralized control infeasible, while purely distributed approaches risk costly miscoordination when platforms operate with only local information. This talk presents two complementary perspectives on advancing fleet-level autonomy: the first focused on improving collaborative decision-making across large systems, and the second focused on managing the risks introduced when autonomous agents reason about others using imperfect models.
To the first goal, thanks to advances in networking and onboard computing, decision-making can not only be distributed to individual devices and vehicles, but it can also be performed collaboratively by sharing local information or coordinating the execution of joint control policies. I will discuss recent results on coalitional optimization and learning, which enable control architectures and resulting equilibria that exist between the fully centralized and distributed extremes. Specifically, I will present a game-theoretic model for distributed decision-making and demonstrate the possible benefits and costs of increasing communication in terms of gain/loss to equilibrium efficiency and solution complexity.
For the second goal, when synthesizing control policies for use in contested environments with multiple, possibly non-friendly, agents, one must design around a model of other agents' behavior. Whether this model is an optimizing agent with conjectured objectives and constraints or a policy trained on historical observations of the agent’s behavior, there exists the possibility of misspecification between this model and the agent’s day-of behavior. I will present recent findings on the possible consequences of these discrepancies, termed the strategic sim-to-real gap (drawing parallels from the physics-based sim-to-real gap in reinforcement learning). I will illustrate performance-based consequences via parametric analysis of heterogeneous multi-agent solution concepts, as well as dynamic consequences via stability analysis of receding horizon games with misspecification.
Together, these two thrusts advance the foundations of autonomy for large-scale fleets operating in complex naval missions. They highlight both the opportunities unlocked by increased collaboration and the risks that arise when autonomous systems reason imperfectly about others. The results point toward decision-making architectures that are robust, adaptive, and scalable across the breadth of naval operations.

Advancing Fleet-Level Autonomy Through Collaborative and Competitive Control

The Villain BOT and the Large Fraud are bundled microservices that are deployed in the Noblis Edge Environment on a Docker Container. The capability has two different components: an adversarial or "Abliterated" Chat BOT that has had its safety features removed to enable analysts to explore cyber/fraud scenarios from a criminal's perspective, and a fraud detection model with a novel LLM-driven Text-to-SQL interface that allows non-technical analysts to ask plain language questions. The villain BOT address a security use case and highlights a microservice can provide automation to complex task to save time and resources and meet the goals enabling AI-Ready Data for Decision Superiority.

The Large Fraud Model uses raw log transaction data and transformed ML-scored feature data, vectorizing it for LLM consumption and allowing each record to be evaluated against known fraud signatures. 

This is presented to analysts to explore via a Text-to-SQL interface that greatly increases analytical insight without having to engage expensive data scientists, enabling a much more rapid query-building experience and saving time for the analyst.

The BOT uses a local LLM that is isolated from the public internet to allow analysts to explore fraud schemes, data explorations, insider threat scenarios, and vulnerabilities. The ability of the bot to answer questions that are not allowed by standard LLM safety protocols opens opportunities for research, product workflow artifacts, and technical analysis, making analysts much more efficient and insightful. 

The ability of analyst to move back and forth between raw data and plain text discussions around fraud schemes allows for analyst to analyze the data, identify the threat and then compose detection countermeasure scripting within the same application is leap forward in insight, efficiency and ability to inject new fraud detection capabilities. 

This effort is part of the Noblis Sponsored Research Program for a large civilian agency customer and has broad application to other agencies' cyber programs and analyst-based environments.


Villain BOT / Large Fraud Model

This presentation will introduce a new paradigm for testing and evaluation of AI-enabled autonomous systems. In a world where AI is everywhere and is changing how research is conducted, collapsing expected timescales for innovation, and increasing efficiencies across domains how can AI help you? Industry-developed products are being rolled out as an unprecedented pace with shiny new capabilities or improved performance on specific tasks. However, translation of these performance gains to your missions is not guaranteed and introduces distinct tradeoffs. An inability to rapidly evaluate thesetradeoffs can delay delivery timelines and compromise our intelligence posture by chasing, rather than driving, state-of-the-art AI innovation. High-consequence missions present different constraints that influence Testing, Resources, Assurance, Data, and Expressivity (TRADE) considerations for development of viable solutions. Developing the technical capability, and associated testbeds, to evaluate models against mission TRADE requirements will position the US to assess and integrate AI-enabled autonomous systems at mission timescales.

Evaluating TRADE-Constrained AI for Development of Deployable Autonomous Systems

Reliable vehicle tracking from remote sensing data often requires robust state estimation 
across widely varying temporal resolutions. In this work, we propose a physics-informed 
attentive neural process (PI-AttNP) for multi-scale vehicle tracking that explicitly integrates 
simplified kinematic models as structural priors within a neural process framework. This 
design enforces adherence to physical motion laws while reducing error and accelerating 
convergence. The model leverages time-aware attention weights to capture statistical 
dependencies across irregularly spaced observations and employs split conformal 
prediction to provide distribution-free uncertainty guarantees. We evaluate the PI-AttNP on 
two distinct remote sensing applications—aircraft tracking using ADS-B data and ship 
trajectory prediction using AIS data—spanning sampling intervals from one minute to one 
day. Results demonstrate that PI-AttNP achieves competitive or superior performance 
compared to state-of-the-art baselines while offering principled confidence measures. 
These findings highlight the potential of physics-informed neural processes as a general 
and reliable tool for uncertainty-aware vehicle tracking in remote sensing.


State estimation using physics informed attentive neural processes for target tracking

This paper presents a novel approach aimed at enhancing Machine Learning (ML) algorithms for object recognition. Improvements in ML model performance are often attributed to both the quality of data and tuning of the model. However, determining which factor in object appearance is responsible for the improvement can be challenging. In contrast to a large number of works that are devoted to model improvements, we are focusing on properties in data which could help to isolate the impact of intrinsic object characteristics (such as shape) on the performance of ML models. The project employs three sources of data: collected data from aerial platforms, synthetic images generated by rendering CAD models, and lab-acquired images of 3D printed scale models from exactly the same CAD models. The project involves collecting various factors, such as pose, articulation, light direction, background color and shape, scale, brightness, contrast, color, pattern, shape/geometry, and reflectivity/propagation/diffusion (physics). The goal is not to evaluate the model's absolute performance, but to analyze changes in performance for general ML models or ways to improve the effectiveness of the dataset for training. By training a large ensemble of pre-trained neural networks with diverse multi-source datasets and testing on real-world collected images, we analyze performance changes and identify strategies to improve dataset effectiveness for training.
An initial large ensemble of neural networks trained on target images (>200,000 chips) with various multi-source datasets and tested on collected real objects (>50,000 tests). Our results demonstrate that even with limited real-world imagery, multi-source datasets significantly improve object recognition performance in untuned ML models. This approach, which enables adjusting recognition to object shape rather than overall appearance, highlights the critical role of diverse data sources in enhancing ML object recognition capabilities.

Machine Learning Enhancement Using Data Sets Variety

Real world problems encounter distribution drift, requiring models to be adaptable through time. However, many existing architectures remain black-box in nature, limiting trust and usability. In this paper, we address continuous distribution drift in both classification and regression settings by enforcing a Gaussian Process over the weights of a neural network across time. The architecture is optimized with second order optimization methods, employing multiple Hessian approximations. These approximations serve as precision matrices for multivariate normal distributions over layer weights, enabling sampling based-confidence intervals for each layer. Furthermore, we provide a basis function interpolation framework that models weight trajectories between temporal knots, substantially reducing computational cost while maintaining comparable predictive performance. Our framework applies to arbitrary neural network architectures, and we demonstrate its effectiveness on benchmark datasets, achieving performance competitive with state of the art methods.

A Bayesian Approach to Drift-Aware Parameter Modeling

Power system faults leading to outages and equipment failure are detrimental to ships and naval bases. While there are data recorders being installed in several key areas, these require an engineer to collect the raw data, process it, and analyze it to determine the cause of the fault. A Large Language Model (LLM) will be trained on MIL-STD-1399 (Interface Standard for Shipboard Systems) specifications. This standard defines the power quality requirements for ship power systems at different voltage levels. The LLM will utilize tools to interact with processed data and provides an explanation of the power system faults based on the MIL-STD-1399 requirement being violated. This would allow end users who do not have a background in data processing to be able to locally analyze all the data being recorded, saving months of lead time for an engineer to just get the raw data. With the nature of where these data recorders are being installed, the LLM will need to operate on the edge without internet access. Therefore, the LLM needs to be lightweight enough to run on a small form factor computing device like the NVIDIA Jetson Thor. The Thor device will be connected to the storage used by the data collection system and host a pipeline for data processing. The LLM will then use tools to query a database of processed data to check for MIL-STD 1399 violations and generate plots of the violations if needed. This system will allow for significantly faster data analysis of power system failures that can be accessed anywhere electronics are permitted to operate, and more importantly, in an isolated network environment.

Artificial Intelligence for Data Acquisition System

Abstract: Optimizing communications resources in contested environments imposes a high cognitive load on tactical operators and planners, who must simultaneously consider physical connectivity (Carrier-to-Noise Ratio), operational stealth (Low Probability of Detection), doctrinal compliance (EMCON), and emergent situations. While Language Models (LMs) offer promising semantic inference capabilities for decision support, their stochastic nature often leads to "Constraint Collapse"—the failure to adhere to rigid physical laws and safety protocols in high-stakes single-step decision spaces. We propose a decision support system that considers high-fidelity physics-based simulations, EMCON, commander’s intent, and ad-hoc situations prompted by the user.

We developed a high-fidelity simulation modeling ground vehicles traversing mountainous terrain, generating a dataset of 13 discrete antenna pattern and power configurations. We evaluated these options using a suite of LMs ranging from 8B parameters to GPT-4o, benchmarking them against a deterministic rule-based engine and human expert decisions. We emphasize the utility of small LMs, as contested environments often necessitate deployment on resource-constrained edge devices.

Our results indicate that while LMs face known challenges, their performance in tactical environments is highly context-dependent. Specifically, we find that off-the-shelf models struggle significantly with numerical reasoning and order bias, when compared to general benchmarks. However, we demonstrate that large LMs can achieve performance comparable to human experts, and crucially, that small LMs can match this performance when supported by a neuro-symbolic architecture that incorporates rule-guided reasoning and verifiers.

2B Models Outperform 72B: Neuro-Symbolic Decision Support for Tactical Comms

Advances in generative AI, large language models (LLMs), and multi-agent systems are reshaping the defense landscape, creating a new opportunity for the U.S. Navy to rapidly accelerate the development of scenario-learning technologies that can adapt to emerging situations, expose hidden biases, and support high-consequence decision-making. However, most naval AI programs face a bottleneck: the accreditation and deployment of full-scale, multimodal AI architectures at high classification levels (IL6 and above) often requires lengthy preparation, extensive data governance design, and strict infrastructure controls. This presentation proposes a complementary strategic approach—one that delivers meaningful capabilities early while paving the way for a full envisioned solution.

The method centers on rapid prototyping using Python and AutoGen, executed initially at IL4 for demonstration, yet engineered for immediate portability into other IL6+ environments. This rapid-prototyping approach is not intended to represent an entire AI solution architecture, but rather to embody one essential slice of it: the cognitive, interactive, human-AI collaboration layer that enables adaptive scenario learning, counterfactual analysis, and explainable decision support. This human-AI interface is where operators, analysts, and planners will most directly encounter the system—making it an ideal starting point for experimentation, evaluation, and early capability delivery.

References
Black, S. E. (2024). Mastering the digital art of war: Developing intelligent combat simulation agents for wargaming using hierarchical reinforcement learning. Naval Postgraduate School. 
Black, S. E., & Darken, C. (2023). Scaling intelligent agents in combat simulations for wargaming. Naval Postgraduate School. 
Chen, Y., & Chu, S. (2024). Large language models in wargaming: Methodology, application, and robustness. 
Han, W., Zhang, B., Wang, Q., Luo, J., Ran, W., & Xu, Y. (2019). A multi-agent based intelligent training system for unmanned surface vehicles. Applied Sciences, 9(6), 1089. 
Vasankari, L. (2023). Multi-agent reinforcement learning for littoral naval warfare. Aalto University. 
Weller, D., Meltschack, M., & Schwindling, D. (2024). Leveraging large language models for enhanced wargaming in multi-domain operations. NATO STO.


Rapid Prototyping for Naval AI Wargaming: Multi-Agent Scenario Learning and Counterfactual Reasoning with Python and AutoGen

Automatic Target Recognition (ATR) in the maritime domain presents unique challenges that significantly impact operational effectiveness of naval/intelligence systems. This presentation examines key technical obstacles encountered in maritime ATR and presents research findings on practical solutions for deployment in operational environments, enabled by modern MLOps infrastructure.

Given the drive toward low-cost, autonomous sensing platforms, maritime targets at distance provide minimal pixels for model inference, creating fundamental detection and classification challenges. Our analysis demonstrates how target pixel size directly correlates with recognition accuracy, with performance degradation becoming severe below critical thresholds. We present quantitative relationships between pixel density and ATR confidence scores across multiple target classes.

Contextual factors unique to maritime environments (i.e. sea state, atmospheric conditions, target pose, background clutter) significantly influence ATR performance in ways not sufficiently addressed by current algorithms and approaches. Our research quantifies these environmental impacts and offers approaches that leverage contextual awareness to improve recognition reliability.

Data augmentation strategies, while beneficial for training robustness, require careful consideration in maritime applications. We present comparative analysis of augmentation techniques specifically validated for maritime scenarios, identifying methods that improve generalization without introducing artifacts that degrade real world performance.

The Navy faces significant MLOps gaps that hinder transitioning research to operational systems. We demonstrate how the AMMO platform addresses these challenges by providing collaborative dev into prod pipelines, reproducible model development, and automated workflows. This infrastructure enables cross organizational collaboration for defense R&D within AWS IL4/5/6 GovCloud.

Edge deployment considerations are critical for maritime intelligence platforms with limited computational resources and connectivity. Our work addresses model optimization techniques that maintain sufficient accuracy while meeting latency and power constraints, deployed through standardized MLOps workflows.

Premium content

Next from NAML 2026

Advancing Fleet-Level Autonomy Through Collaborative and Competitive Control

Similar lecture

Agentic AI Benchmark in Enterprise