United States

Background: The Millennium Cohort Study (MCS) is a prospective cohort study designed to examine the long-term effects of military service with surveys administered every 3-5 years. Participant non-response to follow-up surveys is a potential threat to the validity and generalizability of study findings. In recent years, predictive analytics has emerged as a promising tool to identify predictors of non-response.

Methods: Among participants enrolled at baseline and met eligibility criteria (n=70,117), we utilized a temporal cross-validation method to predict response to the 2019-2021 MCS follow-up survey and compared prediction performance of six machine learning algorithms. Using Latent Class Analysis (LCA), we classified participants based on historical response to the baseline survey administered in 2001-2003 and follow-up surveys administered in 2004-2006, 2007-2008, 2011-2013, 2014-2016. We compared prediction performance with and without the LCA variable. Feature analysis was subsequently run on the best performing model to identify the most important predictors with and without LCA. 

Results: The LCA variable consisted of six distinct classes, comprised of consistent responders (web, paper, and mode switcher), consistent non-responders (early and late drop-offs), and inconsistent responders. Notably, the LCA classes were found to be statistically significant across various military and demographic characteristics. 

Random forest outperformed all other machine learning algorithms in both analyses, with and without LCA. Without the LCA variable, random forest outperformed the benchmark regression model by 9.3% in area under the curve (AUC) of the receiver operating characteristic (ROC) curve and 7.7% in AUC of precision-recall. Including the LCA variable, all six algorithms performed comparably with one another, with random forest outperforming the benchmark regression model by 2.3% ROC AUC and 1.6% for precision-recall AUC. Moreover, inclusion of the LCA variable increased the ROC AUC for all algorithms by 10% or more, except for random forest and interaction forest which improved by approximately 5% in ROC AUC.

Feature analysis indicated LCA was the most important predictor in the model. In general, participants identified as consistent responders were predicted to respond to follow-up surveys (predicted probability, ŷ &gt; 0.95), while consistent non-responders were predicted to not respond to follow-up surveys (ŷ &lt; -0.84). Predicting response behavior among inconsistent responders appeared to be the most challenging (ŷ = 0.41).

Conclusion: Our findings suggest that historical response patterns can significantly improve the ability to predict participant response to follow-up surveys. Machine learning algorithms may be valuable when historical data are not available. Implementing these methods in longitudinal studies can enhance outreach efforts by strategically targeting participants and ultimately boost survey response and mitigate non-response. 


NAML 2024

Enhancing Targeted Outreach for Longitudinal Surveys: Predictive Analytics for Participant Response in the Millennium Cohort Study

survey outreach efforts

latent class analysis

survey response

survey nonresponse

prospective cohort study

millennium cohort study

predictive analytics

longitudinal studies

machine learning

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-26 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to AAAI 2026". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you are required to register.
Please complete your registration to continue.

We recommend reading [**the registration information**](https://aaai.org/conference/aaai/aaai-26/registration/) first.

**Online Registration Form**: https://aaai.getregistered.net/conference-2026 

Registration Required

AAAI 2026

We are pleased to announce the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), which will be held in Singapore EXPO from January 20 to January 27, 2026.

Background: The Millennium Cohort Study (MCS) is a prospective cohort study designed to examine the long-term effects of military service with surveys administered every 3-5 years. Participant non-response to follow-up surveys is a potential threat to the validity and generalizability of study findings. In recent years, predictive analytics has emerged as a promising tool to identify predictors of non-response.

Methods: Among participants enrolled at baseline and met eligibility criteria (n=70,117), we utilized a temporal cross-validation method to predict response to the 2019-2021 MCS follow-up survey and compared prediction performance of six machine learning algorithms. Using Latent Class Analysis (LCA), we classified participants based on historical response to the baseline survey administered in 2001-2003 and follow-up surveys administered in 2004-2006, 2007-2008, 2011-2013, 2014-2016. We compared prediction performance with and without the LCA variable. Feature analysis was subsequently run on the best performing model to identify the most important predictors with and without LCA. 

Results: The LCA variable consisted of six distinct classes, comprised of consistent responders (web, paper, and mode switcher), consistent non-responders (early and late drop-offs), and inconsistent responders. Notably, the LCA classes were found to be statistically significant across various military and demographic characteristics. 

Random forest outperformed all other machine learning algorithms in both analyses, with and without LCA. Without the LCA variable, random forest outperformed the benchmark regression model by 9.3% in area under the curve (AUC) of the receiver operating characteristic (ROC) curve and 7.7% in AUC of precision-recall. Including the LCA variable, all six algorithms performed comparably with one another, with random forest outperforming the benchmark regression model by 2.3% ROC AUC and 1.6% for precision-recall AUC. Moreover, inclusion of the LCA variable increased the ROC AUC for all algorithms by 10% or more, except for random forest and interaction forest which improved by approximately 5% in ROC AUC.

Feature analysis indicated LCA was the most important predictor in the model. In general, participants identified as consistent responders were predicted to respond to follow-up surveys (predicted probability, ŷ > 0.95), while consistent non-responders were predicted to not respond to follow-up surveys (ŷ < -0.84). Predicting response behavior among inconsistent responders appeared to be the most challenging (ŷ = 0.41).

Conclusion: Our findings suggest that historical response patterns can significantly improve the ability to predict participant response to follow-up surveys. Machine learning algorithms may be valuable when historical data are not available. Implementing these methods in longitudinal studies can enhance outreach efforts by strategically targeting participants and ultimately boost survey response and mitigate non-response.

poster

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this workshop.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Recognizing the imperative to address the reliability and transparency issues of
Large Language Models (LLM), this work proposes an LLM maturity model
tailored for text-to-query applications. This maturity model seeks to fill the existing void in evaluating LLMs in such applications by incorporating dimensions
beyond mere correctness or accuracy. Moreover, this work introduces a realworld use case from the law enforcement domain and showcases QueryIQ, an
LLM-powered, domain-specific text-to-query assistant to expedite user workflows and reveal hidden relationship in data.

Harvesting Trustworthy AI for DOD Superiority

Decision tree models form the basis for many machine learning approaches and have several useful properties; however, single-tree applications are limited by the balance between predictive accuracy and generalization. We introduce a new method for increasing the generalized predictive performance of pre-trained decision tree models. Using this method, the structure of the tree is leveraged to derive a set of decision threshold-based weights controlling the relative contribution of prediction values. The final prediction is then the result of an aggregation of these weighted values. This “soft trees” approach is applied to a set of benchmark datasets and is shown to increase r2 performance over standard decision trees. We further explore the introduction of missing or categorical input features, utility in probabilistic classification, and applications to tasks in resource-limited hardware environments.

Improving decision tree generalization using threshold smoothing

An average enterprise may use over 100 applications (even if no one can actually count them all), some redundant, some outside the visibility of a documented enterprise architecture and certainly not all integrated. As a result, users spend hours moving information between applications and ensuring the consistency of that data to the extent of their desire and ability. 
This results in mindless repetition that takes its toll creating stress-related physical conditions costing trillions of dollars and introducing expensive errors into every industry. Data quality is a root cause of many of those errors. At a time when the volume and importance of data in every enterprise is growing at exponential rates, ways to ensure the quality and accuracy of enterprise data are needed; automation is one way which brings almost unbelievable speed to the processing of it.
Automation technologies build the foundation for expanding into cognitive automation with the integration of machine learning, deep learning, sensor technology, and other emerging technologies. Strengthening operational insights with the ability to combine analytics of larger sets of data with automation is an additional benefit of adoption. 
Within the context of digital transformation, automation refers to the methods and technologies used to generate software-based execution of operational processes or tasks within processes to minimize the involvement of humans. “Intelligent automation, also referred to as hyperautomation] is achieved by mimicking the capabilities that knowledge workers use in performing their work activities (e.g., language, vision, execution and thinking & learning).” (“Intelligent Automation,” Bornet, et al.) Multiple technologies support each of the four work activities; there is no single, pre-integrated, all-encompassing technology solution. 
There are many technologies that have been in use for years, such as optical character recognition and programmatic interfaces, that feed into newer automation technologies such as smart workflows, low-code/no-code software development, and robotic process automation. An enterprise automation platform will include many tools, connections, and reusable components that can link applications and systems of record, presenting a more powerful engagement layer to interface with customers and employees. As an enterprise gains in automation maturity, more elements of artificial intelligence, including machine learning and deep learning, can be integrated along with the data collected by biometrics. API management and application management are perhaps the oldest automation techniques and most found within an enterprise architecture. 
This poster breaks down the need for intelligent automation and shares the progression of technologies that are in. Use cases within the Navy mission will be shared along with NIWC successes.


Intelligent Automation; Why and How

HorizonHunt is a cutting-edge GenAI system designed to aggregate and assimilate multifaceted data and render it fully searchable through an intuitive conversational interface. The HorizonHunt data lake and AI multi-modal platform uniquely integrates image, audio, text, PDF, csv, and other data modalities at enterprise scale and speed, enabling end-users to intuitively query within and across datasets using plain English. 

Benefits include: faster time to insight, multi-modal data analysis, domain-specific insights, adaptive learning, and reliable intelligence for Naval operations. 

Prototype datasets include: 
Static Data – 9/11 Commission Report including: PDFs, Text, Audio, Images
Dynamic Data – GDELT including: Text, CSV, Long/Lat 


Looking beyond the obvious – utilizing generative AI and LLMs to investigate relationships within multimodal datasets within a data lake.

The objective of this project is to use machine learning convolutional neural networks (CNN) to classify marine mammal spectrograms. CNNs excel at image recognition and can identify features and patterns in images. Spectrograms of marine mammal acoustics can be treated as images and put into a standardized format to be used as an input into a CNN model for classification, allowing for a more structured representation of the data than raw time series. The Marine Mammal Monitoring (M3R) on Navy ranges has an extensive data set of both labeled and unlabeled recordings, online sources such as the Watkins marine mammal sound data base have a smaller extent of data but more accessible labeled data. By taking advantage of these online data sets a robust CNN model can be developed and applied to a subset of M3R data to make classifications of marine mammals on navy ranges. 


Marine Mammal Classification using CNNs 

Generative AI (GenAI) and Large Language Models (LLMs) have become increasingly prominent in National Security, especially in the Department of the Navy (DON). Navy CIO Jane Rathbun's September 6th memo, "DON Guidance on Generative AI & LLMs," highlights concerns about LLMs producing biased or inaccurate text. Understanding the importance of reducing biased results, SGS data scientists have researched approaches to mitigate “hallucinations.” The first is simple; set LLM temperature to zero. Temperature is a hyper-parameter that impacts output diversity or randomness. The second is also simple, but mundane, time consuming, and difficult to scale. Provide structured text during training. A problem facing the Navy is the majority of this text is unstructured and locked away in Portable Document Format (PDF) format. SGS data scientists have applied a suite of labeling tools to develop a significantly more effective and efficient method for parsing PDFs. This novel method allows us to train a model that adaptively learns how certain publications are structured, and is an approach to enable the DON's use of GenAI.

Navigating the Seas of Uncertainty: Mitigating Risks in Large Language Models (LLMs)

The Office of the Naval Inspector General (NAVINSGEN) independently and objectively inspects, investigates, and inquires into matters of importance to the Department of the Navy. This includes handling hotline complaints received in unpredicted volumes with various priority levels based on the importance and emergency level of the case. To improve the intake process, BAH developed a Machine Learning pipeline to convert complaint forms to structural data, clean text data, and leverage Natural Language Processing (NLP) models to predict and rank the priority level for the IG intake team. The solution is backed by Databricks and lives on Naval data framework, Jupiter on Advana. In finding the optimal NLP model, the team has experimented with various transformers and techniques combined with MLFlows. This solution allows decision makers in NAVINSGEN to speed up the complaint review process by automatically suggesting and flagging critical cases.

NLP Application on Naval IG Hotline using Jupiter Platform

Snowflake Cortex is an intelligent, fully managed service that offers access to industry-leading AI models, LLMs and vector search functionality to enable organizations to quickly analyze data and build AI applications. Snowflake Container Services enables developers to effortlessly deploy, manage and scale custom containerized workloads and models for tasks such as fine-tuning open-source LLMs using secure Snowflake-managed infrastructure with GPU, in partnership with NVIDIA, instances all within the boundary of their Snowflake account.


Obfuscating GenAI Complexities

A new data-adaptive model via machine learning is investigated as a prospective capability for ocean wave height estimation based on simulated ship motion time-series. Long Short-Term Memory (LSTM) neural networks are trained and tested with physics-based Large Amplitude Motion Program (LAMP) simulation data for the ship geometry and loading conditions of the David Taylor Model Basin (DTMB) Model 5415. From this initial study, data-adaptive machine learning appears to be a suitable candidate for continued investigation of estimation of the ocean wave field based on ship motions.

Ocean Wave Height Estimation via Machine Learning

Radio frequency fingerprinting (RFF) identifies radio frequency (RF) emitters based on signatures that the individual emitter’s hardware imprints into the RF signal. There have been significant advancements in RFF but most of the advancements investigated modern machine learning approaches to radio frequency signals where classic RF processing techniques and machine learning are combined. However, these methods often lack the robustness to operate in many DOD environments that are highly uncertain, complex, and dynamic which, in the RF communication application, is often due to the communication channel. To overcome this communication channel problem, this work begins an investigation of integrating physics-based and explainable models directly into the machine learning approach. This enables RF subject matter expertise to improve the RF fingerprinting results. In this work two approaches are investigated: networks that respect complex multiplication (complex networks); and data collections from highly variable channel conditions. 
Radio frequency fingerprinting has shown great potential for device authentication and access control in wireless communication networks. The signal intelligence (SIGINT) and data science communities have developed efficient methods to identify transmitters (Tx) based on unique signatures of individual Tx hardware. These signatures are due to intrinsic characteristics of the Tx hardware which unintentionally affect the waveforms leading to deviation from ideal waveforms. The hardware specific deviation from the ideal waveform is usually minor enough that the signal message is not impacted, the bit error rate (BER) is maintained, and the decoded messages remain intact. Since these characteristics are created by non-ideal components, such as parasitic circuitry, component tolerance, or temperature stability, they are hard to mimic in an electronic warfare (EW) when compared to software-level identification such as internet protocol (IP) or media access control (MAC) addresses.
As is the case with any wireless communication systems, the received signal is heavily impacted by noise and the channel i.e., attenuation, reflection, diffraction, and refraction paths. These effects will cause modifications of the received signal when compared to training signals which can cause a degradation in the classification performance. A RF reverberation chamber (R-chamber) is used to introduce data drift caused by multipath channel effects to the RF transmission. The R-chamber's tuner is rotated during the training stage to randomize the electromagnetic field inside the chamber and simulate a random multipath. 
Previous RF fingerprinting studies used deep neural network classifiers, and it has been observed that for Wi-Fi signals, RF fingerprinting was improved using complex-valued networks. Their study was in a structured environment with the only channel noise arising from atmospheric conditions. This work extends further explores the complex network concepts using more sophisticated channel noise generated using a R-chamber. We show an increase in performance when the network is tested in channel conditions not exposed to the training dataset demonstrating the utility of complex-valued network.

Premium content

Next from NAML 2024

Harvesting Trustworthy AI for DOD Superiority

Similar lecture

SpiderGen: Towards Procedure Generation for Carbon Life Cycle Assessments with Generative AI