United States

State of the art techniques for learning predictive models from naturally occurring data are plagued by the fundamental problem of shortcut learning – often thought of as overfitting in classical machine learning settings. One of these ways is the now infamous adversarial example to a deep image classifier – an imperceptible perturbation (highly correlated with an incorrect label) of the input that drastically reduces the model’s accuracy (Szegedy et al., 2013). A plethora of mitigation strategies have been proposed, the most effective of which rely on adversarial training: teaching the model to be invariant to these kinds of spurious signals by exposing it to adversarial examples themselves during training (Madry et al., 2017).In this talk, we demonstrate that although state-of-the-art adversarial training techniques mitigate spurious correlations that are based on noise, adversarial training does not mitigate more semantically meaningful shortcuts. For example, we show that a model defended with adversarial training will associate the “balance beam” label with the presence of a gymnast in the ImageNet dataset, even though a human would not make that same association to assist in classifying an image as a balance beam. We show the presence of this spurious correlation in the trained model by constructing a series of adversarial examples which are both recognizable to a human, in that adversarial example targeting the stingray class resemble images of stingrays also containing a recognizable snorkeler, demonstrating a semantically incorrect association between the class and the learned features. Using examples like these, we then argue that adversarial training does not solve the core shortcut learning problem -- rather the problem persists at a higher semantic level. 

This finding is important to the Navy: if adversarial training causes models to infer semantically meaningful spurious correlations, and these correlations reveal critical information about the method by which the data was collected, then adversarial training will violate policies related to data security. To this end, we provide proof of concept tools for examining the shortcuts taken by models defended with adversarial training. 

References:
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., &amp; Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., &amp; Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.

Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., &amp; Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). Ieee.

NAML 2021

Current Robustness Techniques do Not Solve the Shortcut Learning Problem

image classification

shortcut learning

adversarial robustness

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

State of the art techniques for learning predictive models from naturally occurring data are plagued by the fundamental problem of shortcut learning – often thought of as overfitting in classical machine learning settings. One of these ways is the now infamous adversarial example to a deep image classifier – an imperceptible perturbation (highly correlated with an incorrect label) of the input that drastically reduces the model’s accuracy (Szegedy et al., 2013). A plethora of mitigation strategies have been proposed, the most effective of which rely on adversarial training: teaching the model to be invariant to these kinds of spurious signals by exposing it to adversarial examples themselves during training (Madry et al., 2017).In this talk, we demonstrate that although state-of-the-art adversarial training techniques mitigate spurious correlations that are based on noise, adversarial training does not mitigate more semantically meaningful shortcuts. For example, we show that a model defended with adversarial training will associate the “balance beam” label with the presence of a gymnast in the ImageNet dataset, even though a human would not make that same association to assist in classifying an image as a balance beam. We show the presence of this spurious correlation in the trained model by constructing a series of adversarial examples which are both recognizable to a human, in that adversarial example targeting the stingray class resemble images of stingrays also containing a recognizable snorkeler, demonstrating a semantically incorrect association between the class and the learned features. Using examples like these, we then argue that adversarial training does not solve the core shortcut learning problem -- rather the problem persists at a higher semantic level. 

This finding is important to the Navy: if adversarial training causes models to infer semantically meaningful spurious correlations, and these correlations reveal critical information about the method by which the data was collected, then adversarial training will violate policies related to data security. To this end, we provide proof of concept tools for examining the shortcuts taken by models defended with adversarial training. 

References:
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.

Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). Ieee.

technical paper

Computational Modeling of Trust Factors in Recognition of

Warfighting Ships

S. Lamczyk, L. Vidyaratne, M. Rahman, and K. M. Iftekharuddin

Vision Lab, Department of Electrical and Computer Engineering, Old Dominion University,

Norfolk, VA 23529
ABSTRACT

As machine-learning (ML) algorithms continue to handle increasingly intricate problems, these methods are often required to process incomplete, imprecise, and ambiguous sensor data for decision making. The capabilities of these ML algorithms must be developed further in order to meet the increasingly complex demand associated with the deployment and integration of decision-making capabilities of intelligent systems into modern society. Furthermore, the variability in the performance of modern ML methods in dynamic environments leads to ambiguity of trust in the decisions made by such algorithms. Consequently, we propose a novel computational modeling for trust quantification to analyze the performance and improve the trustworthiness of autonomous decision-making algorithms. The approach relies on the implementation of a neural network-based reinforcement learning (RL) model known as an adaptive critic design to model an adaptive decision-making process that is regulated by quantitative measure of risk associated with each possible decision. Specifically, this work offers risk-directed exploration strategies of RL to obtain quantitative risk factors for automated object recognition in the presence of imprecise data. The proposed framework is used for automated recognition of warfighting ships in well-known Janes’ Fighting Ships image dataset along with other publicly available crowdsourced images. The results show improved recognition performance with the proposed risk quantification framework. Finally, we present extensions to the proposed recognition model with multimodal input processing capabilities by combining the image recognition model with a state-of-the-art deep learning based optical character recognition (OCR) architecture to generate more specific recognition such as the class, type, and/or name of the ship.

Computational Modeling Of Trust Factors in Recognition of Warfighting Ships

Object tracking is a fundamental computer vision problem found in a wide range of applications. Most state-of-the-art object trackers adopt deep neural networks for high tracking performance at the expense of huge computational resources and heavy memory use. Here, we target at a light-weight high-performance tracker of a much smaller model size, thus meeting the size, weight and power (SWAP) requirements of devices used in modern battlefields.

Light-Weight High-Performance Object Tracking

Reinforcement learning (RL) has been demonstrated suitable to develop agents that play complex games with human-level performance. However, it is not understood how to effectively use RL to perform cybersecurity tasks. To develop such understanding, it is necessary to develop RL agents using simulation and emulation systems allowing researchers to model a broad class of realistic threats and network conditions. Demonstrating that a specific RL algorithm can be effective
for defending a network under certain conditions may not necessarily give insight about the performance of the algorithm when the threats, network conditions, and security goals change. FARLAND is a novel framework to facilitate the development of RL network defenders that are robust against attacks aimed at the agent’s learning. Our contributions include an architecture and implementation of FARLAND, as well as threat modeling concepts that should be considered when designing an AI-driven network defender. We discuss examples that illustrate FARLAND’s advantages for developing RL-enabled defenders against realistic adversaries.


FARLAND (Framework for Advanced Reinforcement Learning for Autonomous Network Defense)

In recent years there has been a significant increase in the amount of disclosed cybersecurity vulnerabilities resulting in lengthy analysis delays, allowing for cyber-attacks to exploit these vulnerabilities before pertinent data can be published to inform cybersecurity professionals. This data includes impact metrics, vulnerability types, and applicability statements. Previous research has focused on predicting the common vulnerability scoring system (CVSS) score calculated from impact metrics, as shown in the National Vulnerability Database (NVD), using methods such as term frequency-inverse document frequency (TF-IDF). This research utilizes natural language processing (NLP) and topic modeling to derive meaning from NVD vulnerability description texts to classify the vulnerabilities, with a predictive model trained with previous years’ data, as related to applications, hardware devices, and/or operating systems. From this classification the total analysis time of future vulnerabilities will be reduced, and cybersecurity professionals will be able to make more informed decisions regarding vulnerability management of assets to aid in preventing future cyber-attacks.

Categorization of Cybersecurity Vulnerabilities Utilizing Machine Learning Methods for Vulnerability Management

This talk consists of four parts as follows: (1) (15min) An overview of the Navy sponsored, ORNL administrated AI ATAC Prize Challenges 1 & 2 will be given. These two challenges were large-scale experiments to evaluate market-leading cybersecurity tools that leverage AI/ML for endpoint malware detection (Challenge 1) and network traffic analysis to detect multi-step adversarial campaigns (Challenge 2). (2) (10min) The second part of this talk gives a deep-dive on how CORR (ORNL's Cyber Operation Research Range), a cybersecurity testbed, was designed and stood-up to host these and other evaluations. (3) (5min) The third portion of the talk gives a brief overview of the machine learning and software engineering task to create D2U, the Data-Driven User Emulator. This allows orchestration of hundreds of VMs on CORR that, guided by trained probabilistic generative models, take actions as real users. (4) (10 min) Finally, the fourth and final portion gives an overview of a cost-benefit framework for simulating how SOCs will use the cybersecurity tools in the test. It allows integration of the many statistical measures (precision, recall, time-to-detect, etc.) into a single comparable metric (cost) for ranking and reasoning about tradeoff's when considering cybersecurity tools.

Artificial Intelligence Applications to Cybersecurity (AI ATAC) Prize Challenges I&II

Forecasting the weather over the oceans of the Earth is a necessary but difficult requirement of US Navy operations around the globe. Before computer science formally existed as a discipline, meteorologists were using the first computers to explore the possibilities of Numerical Weather Prediction (NWP) to support this forecasting. The foundation of NWP is a fluid dynamics model of the atmosphere interacting with a multitude of physical processes that drive the spatiotemporal vector fields of wind and scalar fields of pressure, temperature, and water vapor mixing ratio. Because the spatial domain of the model is essentially the entire Earth and practical forecasting operates on a cadence of updating every six hours, the computational constraints are formidable and require a number of compromises in formulating the model.
The Naval Research Laboratory (NRL) has over a few decades developed the Navy’s primary NWP model called the Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS). COAMPS has effectively supported naval operations at the Fleet Numerical Meteorology and Oceanography Operations Center (FNMOC) for the past 20 years but is going to be replaced by a more advanced model called Neptune in the next few years. As a part of this process, NRL has a project to understand how Machine Learning (ML) might be used to improve the forecasting accuracy and computational efficiency of this new system. In particular, there has recently been significant progress in applying ML to model turbulence in fluid dynamics. Since proper representation of turbulence in NWP models has been identified as a weakness, the opportunity of using ML to improve weather forecasting has focused on this aspect of the problem.
To support a data-driven ML approach to improving the NWP model, a series of Large Eddy Simulations (LES) have been performed at spatial resolutions not possible with operational NWP models to provide “ground truth” data for how turbulence behaves in the atmosphere. The goal is to learn both the relevant combinations of state variables and the partial differential equations of the spatial dynamics for insertion into the NWP model. This approach requires data and model compression to find useful state variables (or feature vectors) via unsupervised learning since not all quantities available in the LES data can be represented by the low resolution NWP model. Given a suitable space of feature vectors, a supervised learning step finds the partial differential equations for the NWP model that match the dynamics of the LES data. The talk will emphasize the broader issues of how ML can be used in the scientific domain when complex modeling choices can be better resolved with a data-driven approach.


Using Machine Learning to Improve Weather Forecasting for Naval Operations

The traditional approach to photonic design (the inverse problem) is to generate a solution based on intuition and expert knowledge, evaluate how closely it approximates the desired result by electromagnetic simulation (forward problem), and optionally optimize over a few input parameters. However, this approach is limited in what areas of the design space it can reach.. Alternatively, we may computationally optimize over many parameters simultaneously, effectively parameterizing the entire design space, but algorithms for doing so require many costly evaluations of the forward problem. In this work we apply a neural network to approximate simulation output, making the forward problem less costly to solve and the inverse problem more tractable. The neural network is used to assist traditional gradient descent in finding the optimum solution for a given design target. We consider the design of a non periodic grating as a proof of concept. Combining the neural network with gradient-based optimization, our algorithm designs a grating with a spectrum matching a target from the test set with R^2=.99 and an arbitrary spectrum with R^2=.91. These results serve as useful proof of concept for this technique and an opportunity to apply it to new problems in photonic design. 


Deep Inverse Design of a Photonic Structure

Simulating a fluid flow involving chemical reactions is critical to many scientific problems, from climate modeling to propulsion system design. Since chemical reactions can occur very quickly, these simulations typically solve the fluid state separately from the chemical state, syncing up at regular intervals. Unfortunately, the chemistry is a significant bottleneck in many practical problems: it scales very poorly as reactions get more complex. As a result, many useful simulations are computationally prohibitive, forcing researchers to rely on low-fidelity reduced order models, very small-scale simulations, or even forego the problem altogether. Deep learning is gaining traction as a way to develop data-driven surrogate models that are accurate and fast enough to overcome these bottlenecks; however, little work has been done to apply this to chemical kinetics. Today, we will present our work evaluating a broad spectrum of surrogate modeling approaches for stiff chemical kinetics. We discuss the strengths and weaknesses of different modeling and training strategies, highlighting the unique challenges of stiff time-series forecasting and recommending several promising paths forward. Finally, we will discuss the potential for machine learning for developing surrogate chemistry solvers in practical CFD.

Introducing Machine Learning Solutions for Chemically Reacting Flow Simulations

SpaceNet 6 was a geospatial data science challenge centered on foundational mapping from multimodal data. It was one in a series of challenges organized by SpaceNet, which accelerates geospatial deep learning by releasing open datasets, organizing deep learning challenges, and open-sourcing the resulting algorithms and analysis.

SpaceNet 6 introduced a new dataset covering the vicinity of the port of Rotterdam in the Netherlands. The multimodal dataset includes building footprint polygons, electro-optical imagery, and synthetic aperture radar (SAR). The high resolution of the SAR (processed at 0.5m for the challenge, the same resolution as the corresponding optical data) makes the dataset especially valuable. SAR works under nighttime and cloudy conditions when visible satellite imagery cannot be acquired. To make the challenge mimic such a real-world scenario, participants were permitted to use the SAR and electro-optical data for training but were only provided SAR for testing. The specific task was to identify building footprints in the SAR imagery.

An analysis of the winning models from SpaceNet 6 provides many lessons about using deep learning with SAR. Winning models significantly outperformed a baseline model by using EfficientNet architectures and incorporating pretrained weights. The ability to detect buildings varies with building size and height, with similar trends seen across models. Layover, a geometric effect intri

SpaceNet 6: Results from a Geospatial Deep Learning Challenge with Synthetic Aperture Radar Imagery

Threat Evaluation algorithms are designed for aiding in identifying which incoming tracks are threats to the surface fleet. The use of Artificial Intelligence (AI) and Machine Learning (ML) algorithms will speed up the threat identification process, allowing more time for coordinating defensive measures. Currently, the threat evaluation process is performed through verbal communication by a set of human operators making real time decisions on tracks as seen on the radar console. There are many factors that can be considered in threat evaluation; the data load and data processing rate required for an optimal real-time assessment far exceed human capacity. The conversion of as-received data to informed decisions can be performed faster with AI/ML. One such AI/ML approach that can be used is the Random Forest Classifier algorithm. This is not only more efficient but also more accurate than the current threat evaluation process because it can be accomplished with a series of measurements happening at machine speeds instead of human processing time. The Random Forest Classifier is chosen for this analysis because unlike most AI/ML algorithms, its process is transparent. It consists of a series of decision trees, so an operator can look at the decisions being made by the algorithm and make a final decision. In Random Forest implementations, each decision tree runs on a subset of all the features that are provided and evaluates the split of data made on those features using a mathematical equation. The equation implemented in this iteration of Random Forest is a Gini Impurity split measurement. This measurement takes into consideration the probability of a particular feature being wrongly classified when the data is split. This equation is able to handle both numerical and categorical data and performs well with large data sets. As with any AI model, Random Forest needs to be trained. The current iteration of the Random Forest Classifier was first run on a known set of Sonar Data obtained from open sources and correctly identified those data points 84% of the time. Next, training data was obtained by using the Ship Air Defence Model (SADM) and the Joint Cognitive Opertional Research Environment (JCORE) running multiple scenarios containing both threat and non-threat air entities to create radar and Electronic Support track data. This paper provides the results of running the Random Forest Classifier on test sets selected from the training data. The results are then discussed and plans for future work are presented. 
Classification: UNCLASSIFIED
Distribution Statement: Distribution Statement A



Premium content

Current Robustness Techniques do Not Solve the Shortcut Learning Problem

Downloads

Next from NAML 2021

Computational Modeling Of Trust Factors in Recognition of Warfighting Ships

Similar lecture

Quantifying Uncertainty in Multispectral Satellite Image Classification using Bayesian Deep Learning