Italy

In this paper, we propose the research of the recursive division approach to get an explanation for a particular decision of the shallow black box model. The core of the proposed method is the division of the image being classified into separate rectangular parts, followed by the analysis of their influence on the classification result. Such divisions are repeated recursively until the explanation of the classification result is found, or the size of parts is too small. As a result, the pair of images with complement hidden parts is discovered, the first one of which preserves both the most valuable parts and the classification result of the initial image. The second image represents the result of hiding the most valuable parts of an initial image that leads to the different classification for the binary classification problem. Experimental research (applied for Food-5K and concrete crack images datasets) proved that the quality of the proposed method might be close to LIME or even better, while the performance of recursive division is better.

ICPR 2020

Recursive Division of Image for Explanation of Shallow CNN Models

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


technical paper

Deep Neural Networks are becoming the prominet solution
when using machine learning models. However, they suffer of a black-
box effect that renders complicated their inner workings interpretation
and thus the understanding of their successes and failures. Information
visualisation is one way among others to help in their interpretability
and hypothesis deduction. This paper presents a novel way to visualize
a trained DNN in order to depict at the same time its architecture
and its way of treating the classes of a test dataset at the layer level.
This way, it is possible to visually detect where the DNN starts to be
able to discriminate the classes or where it could decrease its separation
ability (and thus detect an oversized network). We have implemented the
approach and validated it using several well-known datasets and networks.
Results show the approach is promising and deserves further studies.

Samples Classification Analysis Across DNN Layers with Fractal Curves

Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical especially when they are now being adopted over diverse aspects of human life. Taking the advantages of interpretable machine learning (interpretable ML), this paper proposes a novel tool called Catastrophic Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual learning settings. We also introduce a new method called Critical Freezing based on the observations of our tool. Experiments on ResNet-50 articulate how catastrophic forgetting happens, particularly showing which components of this famous network are forgetting. Our new continual learning algorithm defeats various recent techniques by a significant margin, proving the capability of the investigation. Critical freezing not only attacks catastrophic forgetting but also exposes explainability.

Explaining How Deep Neural Networks Forget by Deep Visualization

In this talk we present our work on a perplexity-based sparsity definition to derive and visualise layer-wise activation measures. These novel explainable AI strategies reveal a surprising relationship between activation sparsity and overfitting. Our methods reveal novel insights into the contradicting concepts of activation sparsity and network capacity by demonstrating that dense activations can enable discriminative feature learning while efficiently exploiting the capacity of deep models without suffering from overfitting, even when trained excessively.

The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks

Explanations for algorithmic decision systems can take different forms, they can target different types of users with different goals. One of the main challenges in this area is therefore to devise explanation methods that can accommodate this variety of situations. A first step to address this challenge is to allow explainees to express their needs in the most convenient way, depending on their level of expertise and motivation. In this paper, we present a solution to this problem based on a multi-layered approach allowing users to express their requests for explanations at different levels of abstraction. We illustrate the approach with the application of a proof-of-concept system called IBEX to two case studies.

A Multi-layered Approach for Tailored Black-box Explanations

Machine learning is currently successfully used for addressing several cybersecurity detection and classification tasks. Typically, such detectors are modeled through complex learning algorithms employing a wide variety of features. Although these settings allow achieving considerable performances, gaining insights on the learned knowledge turns out to be a hard task. To address this issue, research efforts on the interpretability of machine learning approaches to cybersecurity tasks is currently rising. In particular, relying on explanations could improve prevention and detection capabilities since they could help human experts to find out the distinctive features that truly characterize malware attacks. In this perspective, Android ransomware represents a serious threat. Leveraging state-of-the-art explanation techniques, we present a first approach that enables the identification of the most influential discriminative features for ransomware characterization. We propose strategies to adopt explanation techniques appropriately and describe ransomware families and their evolution over time. Reported results suggest that our proposal can help cyber threat intelligence teams in the early detection of new ransomware families, and could be applicable to other malware detection systems through the identification of their distinctive features.

Explanation-driven Characterization of Android Ransomware

This paper profiles the recent research work on eXplainable AI (XAI), at the Insight Centre for Data Analytics. This work concentrates on post-hoc ex-planation-by-example solutions to XAI as one approach to explaining black box deep-learning systems. Three different methods of post-hoc ex-planation are outlined for image and time-series datasets: that is, factual, counterfactual, and semi-factual methods). The future landscape for XAI solutions is discussed.

Post-Hoc Explanation Options for XAI in Deep Learning: The Insight Centre for Data Analytics Perspective

Neural networks are increasingly applied to high-stakes tasks, such as autonomous driving and medical applications. For these tasks, it is important to explain the contributions of data components to a model prediction. In recent times, mask-based methods have been proposed for visual explanations. They optimize masks that maximally affect the model output. In this work, we propose a novel mask-based saliency method for given black box classifiers. We jointly optimize positive and negative masks to achieve a faithful feature importance map. To optimize them effectively, we define a distance between them in terms of the selected activation maps; the corresponding kernels express the important features of an input image. Then, we impose the distance to the objective function of each mask as a regularizer. By forcing both masks to be dissimilar in terms of the influential features, they can focus on essential parts of an object while alleviating noises.

Jointly Optimize Positive and Negative Saliencies for Black Box Classifiers

In the astrophysics domain, the detection and description of gamma rays is a research direction for our understanding of the universe. Gamma-ray reconstruction from Cherenkov telescope data is multi-task by nature. The image recorded in the Cherenkov camera pixels relates to the type, energy, incoming direction and distance of a particle from a telescope observation. We propose γ-PhysNet, a physically inspired multi-task deep neural network for gamma/proton particle classification, and gamma energy and direction reconstruction. As ground truth does not exist for real data, γ-PhysNet is trained and evaluated on large-scale
Monte Carlo simulations. Robustness is then crucial for the transfer of the performance to real data. Relying on a visual explanation method, we evaluate the influence of attention on the variability due to weight initialization, and how it helps improve the robustness of the model. All the experiments are conducted in the context of single telescope analysis for the Cherenkov Telescope Array simulated data analysis.

Deep Learning for Astrophysics, Understanding the Impact of Attention on Variability Induced by Parameter Initialization

Machine Learning (ML) is becoming an increasingly critical technol-ogy in many areas such as health, business but also in everyday applications of significant societal importance. However, the lack of explainability or ability of ML systems to offer explanation on how they work, which refers to the model (related to the whole data) and sample explainability (related to specific sam-ples) poses significant challenges in their adoption, verification, and in ensur-ing the trust among users and general public. We present novel integrated Ran-dom Forest Model and Sample Explainer – RFEX. RFEX is specifically de-signed for important class of users who are non-ML experts but are often the domain experts and key decision makers. RFEX provides easy to analyze one-page Model and Sample explainability summaries in tabular format with wealth of explainability information including classification confidence, tradeoff be-tween accuracy and features used, as well as ability to identify potential outlier samples and features. We demonstrate RFEX on two case studies: mortality prediction for COVID-19 patients from the data obtained from Huazhong Uni-versity of Science and Technology, Wuhan, China, and classification of cell type clusters for human nervous system based on the data from J. Craig Venter Institute. We show that RFEX offers simple yet powerful means of explaining RF classification at model, sample and feature levels, as well as providing guid-ance for testing and developing explainable and cost-effective operational pre-diction models.

Random Forest Model and Sample Explainer for non-experts in Machine Learning – Two Case Studies

Emotions are very important in dealing with human decisions, interactions, and cognitive processes. Art is an imaginative human creation that should be appreciated, thought-provoking, and elicits an emotional response. The automatic recognition of emotions triggered by art is of considerable importance. It can be used to categorize artworks according to the emotions they evoke, recommend paintings that accentuate or balance a particular mood, and search for paintings of a particular style or genre that represent custom content in a custom state of impact. In this paper, we propose an attention-based multi-modal approach to emotion recognition that aims to use information from both the painting and title channels to achieve more accurate emotion recognition. Experimental results on the WikiArt emotion dataset showed the efficiency of the model we proposed and the usefulness of image and text modalities in emotion recognition.

Premium content

Recursive Division of Image for Explanation of Shallow CNN Models

Downloads

Next from ICPR 2020

Samples Classification Analysis Across DNN Layers with Fractal Curves

Similar lecture

Feature Pyramid Hierarchies for Multi-scale Temporal Action Detection