Italy

Generative adversarial networks (GANs) are deep neural networks that allow us to sample from an arbitrary probability distribution without explicitly estimating the distribution. There is a generator that takes a latent vector as input and transforms it into a valid sample from the distribution. There is also a discriminator that is trained to discriminate such fake samples from true samples of the distribution; at the same time, the generator is trained to generate fakes that the discriminator cannot tell apart from the true samples. Instead of learning a global generator, a recent approach involves training multiple generators each responsible from one part of the distribution. In this work, we review such approaches and propose the hierarchical mixture of generators, inspired from the hierarchical mixture of experts model, that learns a tree structure implementing a hierarchical clustering with soft splits in the decision nodes and local generators in the leaves. Since the generators are combined softly, the whole model is continuous and can be trained using gradient-based optimization, just like the original GAN model. Our experiments on five image data sets, namely, MNIST, FashionMNIST, UTZap50K, Oxford Flowers, and CelebA, show that our proposed model generates samples of high quality and diversity in terms of popular GAN evaluation metrics. The learned hierarchical structure also leads to knowledge extraction.

ICPR 2020

Hierarchical Mixtures of Generators for Adversarial Learning

hierarchical mixtures of experts

generative adversarial networks

unsupervised learning

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


technical paper


Online streaming services have been growing at a fast pace. To provide the best user experience, it is needed to detect low quality images from videos so that we can repair or improve them before showing to customers. For example, for movie and TV-show streaming services, it is important to check if an original (master) video produced from a studio has low quality images that contain artifacts such as up-scaling or interlacing; for live streaming services, it is important to detect if a streamed video have hits due to encoding such as H.264 or MPEG-2. The impairment detection is usually measured by the non-reference (NR) metrics because it is often difficult and sometimes impossible to get the original (master) videos. On the other hand, today researches in the image quality area, such as super-resolution, are mainly focused on the full reference (FR) metrics like PSNR or VMAF. In this paper, we present an algorithm that is able to reliably compute five types of spatial NR metrics that are commonly used in the online streaming industries. The algorithm consists of two components: a pre-processing step that spatially de-correlates pixel intensity values and a novel deep neural network (DNN) that is able to quantify the NR metrics at the image region level. We show that our algorithm achieves better performance than state-of-art algorithms in this area.

A Scalable Deep Neural Network to Detect Low Quality Images Without a Reference

Aesthetic assessment is subjective, and the distribution of the aesthetic levels is imbalanced. In order to realize the auto-assessment of photo aesthetics, we focus on using repetitive self-revised learning (RSRL) to train the CNN-based aesthetics classification network by imbalanced data set. As RSRL, the network is trained repetitively by dropping out the low likelihood photo samples at the middle levels of aesthetics from the training data set based on the previously trained network. Further, the retained two networks are used in extracting highlight regions of the photos related with the aesthetic assessment. Experimental results show that the CNN-based repetitive self-revised learning is effective for improving the performances of the imbalanced classification.

CNN-based repetitive self-revised learning for photos’ aesthetics imbalanced classification

Human activity recognition is a fundamental and challenging task in computer vision. Its solution can support multiple and diverse applications in areas including but not limited to smart homes, surveillance, daily living assistance, Human-Robot Collaboration (HRC), etc. In realistic conditions, the complexity of human activities ranges from simple coarse actions, such as sitting or standing up, to more complex activities that consist of multiple actions with subtle variations in appearance
and motion patterns. A large variety of existing datasets target specific action classes, with some of them being coarse and others being fine-grained. In all of them, a description of the action and its complexity is manifested in the action label sentence. As the action/activity complexity increases, so is the label sentence size and the amount of action-related semantic information contained in this description. In this paper, we propose an approach to exploit the information content of these
action labels to formulate a coarse-to-fine action hierarchy based on linguistic label associations, and investigate the potential benefits and drawbacks. Moreover, in a series of quantitative and qualitative experiments, we show that the exploitation of this hierarchical organization of action classes in different levels of granularity improves the learning speed and overall performance of a range of baseline and mid-range deep architectures for human action recognition (HAR).


Extracting Action Hierarchies from Action Labels and their Use in Deep Action Recognition

Multi-step (also called n-step) methods in Reinforcement Learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in tasks exploiting tabular representation of the value-function. Recently, research in Deep Reinforcement Learning (DRL) also shows that multi-step methods improve learning speed and final performance in applications where the value-function and policy are represented with deep neural networks. However, there is a lack of understanding about what is actually contributing to the boost of performance. In this work, we analyze the effect of multi-step methods on alleviating the overestimation problem in DRL, where multi-step experiences are sampled from a replay buffer. Specifically building on top of Deep Deterministic Policy Gradient (DDPG), we propose Multistep DDPG (MDDPG), where different step sizes are manually set, and a variant called Mixed Multi-step DDPG (MMDDPG) where an average over different multi-step backups is used as an update target for the Q-value function. Empirically, we show that both MDDPG and MMDDPG are significantly less affected by the overestimation problem than DDPG with 1-step backup, which consequently results in better final performance and learning speed. We also discuss the advantages and disadvantages of different ways to do multi-step expansion in order to reduce approximation error, and expose the tradeoff between overestimation and underestimation that underlies offline multi-step methods. Finally, we compare the computational resource needs of MDDPG and MMDDPG with those of Twin Delayed Deep Deterministic Policy Gradient (TD3), a state-of-the-art algorithm proposed to address overestimation in actor-critic methods, since they show comparable final performance and learning speed.

The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning

Light-field cameras are becoming more popular in the consumer market. Their data redundancy allows, in theory, to accurately refocus images after acquisition and to predict the depth of each point visible from the camera. Combined, these two features allow for the generation of full-focus images, which is impossible in traditional cameras.

Multiple methods for depth prediction from light fields (or stereo) have been proposed over the years. A large subset of these methods relies on cost-volume estimates -- 3D objects where each layer represents a heuristic of whether each point in the image is at a certain distance from the camera. Generally, this volume is used to regress a depth map, which is then refined for better results. In this paper, we argue that refining the cost volumes is superior to refining the depth maps in order to further increase the accuracy of depth predictions. We propose a set of cost-volume refinement algorithms and show their effectiveness.

Cost Volume Refinement for Depth Prediction

Unpaired image-to-image (I2I) translation has received considerable attention in pattern recognition and computer vision because of recent advancements in generative adversarial networks (GANs). However, due to the lack of explicit supervision, unpaired I2I models often fail to generate realistic images, especially in challenging datasets with different backgrounds and poses. Hence, stabilization is indispensable for GANs and applications of I2I translation. Herein, we propose Augmented Cyclic Consistency Regularization (ACCR), a novel regularization method for unpaired I2I translation. Our main idea is to enforce consistency regularization originating from semi-supervised learning on the discriminators leveraging real, fake, reconstructed, and augmented samples. We regularize the discriminators to output similar predictions when fed pairs of original and perturbed images. We qualitatively clarify why consistency regularization on fake and reconstructed samples works well. Quantitatively, our method outperforms the consistency regularized GAN (CR-GAN) in real-world translations and demonstrates efficacy against several data augmentation variants and cycle-consistent constraints.

Augmented Cyclic Consistency Regularization for Unpaired Image-to-Image Translation

Scene text recognition (STR) is still a hot research topic
in computer vision field due to its various applications.
Existing works mainly focus on learning a general model
with a huge number of synthetic text images to recognize
unconstrained scene texts, and have achieved substantial
progress. However, these methods are not quite applicable
in many real-world scenarios where 1) high recognition
accuracy is required, while 2) labeled samples are lacked.
To tackle this challenging problem, this paper proposes a
few-shot adversarial sequence domain adaptation (FASDA)
approach to build sequence adaptation between the synthetic
source domain (with many synthetic labeled samples) and a
specific target domain (with only some or a few real
labeled samples). This is done by simultaneously learning
each character’s feature representation with an attention
mech- anism and establishing the corresponding
character-level latent subspace with adversarial learning.
Our approach can maximize the character-level confusion
between the source domain and the target domain, thus
achieves the sequence-level adaptation with even a small
number of labeled samples in the target domain. Extensive
experiments on various datasets show that our method
significantly outperforms the finetuning scheme, and
obtains comparable performance to the state-of-the-art STR
methods.

Text Recognition in Real Scenarios with a Few Labeled Samples

With the rise of the so-called Adversarial Attacks, there
is an increased concern on model security. In this paper we
present two different contributions: novel measures of
robustness (based on adversarial attacks) and a novel
adversarial attack. The key idea behind these metrics is to
obtain a measure that could compare different
architectures, with independence of how the input is
preprocessed (robustness against different input sizes and
value ranges). To do so, a novel adversarial attack is
presented, performing a delayed elastic-net adversarial
attack (constraints are only used whenever a successful
adversarial attack is obtained). Experimental results show
that our approach obtains state-of-the-art adversarial
samples, in terms of minimal perturbation distance.
Finally, a benchmark of ImageNet pretrained models is used
to conduct experiments aiming to shed some light about
which model should be selected whenever security is a role
factor.

A Delayed Elastic-Net Approach for Performing Adversarial Attacks

Along with the performance improvements of neural network models, developing methods that enable the explanation of their behavior is a significant research topic. 
For convolutional neural networks, the explainability is usually achieved with attribution (heatmap) that visualizes pixel-level importance or contribution of input to its corresponding result. 
This attribution should reflect the relation (dependency) between inputs and outputs, which has been studied with a variety of methods, e.g., derivative of an output with respect to an input pixel value, a weighted sum of gradients, amount of output changes to input perturbations, and so on. 
In this paper, we present a new method that improves the measure of attribution and incorporates it into the integrated gradients method. To be precise, rather than using the conventional chain-rule, we propose a method called guided non-linearity that propagates gradients more effectively through non-linear units (e.g., ReLU and max-pool) so that only positive gradients backpropagate through non-linear units. 
Our method is inspired by the mechanism of action potential generation in postsynaptic neurons, where the firing of action potentials depends on the sum of excitatory (EPSP) and inhibitory postsynaptic potentials (IPSP). 
We believe that paths consisting of EPSP-giving-neurons faithfully reflect the contribution of inputs to the output, and we make gradients flow only along those paths (i.e., paths of positive chain reactions). Experiments with 5 deep neural networks have shown that the proposed method outperforms others in terms of the deletion metrics, and yields fine-grained and more human-interpretable attribution.

Improving Explainability of Integrated Gradients with Guided Non-Linearity

Traditionally, a Neural Network classification training loss function follows the same principle: minimizing the distance between samples that belong to the same class, while maximizing the distance to the other classes. There are no restrictions on the spatial placement of deep features (last layer input). This paper addresses this issue when dealing with Neural Networks, providing a set of loss functions that are able to train a classifier by forcing the deep features to be projected over a predefined orthogonal basis. Experimental results shows that these `data placement' functions can overcome the training accuracy provided by the classic cross-entropy loss function.

Premium content

Hierarchical Mixtures of Generators for Adversarial Learning

Downloads

Next from ICPR 2020

A Scalable Deep Neural Network to Detect Low Quality Images Without a Reference

Similar lecture

Hierarchical Mixtures of Generators for Adversarial Learning