Italy


Matching information across image and text modalities is a fundamental challenge for many applications that involve both vision and natural language processing. The objective is to find efficient similarity metrics to compare the similarity between visual and textual information. Existing approaches mainly match the local visual objects and the sentence words in a shared space with attention mechanisms. The matching performance is still limited because the similarity computation is based on simple comparisons of the matching features, ignoring the characteristics of their distribution in the data. In this paper, we address this limitation with an efficient learning objective that considers the \textit{discriminative feature distributions} between the visual objects and sentence words. Specifically, we propose a novel \textit{Adversarial Discriminative Domain Regularization} (ADDR) learning framework, beyond the paradigm metric learning objective, to construct a set of discriminative data domains within each image-text pairs. Our approach can generally improve the learning efficiency and the performance of existing metrics learning frameworks by regulating the distribution of the hidden space between the matching pairs. The experimental results show that this new approach significantly improves the overall performance of several popular cross-modal matching techniques on the MS-COCO and Flickr30K benchmarks.

ICPR 2020

Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization

vision and language; cross-modal matching; metric learning

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


poster


Image based analysis of plants is a high-throughput and non-invasive approach to study plant traits. The quantitative estimation of many plant traits (leaf area index, biomass etc.) from plant images is primarily based on accurate segmentation of individual leaves. This is a challenging task due to the presence of overlapped leaves and lack of discernible boundaries between them. To overcome these limitations, state-of-the-art supervised deep learning algorithms have been recently employed. However, the annotations of individual leaf instances is time consuming, in addition the variability in leaf shapes and its arrangement among different plant species limits the broad utilisation of these algorithms. We propose a novel framework that relies on a graph based formulation to extract leaf shape knowledge for the task of leaf instance segmentation. These shape priors are generated based on leaf shape characteristics independent of plant species. Thus, relieving the current bottleneck of annotated data.

A Hierarchical Framework for Leaf Instance Segmentation: Application to Plant Phenotyping


This paper studies the problem of Person Re-Identification (ReID)
for large-scale applications. Recent research efforts have been devoted
to building complicated part models, which introduce considerably
high computational cost and memory consumption, inhibiting its practicability
in large-scale applications. This paper aims to develop a novel learning
strategy to find efficient feature embeddings while maintaining the
balance of accuracy and model complexity. More specifically, we find
by enhancing the classical triplet loss together with cross-entropy
loss, our method can explore the hard examples and build a discriminant
feature embedding yet compact enough for large-scale applications.
Our method is carried out progressively using Bayesian optimization,
and we call it the Progressive Learning Algorithm (PLA). Extensive
experiments on three large-scale datasets show that our PLA is comparable
or better than the state-of-the-arts. Especially, on the challenging
Market-1501 dataset, we achieve Rank-1=94.7%/mAP=89.4% while saving
at least 30\% parameters than strong part models.

Directionally Paired Principal Component Analysis for Bivariate Estimation Problems


We present a multimodal deep learning framework that can generate summarization text supporting the main idea of an information graphic for presentation to a person who is blind or visually impaired. The framework utilizes the visual, textual, positional, and size characteristics extracted from the image to create the summary. Different and complimentary neural architectures are optimized for each task using crowdsourced training data. From our quantitative experiments and results, we explain the reasoning behind our framework and show the effectiveness of our models. Our qualitative results showcase text generated from our framework and show that Mechanical Turk participants favor them to other automatic and human generated summarizations.
We describe the design and results of an experiment to evaluate the utility of our system for people who have visual impairments in the context of understanding Twitter Tweets containing line graphs.

Information Graphic Summarization using a Collection of Multimodal Deep Neural Networks

Learning musical instruments like piano or electronic keyboard on average takes a decade. Currently, learning an instrument requires continuous supervision from a tutor, and it often becomes unrealistic to stay connected with the music tutor for a long time. Online music learning platforms remain unscalable as there is no way for these platforms to verify whether user pressed a key note with the intended finger, which is significant for learning finger based musical instruments. To overcome this, an autonomous system to evaluate and guide in the learning process by continuously tracking finger movements via a non-camera based solution is proposed. Finger press triggers the muscle movements which are detected at the surface of
the forearm in the form of surface Electromyography (sEMG) signals. The paper proposes tracking of finger press for 10seconds on an electronic keyboard using MyoBand [1] wearable device that provided 8 channels of sEMG signals. Eleven time and frequency domain features of sEMG signals were extracted from sEMG signals to classify musical note played by the instrument on corresponding finger press. The feature set was standardized using standard scaler approach, and vector dimensions were reduced by Linear Discriminant Analysis (LDA) method. The
resulting reduced dimension data was applied on Random Forest (RF) classifier to report best classification accuracy for our application. Experiments involved single finger press to render a note in the musical instrument, and multiple finger press to define chord sequence on an electronic musical keyboard. Further analysis was performed to maximize the classification accuracy over the number of trials and optimize the position of electrodes for successful identification of musical note played. The proposed method achieves a classification accuracy of 74.25%
for 5 musical note played on an electronic keyboard instrument with 4 MyoBand electrodes, and an accuracy of 95.83% with one electrode for identifying between four musical events including two major chords sequence and two musical notes.


Feasibility Study of using MyoBand for Learning Electronic Keyboard

Artificial intelligence for autonomous driving must meet strict requirements on safety and robustness, which motivates the thorough validation of learned models. However, current validation approaches mostly require ground truth data and are thus both cost-intensive and limited in their applicability. We propose to overcome these limitations by a model agnostic validation using a-priori knowledge from street maps. In particular, we show how to validate semantic segmentation masks and demonstrate the potential of our approach using OpenStreetMap. We introduce validation metrics that indicate false positive or negative road segments. Besides the validation approach, we present a method to correct the vehicle's GPS position so that a more accurate localization can be used for the street-map based validation. Lastly, we present quantitative results on the Cityscapes dataset indicating that our validation approach can indeed uncover errors in semantic segmentation masks.

Street-Map Based Validation of Semantic Segmentation in Autonomous Driving

This paper aims to develop a deep neural network for Continuous Sign Language Recognition (CSLR) with iterative Gloss Recognition (GR) fine-tuning. CSLR has been a popular research field in the last years and iterative optimization methods are well established. This paper introduces our proposed architecture involving Spatiotemporal feature-extraction model to segment useful ``gloss-unit" features and BiLSTM with CTC as a sequence model. Spatiotemporal Feature Extractor is used for both image features extraction and sequence length reduction. To this end, we compare different architectures for feature extraction and sequence model. In addition, we iteratively fine-tune feature extractor on gloss-unit video segments with alignments from the end2end model. During the iterative training, we use novel alignment correction technique, which is based on minimum transformations of Levenshtein distance. All the experiments were conducted on the RWTH-PHOENIX-Weather-2014 dataset.

Continuous Sign Language Recognition withIterative Spatiotemporal Fine-tuning

There has been growing interest recently in using machine learning techniques as an aid in clinical medicine. Machine learning offers a range of classification algorithms which can be applied to medical data to aid in making clinical predictions. Recent studies have demonstrated the high predictive accuracy of various classification algorithms applied to clinical data. Several studies have already been conducted in diagnosing or predicting chronic kidney disease at various stages using different sets of variables. In this study we are investigating the use of machine learning techniques with blood test data. Such a system could aid renal teams in making recommendations to primary care general practitioners to refer patients to secondary care where patients may benefit from earlier specialist assessment and medical intervention. We are able to achieve an overall accuracy of 88.48% using logistic regression, 87.12% using ANN and 85.29% using SVM. ANNs performed with the highest sensitivity at 89.74% compared to 86.67% for logistic regression and 85.51% for SVM.

Using Machine Learning to Refer Patients with Chronic Kidney Disease to Secondary Care

3D points distribute in a continuous 3D space irregularly, thus directly adapting 2D image convolution to 3D points is not an easy job. Our model can naturally generalizes image CNN to deal with point clouds. The key idea of our model is learning to probe the 3D space in an adaptive manner. Experiments across three tasks demonstrate that our model achieves the state-of-the-art or competitive performance.

Deep Space Probing for Point Cloud Analysis


In the field of multimodal segmentation, the correlation between different modalities can be considered for improving the segmentation results. In this paper, we propose a multi-modality segmentation network with a correlation constraint. Our network includes N model-independent encoding paths with N image sources, a correlation constraint block, a feature fusion block, and a decoding path. The model independent encoding path can capture modality-specific features from the N modalities. Since there exists a strong correlation between different modalities, we first propose a linear correlation block to learn the correlation between modalities, then a loss function is used to guide the network to learn the correlated features based on the linear correlation block. This block forces the network to learn the latent correlated features which are more relevant for segmentation. Considering that not all the features extracted from the encoders are useful for segmentation, we propose to use dual attention based fusion block to recalibrate the features along the modality and spatial paths, which can suppress less informative features and emphasize the useful ones. The fused feature representation is finally projected by the decoder to obtain the segmentation result. Our experiment results tested on BraTS-2018 dataset for brain tumor segmentation demonstrate the effectiveness of our proposed method.

3D Medical Multi-modal Segmentation Network Guided by Multi-source Correlation Constraint

For full process automation in Industry 4.0 with smart machines, localizing objects is a crucial capability. Enabling such an object localization with Machine Learning is tempting, as it avoids a manual engineering of features for every object class, yet gathering large amounts of accurate pose data to train neural networks can be just as difficult.

In this work, a novel self-supervised approach based on point clouds is presented, which resolves these issues and allows a robust detection and localization of objects in cluttered scenes, while compensating for noise, occlusions and symmetries. It leverages 3D sensor data simulation being simpler than light rendering to construct random scenes using a stochastic process by maintaining object relations with placement operations like cloning, stacking and storing, and to then train a fully convolutional voting network with random scans from those scenes.

Premium content

Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization

Downloads

Next from ICPR 2020

A Hierarchical Framework for Leaf Instance Segmentation: Application to Plant Phenotyping

Similar lecture

Segmenting Messy Text: Detecting Boundaries in Text Derived from Historical Newspaper Images