Italy

Long-term visual tracking is a challenging problem.
State-of-the-art long-term trackers, e.g., GlobalTrack,
utilize region proposal networks (RPNs) to generate target
proposals. However, the performance of the trackers is
affected by occlusions and large scale or ratio variations.
To address these issues, in this paper, we are the first to
propose a novel architecture with transformers for
long-term visual tracking. Specifically, the proposed
Visual Tracking Transformer (VTT) utilizes a transformer
encoder-decoder architecture for aggregating global
information to deal with occlusion and large scale or ratio
variation. Furthermore, it also shows better discriminative
power against instance-level distractors without the need
for extra labeling and hard-sample mining. We conduct
extensive experiments on three largest long-term tracking
dataset and have achieved state-of-the-art performance.

ICPR 2020

VTT: Long-term Visual Tracking with Transformers

motion and tracking; deep learning; video analysis

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


technical paper

Despite their tremendous success in modelling high-dimensional data manifolds, deep neural networks suffer from the threat of adversarial attacks - Existence of perceptually valid input-like samples obtained through careful perturbation that lead to degradation in the performance of the underlying model. Major concerns with existing defense mechanisms includenon-generalizability across different attacks, models and largeinference time. In this paper, we propose a generalized defense mechanism capitalizing on the expressive power of regularized latent space based generative models. We design an adversarial filter, devoid of access to classifier and adversaries, which makes it usable in tandem with any classifier. The basic idea is to learn a Lipschitz constrained mapping from the data manifold, incorporating adversarial perturbations, to a quantized latent space and re-map it to the true data manifold. Specifically, wesimultaneously auto-encode the data manifold and its pertur-bations implicitly through the perturbations of the regularizedand quantized generative latent space, realized using variationalinference. We demonstrate the efficacy of the proposed formula-tion in providing resilience against multiple attack types (blackand white box) and methods, while being almost real-time. Ourexperiments show that the proposed method surpasses the state-of-the-art techniques in several cases. The implementation codeis available at - https://github.com/mayank31398/lqvae.

Variational Inference with Latent Space Quantization for Adversarial Resilience


We introduce an adaptive L2 regularization mechanism in the setting of person re-identification. In the literature, it is common practice to utilize hand-picked regularization factors which remain constant throughout the training procedure. Unlike existing approaches, the regularization factors in our proposed method are updated adaptively through backpropagation. This is achieved by incorporating trainable scalar variables as the regularization factors, which are further fed into a scaled hard sigmoid function. Extensive experiments on the Market-1501, DukeMTMC-reID and MSMT17 datasets validate the effectiveness of our framework. Most notably, we obtain state-of-the-art performance on MSMT17, which is the largest dataset for person re-identification. Source code is publicly available at https://github.com/nixingyang/AdaptiveL2Regularization.

Adaptive L2 Regularization in Person Re-Identification


In zero-shot cross-lingual transfer, a supervised NLP task trained on a corpus in one language is directly applicable to another language without any additional training. A source of cross-lingual transfer can be as straightforward as lexical overlap between languages (e.g., use of the same scripts, shared subwords) that naturally forces text embeddings to occupy a similar representation space. Recently introduced cross-lingual language model (XLM) pretraining brings out neural parameter sharing in Transformer-style networks as the most important factor for the transfer. In this paper, we aim to validate the hypothetically strong cross-lingual transfer properties induced by XLM pretraining. Particularly, we take XLM-RoBERTa (XLMR) in our experiments that extend semantic textual similarity (STS), SQuAD and KorQuAD for machine reading comprehension, sentiment analysis, and alignment of sentence embeddings under various cross-lingual settings. Our results indicate that the presence of cross-lingual transfer is most pronounced in STS, sentiment analysis the next, and MRC the last. That is, the complexity of a downstream task softens the degree of cross-lingual transfer. All of our results are empirically observed and
measured, and we make our code and data publicly available.

Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks

We propose a framework to analyze vehicles behavior within shared road segments like intersections or at narrow passages. 
The system first identifies critical interaction regions based on topological knowledge. 
It then checks possible colliding trajectories from the current state of vehicles in the scene, defined by overlapping occupation times in road segments. 
The interaction between vehicles is analyzed over a short time horizon based on an initial action from one vehicle and the reaction by the other. 
The observed reaction is classified by attention and the collaboration of the opponent vehicle, whether it helps to resolve the situation or hinders it.
The output is a classification of behavior terms of active participation in the interaction and driving style in terms of collaborative or disruptive behavior.

Visual Prediction of Driver Behavior in Shared Road Areas


The use of administrative documents to communicate and leave record of business information requires of methods able to automatically extract and understand the content from such documents in a robust and efficient way. In addition, the semi-structured nature of these reports is specially suited for the use of graph-based representations which are flexible enough to adapt to the deformations from the different document templates. Moreover, Graph Neural Networks provide the proper methodology to learn relations among the data elements in these documents. 
In this work we study the use of Graph Neural Network architectures to tackle the problem of entity recognition and relation extraction in semi-structured documents. Our approach achieves state of the art results in the three tasks involved in the process. Additionally, the experimentation with two datasets of different nature demonstrates the good generalization ability of our approach.

Named Entity Recognition and Relation Extraction with Graph Neural Networks in Semi Structured Documents

This paper presents a weakly supervised object detection method based on activity label and relationship modeling, which is motivated by the assumption that configuration of human and object are similar in same activity, and joint modeling of human, active object and activity could leverage the recognition of them. Compared to most weakly supervised method taking object as independent instance, firstly, active human and object proposals are learned and filtered based on class activation map of multi-label classification. Secondly, a spatial relationship prior including relative position, scale, overlaps etc are learned dependent on action. Finally, a multistream object detection framework integrating the spatial prior and pairwise ROI pooling are proposed to jointly learn the object and action class. Experiments are conducted on HICODET dataset, and our approach outperforms the state of the art weakly supervised object detection methods.

Activity and Relationship Modeling Driven Weakly Supervised Object Detection

In this presentation, we introduce an interpretable non-linear model called itemset Sparse Bayes (iSB), which builds a Bayesian probabilistic model, while simultaneously considering variable interactions. In order to suppress the resulting large number of variables, sparsity is imposed on regression weights by a sparsity inducing prior. As a subroutine to search for variable interactions, itemset enumeration algorithm is employed with a novel bounding condition.
In computational experiments using real-world dataset, the proposed method performed better than decision tree by 10% in terms of r2. In addition, ISB gives us a clue to under- stand which variables interactions are important in optimizing an unknown function.

Automatically mining relevant variable interactions via sparse Bayesian learning

This paper addresses the task of classifying galaxy clusters, which are the largest known objects in the Universe. Galaxy clusters can be categorized as cool-core (CC), weak- cool-core (WCC), and non-cool-core (NCC), depending on their central cooling times. Traditional classification approaches used in astrophysics are inaccurate and rely on measuring surface brightness concentrations or central gas densities. In this work, we propose a multi-branch attention network that uses spatial attention to classify a given cluster. To evaluate our network, we use a database of simulated X-ray emissivity images, which contains 954 projections of 318 clusters. Experimental results show that our network outperforms several strong baseline methods and achieves a macro-averaged F1 score of 0.83. We highlight the value of our proposed spatial attention module through an ablation study.

Multi-Branch Attention Networks for Classifying Galaxy Clusters


Recent co-part segmentation methods mostly operate in a supervised learning setting, which requires a large amount of annotated data for training. To overcome this limitation, we propose a self-supervised deep learning method for co-part segmentation. Differently from previous works, our approach develops the idea that motion information inferred from videos can be leveraged to discover meaningful object parts. To this end, our method relies on pairs of frames sampled from the same video. The network learns to predict part segments together with a representation of the motion between two frames, which permits reconstruction of the target image. Through extensive experimental evaluation on publicly available video sequences we demonstrate that our approach can produce improved segmentation maps with respect to previous self-supervised co-part segmentation approaches.

Motion supervised co-part segmentation

The target of human pose estimation is to determine the body parts and joint locations of persons in the image. Angular changes, motion blur and occlusion in the natural scenes make this task challenging, while some joints are more difficult to be detected than others. In this paper, we propose an augmented Parallel-Pyramid Net (P2Net) with feature refinement by dilated bottleneck and attention module. During data preprocessing, we proposed a differentiable auto data augmentation (DA2) method. We formulate the problem of searching data augmentaion policy in a differentiable form, so that the optimal policy setting can be easily updated by back propagation during training. DA2 improves the training efficiency. A parallel-pyramid structure is followed to compensate the information loss introduced by the network. We innovate two fusion structures, i.e. Parallel Fusion and Progressive Fusion, to process pyramid features from backbone network. Both fusion structures leverage the advantages of spatial information affluence at high resolution and semantic comprehension at low resolution effectively. We propose a refinement stage for the pyramid features to further boost the accuracy of our network. By introducing dilated bottleneck and attention module, we increase the receptive field for the features with limited complexity and tune the importance to different feature channels. To further refine the feature maps after completion of feature extraction stage, an Attention Module (AM) is defined to extract weighted features from different scale feature maps generated by the parallel-pyramid structure. Compared with the traditional up-sampling refining, AM can better capture the relationship between channels. Experiments corroborate the effectiveness of our proposed method. Notably, our method achieves the best performance on the challenging MSCOCO and MPII datasets.

Premium content

VTT: Long-term Visual Tracking with Transformers

Downloads

Next from ICPR 2020

Variational Inference with Latent Space Quantization for Adversarial Resilience

Similar lecture

VTT: Long-term Visual Tracking with Transformers