Italy

Current compressed video action recognition methods are mainly based on complete data. However, in a real transmission scenario, the compressed video packets are usually disorderly received and even lost due to network jitters or
congestion. To recognize actions in early phases with limited packets, e.g. for quickly forecasting possible potential risks, in this paper, we propose a Temporal Enhanced Multi-Stream Network (TEMSN) towards practical compressed video action
recognition. First, we make use of three modalities in the compressed domain as complementary cues and build a multistream network to capture rich information from compressed video packets. Second, we design a temporal enhanced module
based on an Encoder-Decoder structure, which is applied to each stream to infer missing packets, generating more accurate action dynamics. Thanks to the multiple modalities and their temporal enhancement, our approach better models actions with partial available compressed video packets. Experiments on the HMDB-51 and UCF-101 datasets validate its effectiveness and efficiency.

ICPR 2020

Towards Practical Compressed Video Action Recognition: A
Temporal Enhanced Multi-Stream Network

compressed video

action recognition

video understanding

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


poster


This paper strives for action recognition and detection in video modalities like RGB, depth maps or 3D-skeleton sequences when only limited modality-specific labeled examples
are available. For the RGB, and derived optical-flow, modality
many large-scale labeled datasets have been made available. They
have become the de facto pre-training choice when recognizing
or detecting new actions from RGB datasets that have limited
amounts of labeled examples available. Unfortunately, large-scale
labeled action datasets for other modalities are unavailable for
pre-training. In this paper, our goal is to recognize actions from
limited examples in non-RGB video modalities, by learning from
large-scale labeled RGB data. To this end, we propose a two-step
training process: (i) we extract action representation knowledge
from an RGB-trained teacher network and adapt it to a nonRGB student network. (ii) we then fine-tune the transfer model
with available labeled examples of the target modality. For the
knowledge transfer we introduce feature-supervision strategies,
which rely on unlabeled pairs of two modalities (the RGB and
the target modality) to transfer feature level representations from
the teacher to the student network. Ablations and generalizations
with two RGB source datasets and two non-RGB target datasets
demonstrate that an optical-flow teacher provides better action
transfer features than RGB for both depth maps and 3Dskeletons, even when evaluated on a different target domain, or
for a different task. Compared to alternative cross-modal action
transfer methods we show a good improvement in performance
especially when labeled non-RGB examples to learn from are
scarce.

Feature-Supervised Action Modality Transfer

Feature point matching is an important problem because its applications cover a wide range of tasks in computer vision. Deep learning-based methods for learning local features have recently shown superior performance. However, it is not easy to collect the training data in these methods, especially in cross-spectral settings such as the correspondence between RGB and near-infrared images. In this paper, we propose an unsupervised learning method for general feature point matching. Because we train a convolutional neural network as a feature extractor in order to satisfy the cycle consistency of the correspondences between an input image pair, the proposed method does not require supervision and works even in cross-spectral settings. In our experiments, we apply the proposed method to stereo matching, which is a dense feature point matching problem. The experimental results, which simulate cross-spectral settings with three different settings, i.e., RGB stereo, RGB vs gray-scale, and anaglyph (red vs cyan), show that our proposed method outperforms the compared methods, which employ handcrafted features for stereo matching, by a significant margin.

Feature Point Matching in Cross-Spectral Images with Cycle Consistency Learning

Generative Adversarial Network (GAN) has been widely used for object attribute editing. However, the semantic correlation, resulted from the feature map interaction in the generative network of GAN, may impair the generalization ability of the generative network. In this work, semantic disentanglement is introduced in GAN to reduce the attribute correlation. The feature maps of the generative network are first grouped with an efficient clustering algorithm based on hash encoding, which are used to excavate hidden semantic attributes and calculate
the group-wise orthogonality loss for the reduction of attribute entanglement.
Meanwhile, the feature maps falling in the intersection regions of different groups are further suppressed to reduce the attribute-wise interaction.
Extensive experiments reveal that the proposed GAN generated more genuine objects than the state of the arts. Quantitative results of classification accuracy, inception score and FID score further justify the effectiveness of the proposed GAN.

Group-wise Feature Orthogonalization and Suppression for GAN based Facial Attribute Translation



Malware classification helps to understand its purpose and is also an important part of attack detection. And it is also an important part of discovering attacks. Due to
continuous innovation and development of artificial intelligence, it is a trend to combine deep learning with malware classification. In this paper, we propose an improved malware image rescaling algorithm (IMIR) based on local mean algorithm. Its main goal of IMIR is to reduce the loss of information from samples during
the process of converting binary files to image files. Therefore, we construct a neural network structure based on VGG model, which is suitable for image classification. In the real world, a mass of malware family labels are inaccurate or lacking. To deal with this situation, we propose a novel method to train the deep neural network by Semi-supervised Generative Adversarial Network (SGAN), which only needs a small amount of malware that have accurate labels about families. By integrating SGAN
with weak coupling, we can retain the weak links of supervised part and unsupervised part of SGAN. It improves the accuracy of malware classification by making classifiers more independent of discriminators. The results of experimental demonstrate that our model achieves exhibiting favorable performance. The recalls
of each family in our data set are all higher than 93.75%.

A Weak Coupling of Semi-Supervised Learning with Generative Adversarial Networks for Malware Classification


Object detection in Earth Vision has achieved great progress in recent years. However, tiny object detection in aerial images remains a very challenging problem since the tiny objects contain a small number of pixels and are easily confused with
the background. To advance tiny object detection research in aerial images, we present a new dataset for Tiny Object Detection in Aerial Images (AI-TOD). Specifically, AI-TOD comes with 700,621 object instances for eight categories across 28,036 aerial images. Compared to existing object detection datasets in aerial images, the mean size of objects in AI-TOD is about 12.8 pixels, which is much smaller than others. To build a benchmark for tiny object detection in aerial images, we evaluate the state-of-the-art object detectors on our AI-TOD dataset. Experimental results show that direct application of these approaches on AI-TOD
produces suboptimal object detection results, thus new specialized detectors for tiny object detection need to be designed. Therefore, we propose a multiple center points based learning network (M-CenterNet) to improve the localization performance of tiny object detection, and experimental results show the significant
performance gain over the competitors.

Tiny Object Detection in Aerial Images

Skeleton-based action recognition task has drawn
much attention for many years. Graph Convolutional Network
(GCN) has proved its effectiveness in this task. However, how to
improve the model’s robustness to different human actions and
how to make effective use of features produced by the network
are main topics needed to be further explored. Human actions
are time series sequence, meaning that temporal information is
a key factor to model the representation of data. The ranges
of body parts involved in small actions (e.g. raise a glass or
shake head) and big actions (e.g. walking or jumping) are
diverse. It’s crucial for the model to generate and utilize more
features that can be adaptive to a wider range of actions.
Furthermore, feature channels are specific with the action class,
the model needs to weigh their importance and pay attention
to more related ones. To address these problems, in this work,
we propose a two-stream channel-wise dense connection GCN
(2s-CDGCN). Specifically, the skeleton data was extracted and
processed into spatial and temporal information for better feature
representation. A channel-wise attention module was used to
select and emphasize the more useful features generated by
the network. Moreover, to ensure maximum information flow,
dense connection was introduced to the network structure,
which enables the network to reuse the skeleton features and
generate more information adaptive and related to different
human actions. Our model has shown its ability to improve the
accuracy of human action recognition task on two large datasets,
NTU-RGB+D and Kinetics. Extensive evaluations were conducted
to prove the effectiveness of our model.

Channel-Wise Dense Connection Graph Convolutional Network For Skeleton-Based Action Recognition

Foreground segmentation in the dynamic scenes is an important task in video surveillance. The unsupervised background subtraction method based on
background statistics modeling has difficulties in updating. On the other hand, the supervised foreground segmentation method based on deep learning relies on the large-scale of accurately annotated training data, which limits its cross-scene performance. In this paper, we propose a foreground segmentation method from coarse to fine. First, an across-scenes trained Spatio-Temporal Attention Model (STAM) is used to achieve coarse segmentation, which does not require training on a specific scene. Then the coarse segmentation is used as a reference to help Co-occurrence Pixel-Block Model (CPB) complete the fine segmentation and at the same time help CPB to update its background model. This method is more flexible than those deep learning-based methods which depend on the specific scene training and realizes the accurate online dynamic update of the background model. Experimental results on WallFlower and LIMU validate our method outperforms STAM, CPB, and other methods of participating in comparison.

Coarse-to-fine Foreground Segmentation based on Co-occurrence Pixel-Block and Spatio-Temporal Attention Model

Beyond assigning the correct class, an activity recognition model should also be able to determine, how certain it is in its predictions. We present the first study of how well the confidence values of modern action recognition architectures indeed reflect the probability of the correct outcome and propose a learning-based approach for improving it. First, we extend two popular action recognition datasets with a reliability benchmark in form of the expected calibration error and reliability diagrams. Since our evaluation highlights that confidence values of standard action recognition architectures do not represent the uncertainty well, we introduce a new approach which learns to transform the model output into realistic confidence estimates through an additional calibration network. The main idea of our Calibrated Action Recognition with Input Guidance (CARING) model is to learn an optimal scaling parameter depending on the video representation. We compare our model with the native action recognition networks and the temperature scaling approach - a wide spread calibration method utilized in image classification. While temperature scaling alone drastically improves the reliability of the confidence values, our CARING method consistently leads to the best uncertainty estimates in all benchmark settings.

Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models


Spatio-temporal feature descriptors are of great
importance for characterizing the local changes of 3D deformable
shapes. In this study, we propose a method for characterizing
3D shapes from point clouds and we show a direct application
on a study of organ temporal deformations. As an example,
we characterize the behavior of the bladder during forced
respiratory motion with a reduced number of 3D surface points:
first, a set of geodesically equidistant points representing the
vertices of quadrilateral mesh for the organ surface are tracked
throughout a long dynamic MRI sequence using a large deformation diffeomorphic metric mapping (LDDMM) framework.
Second, a novel 3D shape descriptor invariant to translation,
scale and rotation is proposed for characterizing the temporal
organ deformations by employing an Eulerian Partial Differential
Equations (PDEs) methodology. We demonstrate the robustness
of our feature on both synthetic 3D shapes and realistic dynamic
Magnetic Resonance Imaging (MRI) data sequences portraying
the bladder deformation during a forced breathing exercise.
Promising results are obtained, showing that the proposed feature
may be useful for several computer vision applications such as
medical imaging, aerodynamics and robotics.

A new geodesic-based feature for characterization of 3D shapes

Supervised deep learning methods have recently
achieved remarkable performance in person re-identification.
Unsupervised domain adaptation (UDA) approaches have also
been proposed for application scenarios where only unlabelled
data are available from target camera views. We consider a
more challenging scenario when even collecting a suitable amount
of representative, unlabelled target data for offline training or
fine-tuning is infeasible. In this context we revisit the human-in-
the-loop (HITL) approach, which exploits online the operator’s
feedback on a small amount of target data. We argue that HITL
is a kind of online domain adaptation specifically suited to person
re-identification. We then reconsider relevance feedback methods
for content-based image retrieval that are computationally much
cheaper than state-of-the-art HITL methods for person re-
identification, and devise a specific feedback protocol for them.
Experimental results show that HITL can achieve comparable or
better performance than UDA, and is therefore a valid alternative
when the lack of unlabelled target data makes UDA infeasible.

Premium content

Towards Practical Compressed Video Action Recognition: A Temporal Enhanced Multi-Stream Network

Downloads

Next from ICPR 2020

Feature-Supervised Action Modality Transfer

Similar lecture

Learning Group Activities from Skeletons without Individual Action Labels