Italy

We present a multimodal deep learning framework that can generate summarization text supporting the main idea of an information graphic for presentation to a person who is blind or visually impaired. The framework utilizes the visual, textual, positional, and size characteristics extracted from the image to create the summary. Different and complimentary neural architectures are optimized for each task using crowdsourced training data. From our quantitative experiments and results, we explain the reasoning behind our framework and show the effectiveness of our models. Our qualitative results showcase text generated from our framework and show that Mechanical Turk participants favor them to other automatic and human generated summarizations.
We describe the design and results of an experiment to evaluate the utility of our system for people who have visual impairments in the context of understanding Twitter Tweets containing line graphs.

ICPR 2020

Information Graphic Summarization using a Collection of Multimodal Deep Neural Networks

information graphic

multimodal

summarization

deep learning

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


poster

Though deep learning based scene text detection has achieved great progress, well-trained detectors suffer from severe performance degradation for different domains. In general, a tremendous amount of data is indispensable to train the detector in the target domain. However, data collection and annotation are expensive and time-consuming. To address this problem, we propose a self-training framework to automatically mine hard examples with pseudo-labels from unannotated videos or images. To reduce the noise of hard examples, a novel text mining module is implemented based on the fusion of detection and tracking results. Then, an image-to-video generation method is designed for the tasks that videos are unavailable and only images can be used. Experimental results on standard benchmarks, including ICDAR2015, MSRA-TD500, ICDAR2017 MLT, demonstrate the effectiveness of our self-training method. The simple Mask RCNN adapted with self-training and fine-tuned on real data can achieve comparable or even superior results with the state-of-the-art methods.

Self-Training for Domain Adaptive Scene Text Detection

In this paper, we propose a simple and effective framework
for detecting arbitrary shaped scene text, which detects text
regions and kernels of text-like regions respectively. Moreover,
we introduce a decay learning weight to compress the influence
of the unclear boundary of text instances. The experiments have
verified that our method consistently achieves the satisfying
results on four standard scene text benchmarks, in terms of
accuracy and speed. In the future, we will work on improving
its ability to detect small text instances and extending our
method for end-to-end text spotting.

An Accurate Threshold Insensitive Kernel Detector
for Arbitrary Shaped Text

Gaining control over the dynamics of multi-task learning should help to unlock the potential of the deep network to a great extent. In the existing multi-task learning (MTL) approaches of a deep network, all the parameters of its feature encoding part are subjected to adjustments corresponding to each of the underlying sub-tasks. On the other hand, different functional areas of the human brain are responsible for distinct functions such as the Broca's area of the cerebrum is responsible for speech formation whereas its Wernicke's area is related to the language development etc. Inspired by this fact, in the present study, we used here a Feature Representation Block (FRB) of connection weights spanned over a few successive layers of a deep multi-task learning architecture and stratify the same into distinct subsets for their adjustments exclusively corresponding to different sub-tasks. Additionally, we have introduced a novel regularization component for controlled training of this Feature Representation Block. The purpose of the development of this learning framework is efficient end-to-end recognition of scene texts.

Stratified Multi-Task Learning for Robust Spotting of Scene Texts

Localizing page elements/objects such as tables, figures, equations, etc. is the primary step in extracting information from document images. We propose a novel end-to-end trainable deep network, (CDeC-Net) for detecting tables present in the documents. The proposed network consists of a multistage extension of Mask R-CNN with a dual backbone having deformable convolution for detecting tables varying in scale with high detection accuracy at higher IoU threshold. We empirically evaluate CDeC-Net on the publicly available benchmark datasets with extensive experiments.
Our solution has three important properties: (i) a single trained model CDeC-Net that performs well across all the popular benchmark datasets; (ii) we report excellent performances across multiple, including higher, thresholds of IoU; (iii) by following the same protocol of the recent papers for each of the benchmarks, we consistently demonstrate the superior quantitative performance. Our code and models are publicly available at https://github.com/mdv3101/CDeCNet for enabling reproducibility of the results.

CDeC-Net: Composite Deformable Cascade Network for Table Detection in Document Images

The use of administrative documents to communicate and leave record of business information requires of methods able to automatically extract and understand the content from such documents in a robust and efficient way. In addition, the semi-structured nature of these reports is specially suited for the use of graph-based representations which are flexible enough to adapt to the deformations from the different document templates. Moreover, Graph Neural Networks provide the proper methodology to learn relations among the data elements in these documents. 
In this work we study the use of Graph Neural Network architectures to tackle the problem of entity recognition and relation extraction in semi-structured documents. Our approach achieves state of the art results in the three tasks involved in the process. Additionally, the experimentation with two datasets of different nature demonstrates the good generalization ability of our approach.

Named Entity Recognition and Relation Extraction with Graph Neural Networks in Semi Structured Documents

Scene text detection has been studied extensively. Existing methods detect either words or text lines and use either word-level or line-level annotated data for training. In this work, we propose a dual-task network that can perform word-level and line-level text detection simultaneously and use training data of both levels of annotation to boost the performance. The dual-task network has two detection heads for word-level and line-level text detection, respectively. Then we propose a mutual guidance scheme for the joint training of the two tasks with two modules: line filtering module utilizes the output feature map of the text line detector to filter out the non-text regions for the word detector, and word enhancing module provides prior positions of words for the text line detector depending on the output feature map of the word detector. Experimental results of word-level and linelevel text detection demonstrate the effectiveness of the proposed dual-task network and mutual guidance scheme, and the results of our method are competitive with state-of-the-art methods.

Mutually Guided Dual-Task Network for Scene Text Detection

Scene text detection (STD) aims to locate text in
images and plays an important role in many computer vision
tasks including automatic driving and text recognition systems.
Recently, deep neural networks (DNNs) have been widely and
successfully used in scene text detection, leading to plenty
of DNN-based STD methods including regression-based and
segmentation-based STD methods. However, recent studies have
also shown that DNN is vulnerable to adversarial attacks, which
can significantly degrade the performance of DNN models. In
this paper, we investigate the robustness of DNN-based STD
methods against adversarial attacks. To this end, we propose
a generic and efficient attack method to generate adversarial
examples, which are produced by adding small but imperceptible
adversarial perturbation to the input images. Experiments on
attacking four various models and a real-world STD engine of
Google optical character recognition (OCR) show that the stateof-the-art DNN-based STD methods including regression-based
and segmentation-based methods are vulnerable to adversarial
attacks.

Transferable Adversarial Attacks for Deep SceneText Detection

Dense pixel matching is required for many computer vision algorithms such as disparity, optical flow or scene flow estimation. Feature Pyramid Networks (FPN) have proven to be a suitable feature extractor for CNN-based dense matching tasks. FPN generates well localized and semantically strong features at multiple scales. However, the generic FPN is not utilizing its full potential, due to its reasonable but limited localization accuracy. Thus, we present ResFPN - a multi-resolution feature pyramid network with multiple residual skip connections, where at any scale, we leverage the information from higher resolution maps for stronger and better localized features. In our ablation study, we demonstrate the effectiveness of our novel architecture with clearly higher accuracy than FPN. In addition, we verify the superior accuracy of ResFPN in many different pixel matching applications on established datasets like KITTI, Sintel, and FlyingThings3D.

ResFPN: Residual Skip Connections in Multi-Resolution Feature Pyramid Networks for Accurate Dense Pixel Matching

Deep convolutional neural networks (CNNs) have recently achieved great success for single image super-resolution (SISR) task due to their powerful feature representation capabilities. The most recent deep learning based SISR methods focus on designing deeper / wider models to learn the non-linear mapping between low-resolution (LR) inputs and high-resolution (HR) outputs. These existing SR methods do not take into account the image observation (physical) model and thus require a large number of network's trainable parameters with a great volume of training data. To address these issues, we propose a deep Iterative Super-Resolution Residual Convolutional Network (ISRResCNet) that exploits the powerful image regularization and large-scale optimization techniques by training the deep network in an iterative manner with a residual learning approach. Extensive experimental results on various super-resolution benchmarks demonstrate that our method with a few trainable parameters improves the results for different scaling factors in comparison with the state-of-art methods.

Deep Iterative Residual Convolutional Network for Single Image Super-Resolution

In the last few years, several techniques for facial
manipulation in videos have been successfully developed and
made available to the masses (i.e., FaceSwap, deepfake, etc.).
These methods enable anyone to easily edit faces in video
sequences with incredibly realistic results and a very little effort.
Despite the usefulness of these tools in many fields, if used
maliciously, they can have a significantly bad impact on society
(e.g., fake news spreading, cyber bullying through fake revenge
porn). The ability of objectively detecting whether a face has
been manipulated in a video sequence is then a task of utmost
importance.
In this paper, we tackle the problem of face manipulation detection in video sequences targeting modern facial manipulation
techniques. In particular, we study the ensembling of different
trained Convolutional Neural Network (CNN) models. In the
proposed solution, different models are obtained starting from a
base network (i.e., EfficientNetB4) making use of two different
concepts: (i) attention layers; (ii) siamese training. We show that
combining these networks leads to promising face manipulation
detection results on two publicly available datasets with more
than 119000 videos.

Premium content

Information Graphic Summarization using a Collection of Multimodal Deep Neural Networks

Downloads

Next from ICPR 2020

Self-Training for Domain Adaptive Scene Text Detection

Similar lecture

Information Graphic Summarization using a Collection of Multimodal Deep Neural Networks