Italy


One important and particularly challenging step in the optical character recognition of historical documents with complex layouts, such as newspapers, is the separation of text from non-text content (e.g. page borders or illustrations). This step is commonly referred to as page segmentation. While various rule-based algorithms have been proposed, the applicability of Deep Neural Networks for this task recently has gained a lot of attention. In this paper, we perform a systematic evaluation of 11 different published backbone architectures and 9 different tiling and scaling configurations for separating text, tables or table column lines. We also show the influence of the number of labels and the number of training pages on the segmentation quality, which we measure using the Matthews Correlation Coefficient. Our results show that (depending on the task) Inception-ResNet- v2 and EfficientNet backbones work best, vertical tiling is generally preferable to other tiling approaches, and training data that comprises 30 to 40 pages will be sufficient most of the time.

ICPR 2020

An Evaluation of DNN Architectures for Page Segmentation of Historical Newspapers

optical character recognition software

image segmentation

neural networks

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


poster

Current 3D object detection methods adopt an analogous box prediction structure with the 2D methods, which predict center and size of the object simultaneously in a box regression procedure, leading to the poor performance of 3D detector to a great extent. In this work, we propose S-VoteNet, which converts the prediction of 3D bounding box into two parts: center prediction and size prediction. By introducing a novel spherical proposal, S-VoteNet uses vote groups to predict the center and radius of object rather than all parameters of 3D bounding box. The prediction of radius is used to constrain the object size, and the radius-based spherical center loss is applied to measure the geometric distance between the proposal and ground-truth. To make better use of the geometric information provided by point cloud, S-VoteNet gathers seed points whose corresponding votes are within the vote groups for seed group generation. Seed groups are then consumed for box size regression and orientation estimation. By decoupling the localization and size estimation, our method effectively reduces the regression pressure of the 3D detector. Experimental results on SUN RGB-D 3D detection benchmark demonstrate that our S-VoteNet achieves state-of-the-art performance by using only point cloud as input.

S-VoteNet: Deep Hough Voting with Spherical Proposal for 3D Object Detection

We propose a new hierarchical method to match keypoints by exploiting information across multiple scales. Traditionally, for each keypoint a single scale is detected and the matching process is done in the specific scale. We replace this approach with matching across scale-space. The holistic information from higher scales are used for early rejection of candidates that are far away in the feature space. The more localized and finer details of lower scale are then used to decide between remaining possible points. The proposed multi-scale solution is more consistent with the multi-scale processing that is present in the human visual system and is therefore biologically plausible. We evaluate our method on several datasets and achieve state of the art accuracy, while significantly outperforming others in extraction time.

Multi-Scale Keypoint Matching


In recent years, the advances in Generative Adversarial
Networks (GANs) have shown impressive results for image
generation and translation tasks. In particular, the image-toimage
translation is a method of learning mapping from a source
domain to a target domain and synthesizing an image. Image-toimage
translation can be applied to a variety of tasks, making
it possible to quickly and easily synthesize realistic images from
semantic segmentation masks. However, in the existing image-toimage
translation method, there is a limitation on controlling the
style of the translated image, and it is not easy to synthesize
an image by controlling the style of each mask element in
detail. Therefore, we propose an image synthesis method that
controls the style of each element by improving the existing
image-to-image translation method. In the proposed method, we
implement a mask style encoder that extracts style features for
each mask element. The extracted style features are concatenated
to the semantic mask in the normalization layer, and used
the style-controlled image synthesis of each mask element. In
the experiments, we performed style-controlled images synthesis
using the datasets consisting of semantic segmentation masks
and real images. The results show that the proposed method has
excellent performance for style-controlled images synthesis for
each element.

Mask-based Style-Controlled Image Synthesis Using a Mask Style Encoder

It has recently become a major threat to the public media that fake videos are rapidly spreading over the Internet. The advent of Deepfake, a deep-learning based toolkit, has facilitated a massive abuse of improper synthesized videos, which may influence the media credibility and human rights. A worldwide alert has been set off that finding ways to detect such fake videos is not only crucial but also urgent. This paper reports a novel approach to expose deepfake videos. We found that most fake videos are markedly different from the real ones in the way the eyes move. We are thus motivated to define four features that could well capture such differences. The features are then fed to SVM for classification. It is shown to be a promising approach that without high dimensional features and complicated neural networks, we are able to achieve competitive results on several public datasets.

Exposing Deepfake Videos by Tracking Eye Movements

Video-based person re-identification has received
increasing attention recently, as it plays an important role within
surveillance video analysis. Video-based Re-ID is an expansion
of earlier image-based re-identification methods by learning
features from a video via multiple image frames for each person.
Most contemporary video Re-ID methods utilise complex CNNbased
network architectures using 3D convolution or multibranch
networks to extract spatial-temporal video features.
By contrast, in this paper, we illustrate superior performance
from a simple single stream 2D convolution network leveraging
the ResNet50-IBN architecture to extract frame-level features
followed by temporal attention for clip level features. These clip
level features can be generalised to extract video level features by
averaging without any significant additional cost. Our approach
uses best video Re-ID practice and transfer learning between
datasets to outperform existing state-of-the-art approaches on the
MARS, PRID2011 and iLIDS-VID datasets with 89,62%, 97,75%,
97,33% rank-1 accuracy respectively and with 84:61% mAP for
MARS, without reliance on complex and memory intensive 3D
convolutions or multi-stream networks architectures as found
in other contemporary work. Conversely, our work shows that
global features extracted by the 2D convolution network are a
sufficient representation for robust state of the art video Re-ID.

Not 3D Re-ID: Simple Single Stream 2D Convolution for Robust Video Re-identification

Accurately detecting craters in remotely sensed images is an important task when analysing the properties of planetary bodies. Commonly, only large craters in the range of several kilometres are detected. In this work we provide the first example of automatically detecting tiny craters in the range of several meters with the help of a deep neural network by using only a small set of annotated craters. Additionally, we propose a novel way to group overlapping detections and replace the commonly used non-maximum suppression with a probabilistic treatment. As a result, we receive valuable uncertainty estimates of the detections and the aggregated detections are shown to be vastly superior.

Uncertainty Guided Recognition of Tiny Craters on the Moon


In this paper, we propose to exploit the side-tuning framework for multimodal document classification. Side-tuning is a methodology for network adaptation recently introduced to solve some of the problems related to previous approaches. Thanks to this technique it is actually possible to overcome model rigidity and catastrophic forgetting of transfer learning by fine- tuning. The proposed solution uses off-the-shelf deep learning architectures leveraging the side-tuning framework to combine a base model with a tandem of two side networks. We show that side-tuning can be successfully employed also when different data sources are considered, e.g. text and images in document classification. The experimental results show that this approach pushes further the limit for document classification accuracy with respect to the state of the art.

Multimodal Side-Tuning for Document Classification


This work addresses the problem of automatically segmenting the MR images corresponding to the lumbar spine. The purpose is to detect and delimit the different structural elements like vertebrae, intervertebral discs, nerves, blood vessels, etc. This task is known as semantic segmentation. The approach proposed in this work is based on convolutional neural networks whose output is a mask where each pixel from the input image is classified into one of the possible classes. Classes were defined by radiologists and correspond to structural elements and tissues. The proposed network architectures are variants of the U-Net. Several complementary blocks were used to define the variants: spatial attention models, deep supervision and multi-kernels at input, this last block type is based on the idea of inception. Those architectures which got the best results are described in this paper, and their results are discussed. Two of the proposed architectures outperform the standard U-Net used as baseline.

Automatic Semantic Segmentation of Structural Elements related to the Spinal Cord in the Lumbar Region by Using Convolutional Neural Networks


Image-text matching is an interesting and fascinating task in modern AI research. Despite the evolution of deep-learning-based image and text processing systems, multi-modal matching remains a challenging problem. In this work, we consider the problem of accurate image-text matching for the task of multi-modal large-scale information retrieval. State-of-the-art results in image-text matching are achieved by inter-playing image and text features from the two different processing pipelines, usually using mutual attention mechanisms. However, this invalidates any chance to extract separate visual and textual features needed for later indexing steps in large-scale retrieval systems. In this regard, we introduce the Transformer Encoder Reasoning Network (TERN), an architecture built upon one of the modern relationship-aware self-attentive architectures, the Transformer Encoder (TE).
This architecture is able to separately reason on the two different modalities and to enforce a final common abstract concept space by sharing the weights of the deeper transformer layers.
Thanks to this design, the implemented network is able to produce compact and very rich visual and textual features available for the successive indexing step.
Experiments are conducted on the MS-COCO dataset, and we evaluate the results using a discounted cumulative gain metric with relevance computed exploiting caption similarities, in order to assess possibly non-exact but relevant search results. We demonstrate that on this metric we are able to achieve state-of-the-art results in the image retrieval task.

Transformer Reasoning Network for Image-Text Matching and Retrieval


We present a method to maximize feature matching
performance across stereo image pairs by varying
illumination. We perform matching between
views per lighting condition, finding unique SIFT
correspondences for each condition. These feature
matches are then collected together into a single
set, selecting those features which present the highest
quality match. Instead of capturing each view
under each illumination, we approximate lighting
changes with a pretrained relighting convolutional
neural network which only requires each view captured
under a single specified lighting condition. We
then collect the best of these feature matches over
all lighting conditions offered by the relighting network.
We further present an optimization to limit
the number of lighting conditions evaluated to gain
a specified number of matches. Our method is evaluated
on a set of indoor scenes excluded from training
the network with comparison to features extracted
from pretrained VGG16. Our method offers an average
5.5× improvement in number of correct matches
while retaining similar precision than by the original
lit image pair per scene alone.

Premium content

An Evaluation of DNN Architectures for Page Segmentation of Historical Newspapers

Downloads

Next from ICPR 2020

S-VoteNet: Deep Hough Voting with Spherical Proposal for 3D Object Detection

Similar lecture

Automatically Gather Address Specific Dwelling Images Using Google Street View