Brazil

Despite the availability of benchmark machine learning (ML) repositories (e.g., UCI, OpenML), there is no standard evaluation strategy yet capable of pointing out which is the best set of datasets to serve as gold standard to test different ML algorithms. In recent studies, Item Response Theory (IRT) has emerged as a new approach to elucidate what should be a good ML benchmark. This work applied IRT to explore the well-known OpenML-CC18 benchmark to identify how suitable it is on the evaluation of classifiers. Several classifiers ranging from classical to ensembles ones were evaluated using IRT models, which could simultaneously estimate dataset difficulty and classifiers&#39; ability. The Glicko-2 rating system was applied on the top of IRT to summarize the innate ability and aptitude of classifiers. It was observed that not all datasets from OpenML-CC18 are really useful to evaluate classifiers. Most datasets evaluated in this work (84%) contain easy instances in general (e.g., around 10% of difficult instances only). Also, 80% of the instances in half of this benchmark are very discriminating ones, which can be of great use for pairwise algorithm comparison, but not useful to push classifiers abilities. This paper presents this new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks.

BRACIS 2020

Decoding machine learning benchmarks

openml

machine learning

benchmarking

classification


IWSSIP is an international conference that brings together researchers and developers from both academia and industry reporting the latest scientific and theoretical advances to discuss and debate major issues and state-of-the-art aspects on systems, signals and image processing.

###### For the session discussion, each author will be seated at individual breakout tables.  Please find which table they will be at here:
###### [QA table assignments in Gather.town](https://drive.google.com/file/d/1hBHOD_KF7NSsaTcpYcHCZslXJvQVxJQe/view?usp=sharing)

IWSSIP 2021

Despite the availability of benchmark machine learning (ML) repositories (e.g., UCI, OpenML), there is no standard evaluation strategy yet capable of pointing out which is the best set of datasets to serve as gold standard to test different ML algorithms. In recent studies, Item Response Theory (IRT) has emerged as a new approach to elucidate what should be a good ML benchmark. This work applied IRT to explore the well-known OpenML-CC18 benchmark to identify how suitable it is on the evaluation of classifiers. Several classifiers ranging from classical to ensembles ones were evaluated using IRT models, which could simultaneously estimate dataset difficulty and classifiers' ability. The Glicko-2 rating system was applied on the top of IRT to summarize the innate ability and aptitude of classifiers. It was observed that not all datasets from OpenML-CC18 are really useful to evaluate classifiers. Most datasets evaluated in this work (84%) contain easy instances in general (e.g., around 10% of difficult instances only). Also, 80% of the instances in half of this benchmark are very discriminating ones, which can be of great use for pairwise algorithm comparison, but not useful to push classifiers abilities. This paper presents this new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks.

technical paper

The 9th Brazilian Conference on Intelligent Systems (BRACIS), the 16h Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), and the 12th Concurso de Teses e Dissertações em Inteligência Artificial e Computacional (CTDIAC) was held from 20 to 23 October, 2020.

BRACIS originated from the combination of the two most important scientific events in Brazil in Artificial Intelligence (AI) and Computational Intelligence (CI): the Brazilian Symposium on Artificial Intelligence - SBIA (22 editions), and the Brazilian Symposium on Neural Networks - SBRN (13 editions). ENIAC is a national forum for researchers, practitioners, educators and students to present and discuss innovations, trends, experiences, developments and work in progress in the fields of Artificial and Computational Intelligence. CTDIAC is a satellite event whose goals are to disseminate and award the best dissertations and theses (MSc and PhDs concluded in Brazilian universities from June 2018 to May 2020) in Artificial and Computational Intelligence areas.

In this year, the Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) will be held in conjunction with BRACIS 2020. This sysposium aims at integrating researchers, practitioners, developers, students and users to present their research results, to discuss ideas, and to exchange techniques, tools, and practical experiences – related to the Data Mining and Machine Learning areas.

AGM’s belief revision is one of the main paradigms in the study of belief change operations. Despite its popularity and importance to the area, it is well recognized that AGM’s work relies on a strong idealisation of the agent’s capabilities and on the nature of beliefs themselves. Particularly, it is well-recognized in the area of Epistemology that Belief and Knowledge are hyperintensional notions, but to our knowledge only a few works have explicitly considered how hyperintensionality affects belief change. In this work, we investigate belief change operations based on Berto’s topic-sensitive framework and provide three different pseudo- contraction operations to account for the hyperintensional behaviour of beliefs. Our work highlights the connection of a foundational hyperintensional theory of belief with the results of AGM Belief Change. Also we propose and characterise different possible contraction-like operations in this framework.

Towards a Theory of Hyperintensional Belief Change

Recent advances in language representation using neural networkshave made it viable to transfer the learned internal states of large pretrained language models (LMs) to downstream natural language processing (NLP) tasks. This transfer learning approach improves the overall performance on many tasks and is highly beneficial when labeled data is scarce, making pretrained LMs valuable resources specially for languages with few annotated training examples. In this work, we train BERT (Bidirectional Encoder Representations from Transformers) models for Brazilian Portuguese, which we nickname BERTimbau. We evaluate our models on three downstream NLP tasks: sentence textual similarity, recognizing textual entailment, and named entity recognition. Our models improve the state-of-the-art in all of these tasks, outperforming Multilingual BERT and confirming the effectiveness of large pretrained LMs for Portuguese. We release our models to the community hoping to provide strong baselines for future NLP research: https://github.com/neuralmind-ai/portuguese-bert.

BERTimbau: pretrained BERT models for Brazilian Portuguese

The increasing adoption of industrial robots to boost production efficiency is turning human-robot collaborative scenarios much more frequent. In this context, technical factory workers need to be safe at all times from collisions and prepare for emergencies and potential accidents. Another trend in industrial automation is the usage of machine learning techniques - specifically, deep learning algorithms - for image classification. Following these tendencies, this work evaluates the application of deep learning models to detect physical collision in human-robot interactions. Security camera images are used as the primary information source for intelligent collision detection. Unlike other proposed approaches in the literature that apply sensors like Light Detection And Ranging (LIDAR), Laser Range Finder (LRF), or torque sensors from robots, this work does not consider extra sensors, using only 2D cameras. Results show more than 99% of accuracy in the evaluated scenarios, revealing that approaches adopting deep learning algorithms could be promising for human-robot collision avoidance in industrial scenarios. The proposed models may support safety in industrial environments and reduce the impact of collision accidents.

Assessing Deep Learning Models for Human-Robot Collaboration Collision Detection in Industrial Environments

Performing analytical tasks over graph data has become increasingly interesting due to the ubiquity and large availability of relational information. However, unlike images or sentences, there is no notion of sequence in networks. Nodes (and edges) follow no absolute order, and it is hard for traditional machine learning (ML) algorithms to recognize a pattern and generalize their predictions on this type of
data. Graph Neural Networks (GNN) successfully tackled this problem. They became popular after the generalization of the convolution concept to the graph domain. However, they possess a large number of hyperparameters and their design and optimization is currently hand-made, based on heuristics or empirical intuition. Neural Architecture Search (NAS) methods appear as an interesting solution to this problem. In this direction, this paper compares two NAS methods for optimizing GNN: one based on reinforcement learning and a second based on evolutionary algorithms. Results consider 7 datasets over two search spaces and show that both methods obtain similar accuracies to a random search, raising the question of how many of the search space dimensions are actually relevant to the problem.

Neural Architecture Search in Graph Neural Networks

A major challenge in image classification tasks using Ma-chine Learning, and in particular when using deep neural networks, is domain shifting in deployment. This happens when images during usageare capture in different conditions from those used during training. In this paper, we show that despite previous works on the diagnosis of apple tree diseases using standard Convolutional Neural Networks displaying high accuracy, they do so only when no domain shift is present. When the trained model is asked to classify photos of apples taken in the wild, a 22% reduction in F1 score is observed. We propose to treat the task as a segmentation problem and test two different approaches, showing that using Mask R-CNN allows not only to improve performance in the original domain by 3%, but also significantly reduce losses in the new domain (only 6% reduction in F1 score). We establish segmentation as an important alternative towards improving diagnosis of apple tree diseases from photos.

Diagnosis of Apple Fruit Diseases in the Wildwith Mask R-CNN

This work presents a new hybrid method based on the route-first-cluster-second approach using Greedy Randomized Adaptive Search Procedure (GRASP), Differential Evolution (DE), Evolutionary Local Search (ELS) and set-partitioning problem (SPP) to solve well-known instances of Capacitated Vehicle Routing Problem (CVRP). The CVRP consists of minimizing the cost of a fleet of vehicles serving a set of customers from a single depot, in which every vehicle has the same capacity. The DE heuristic is used to build an initial feasible solution and ELS is applied until a local minimum is found during the local search phase of the GRASP. Finally, the SPP model provides a new optimal solution with regard to the built solutions in the GRASP. We perform computational experiments for benchmarks available in the literature and the results show that our method was effective to solve CVRP instances with a satisfactory performance. Moreover, a statistical test shows that there is not significant difference between the best known solutions of benchmark instances and the solutions of the proposed method.

A new hybridization of evolutionary algorithms, GRASP and set-partitioning formulation for the Capacitated Vehicle Routing Problem

The design of digital logic circuits involves transforming logical expressions into an electronic circuit seeking to minimize certain attributes such as the number of transistors. There are deterministic algorithms for this task, but they are limited to small problems. On the other hand metaheuristics can be used for solving this type of problem and Cartesian Genetic Programming (CGP) is widely adopted in the literature. In CGP, mutation is commonly the only operator for generating new candidate solutions. Thus, the performance of this metaheuristic is dependent on the proper choice of mutation and its parameters. Normally, CGP mutates the candidate solutions according to a uniform distribution. Thus, any modification has the same chance to occur. In order to improve the performance of CGP an adaptive mutation operator using an $\epsilon$-greedy strategy for bias the selection of gates is proposed here. The proposal is evaluated using problems of a benchmark from the literature. The results obtained indicate that the proposed adaptive mutation is promising and that its relative performance increases with the size of the problem.

A Reinforcement Learning Based Adaptive Mutation for Cartesian Genetic Programming Applied to the Design of Combinational Logic Circuits

Radiography is a primary examination used to diagnose chest conditions, as it is fast, low cost, and widely available. If the physician cannot conclude de diagnosis with the radiography, a computed tomography scan may be required. However, this exam is expensive and has low availability, mainly in the public health system of developing countries and low-income locations, which can delay the treatment and cause complications to the patient's health condition. Computer-aided diagnosis systems provide more resources for medical diagnostic decision-making, increasing the accuracy of the assessment of the patient's clinical condition. The main objective of this work is to develop a deep-learning-based approach that performs an automatic analysis of digital images of chest radiographs to aid the detection of pulmonary nodules and masses, aiming to extract sufficient relevant information from the image, optimizing the initial phase of the diagnosis of lung lesions.
The developed approach uses neural networks in a dataset of 8,178 annotated chest radiographs extracted from a public dataset. Half of it is of images annotated with "nodule" or "mass," and the other half is of images with "no findings." We implemented and tested convolutional neural networks and data preprocessing techniques to create a classification model.
A model with five convolution layers that achieved 0.72 accuracy, 0.75 sensitivity, and 0.68 specificity.
The proposed approach achieved results comparable to state of the art for lesion identification using limited computational power and can assist radiological practice as a second opinion, which can improve the rates of early diagnosed cancer.

A deep learning approach for pulmonary lesion identification in chest radiographs

The Cooperative Multi-Robot Observation of Multiple Moving Targets (CMOMMT) considers two types of robots, observers and targets, in a partially observable 2D environment, in which the task of the observers is to monitor the target robots under a limited radial range of the sensor, minimizing the total time the targets escape observation. The Cooperative Target Observation (CTO), a variant of the CMOMMT problem, considers the environment to be fully observable, where targets cooperate with observers by reporting their locations. These problems are at the center of many problems that occur in real-world wildlife research, crowd social movement, and surveillance situations. Regarding research related to the field of surveillance, this work proposes an approach to the extended CTO problem, what we call the COSTA problem: Cooperative Observation of Smart Target Agents. The approach employs genetic algorithm and recurrent deep neural networks to improve the performance of target robots and, therefore, to improve the research of new decision-making strategies in the context of observer robots. The first results were auspicious, as the average number of target evasion increased, considering the other recent approaches to the problem.

Cooperative Observation of Smart Target Agents

As Wireless Sensor Networks - WSNs are being used more and more in applications with high demand for the processing of videos and images, the processing load of the nodes also must be considered by energy-saving strategies. Usually, these strategies focused only on communication. The overloading of nodes can result in a degraded performance regarding the network lifetime and availability of the provided services. A few algorithms, based on heuristics, have been proposed in the literature to balance the load at the application level. These algorithms drawnback are the lack of quality assurance. Thus, this work presents a novel semi-distributed solution, applying techniques to solve Constraint Optimization Problem - COP on the load balancing in WSN. Our approach guarantee that all events are processed during the lifetime of the network and that there is a fixed number of exchanged messages per event, only reducing the lifetime of the network by an estimable time. Experimental results pointed out that the proposed approach allows the network, while alive, to sense all happened events each day, and its performance is better than the one achieved by the state-of-the-art for lower density networks.

Decoding machine learning benchmarks

Downloads

Next from BRACIS 2020

Towards a Theory of Hyperintensional Belief Change

Similar lecture

Cluster Prediction Model based on Automatic Cluster Labeling and Fuzzy System

Decoding machine learning benchmarks

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from BRACIS 2020

Towards a Theory of Hyperintensional Belief Change

Similar lecture

Cluster Prediction Model based on Automatic Cluster Labeling and Fuzzy System

Downloads