Netherlands

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called ``batch prioritization&#39;&#39; and ``label expansion&#39;&#39;. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.

HCOMP 2020

Batch Prioritization of Data Labeling Tasks for Training Classifiers

prioritization

crowdsourcing

classification

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


ICPR 2020

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called ``batch prioritization'' and ``label expansion''. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.

technical paper

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


There are many factors that affect the quality of data received
from crowdsourcing, including cognitive biases, varying levels
of expertise, and varying subjective scales. This work
investigates how the elicitation and integration of multiple
modalities of input can enhance the quality of collective estimations.
We create a crowdsourced experiment where participants
are asked to estimate the number of dots within images
in two ways: ordinal (ranking) and cardinal (numerical) estimates.
We run our study with 300 participants and test how
the efficiency of crowdsourced computation is affected when
asking participants to provide ordinal and/or cardinal inputs
and how the accuracy of the aggregated outcome is affected
when using a variety of aggregation methods. First, we find
that more accurate ordinal and cardinal estimations can be
achieved by prompting participants to provide both cardinal
and ordinal information. Second, we present how accurate
collective numerical estimates can be achieved with significantly
fewer people when aggregating individual preferences
using optimization-based consensus aggregation models. Interestingly,
we also find that aggregating cardinal information
may yield more accurate ordinal estimates.

Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

Recently, Peterson et al. provided evidence of the benefits ofusing probabilistic soft labels generated from crowd annota-tions for training a computer vision model, showing that us-ing such labels maximizes performance of the models overunseen data. In this paper, we generalize these results byshowing that training with soft labels is an effective methodfor using crowd annotations in several otherAItasks besidesthe one studied by Petersonet al., and also when their perfor-mance is compared with that of state-of-the-art methods forlearning from crowdsourced data.

A Case for Soft Loss Functions

The verification of Extended Entity Relationship (EER) diagrams and other conceptual models that capture the design of information systems is crucial to ensure reliable systems. To scale up verification processes to larger groups of experts, Human Computation techniques were used focusing primarily on closed tasks, which constrain the number and variety of reported defects in favour of easy aggregation of derived judgements. To address this limitation of closed tasks, in this paper, we investigate EER verification with open tasks to extend the range of collected results. We also address the challenge of aggregating results of open tasks by proposing a follow-up HC task for defect validation. We evaluate our approach for HC-based EER Verification with open tasks in a set of experiments conducted with junior developers and show that (1) open tasks allow collecting a variety of insights that go beyond a manually built gold standard while still leading to good performance (F1=60%) and (2) HC-based validation can be reliably used for validating the results of open tasks (F1=84% compared to expert validation). 


Verifying Extended Entity Relationship Diagrams with Open Tasks

Image Tagging Algorithms (ITAs) are extensively used in our information ecosystem, from facilitating the retrieval of images in social platforms to learning about users and their preferences. However, audits performed on ITAs have demonstrated that their behaviors often exhibit social biases, especially when analyzing images depicting people. We present OpenTag, a platform that fuses the auditing process with a crowdsourcing approach. Users can upload an image, which is then analyzed by various ITAs, resulting in multiple sets of descriptive tags. With OpenTag, the user can observe and compare the output of multiple ITAs simultaneously, while researchers can study the manner in which users perceive this output. Finally, using the collected data, further audits can be performed on ITAs.

OpenTag: Understanding Human Perceptions of Image Tagging Algorithms

Qualification tests in crowdsourcing are often used to pre-filter workers by measuring their ability in executing microtasks. While creating qualification tests for each task type is considered as a common and reasonable way, this study investigates into its worker-filtering performance when the same qualification test is used across multiple types of tasks. On Amazon Mechanical Turk, we tested the annotation accuracy in six different cases where tasks consisted of two different difficulty levels, arising from the identical real-world domain: four combinatory cases in which the qualification test and the actual task were the same or different from each other, as well as two other cases where workers with Masters Qualification were asked to perform the actual task only. The experimental results demonstrated the two following findings: i) Workers that were assigned to a difficult qualification test scored better annotation accuracy regardless of the difficulty of the actual task; 
ii) Workers with Masters Qualification scored better annotation accuracy on the low-difficulty task but were not as accurate as those who passed a qualification test on the high-difficulty task.

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

While crowdsourcing provides a convenient solution for tapping into human intelligence, a concern is the bias inherent in the data collected. Events related to the COVID-19 pandemic had an impact on people globally, and crowdworkers were no exception. Given the evidence concerning mood and stress on work, we explore how temporal events might affect crowdsourced data. We replicated an image annotation task conducted in 2018, in which workers describe people images. We expected 2020 annotations to contain more references to health, as compared to 2018 data. Overall, we find no evidence that health-related tags were used more often in 2020, but instead we find a significant increase in the use of tags related to weight (e.g., fat, chubby, overweight). This result, coupled with the “stay at home” act in effect in 2020, illustrate how crowdwork is impacted by temporal events.

Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic

We propose a crowdsourced approach for assessing political bias among news publishers. We aggregate pairwise comparisons between publishers into a relative ranking of leanings, in contrast to prior crowdsourcing approaches that request in-dependent ratings of publishers. We verify our approach by a preliminary study with 45 participants, who were asked to rank the political bias of 8 news publishers and to provide qualitative feedback on the study. We report our experimental results and describe our envisioned application scenario.

Assessing Political Bias using Crowdsourced Pairwise Comparisons

How to obtain higher quality task results with a fixed budget (labor cost) is one of the most important problems in practical crowdsourcing. We approach this problem by (1) dynamically choosing appropriate sets of tasks and workers during the worker-task assignment process and (2) introducing an ML worker that performs tasks as if it was a human worker. Our approach, named Dynamic Worker and Task Assignment with ML workers (DWTAML), chooses tasks and workers during the worker-task assignment process, based on the estimation of the reliability of workers and the confidence of task results at each moment. It extends the Dawid&Skene model and IEThresh. The results of an extensive set of experiments showed that DWTAML is effective in most cases.

Dynamic Worker-Task Assignment for High-Quality Task Results with ML Workers

Crowdsourcing offers a new form of work that makes it easy for people to work from home on their own computers. However, for some people with disabilities, such as visual impairment, problems are encountered in completing tasks. In this study, a questionnaire was used to investigate the working conditions of and problems encountered by visually impaired people on Japanese and US crowdsourcing platforms. The problems overcome by visually impaired people, difference between the visually impaired crowd workers from different countries, and possible solutions are provided herein. This research can be extended to more crowdsourcing platforms and languages in the future.

A Survey of Visually Impaired Workers in Japanese and US Crowdsourcing Platforms

In crowdsourcing, although there is a plenty of benefits in terms of cost and time to solve a complex task, there remain many challenges to reach into a consensus judgment from multiple crowd opinions. In this paper, we pay attention to a recently introduced crowd judgment model, termed as ``Constrained Crowd Judgment Analysis'' model. Unlike the traditional crowdsourced opinions (i.e., binary/multiple opinions), the option set in constrained judgment are undefined and there exist multiple components for a particular question. Moreover, there exist some constraints that should be preserved while deriving the final judgment, hence the problem becomes more difficult. Additionally, it is very difficult to rank the crowd workers as there is no ground truth present in this constrained judgment setting. In this paper, we introduce a Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) based multi-objective model for reaching consensus judgment from multiple crowd opinions as well as providing a better ranking among the crowd worker. The preliminary experimental study over a crowdsourced dataset demonstrates the effectiveness of the proposed model.

Batch Prioritization of Data Labeling Tasks for Training Classifiers

Downloads

Next from HCOMP 2020

Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

Similar lecture

Crowdsourced Verification for Operating Calving Surveillance Systems at an Early Stage

Batch Prioritization of Data Labeling Tasks for Training Classifiers

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from HCOMP 2020

Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

Similar lecture

Crowdsourced Verification for Operating Calving Surveillance Systems at an Early Stage

Downloads