Netherlands

Qualification tests in crowdsourcing are often used to pre-filter workers by measuring their ability in executing microtasks. While creating qualification tests for each task type is considered as a common and reasonable way, this study investigates into its worker-filtering performance when the same qualification test is used across multiple types of tasks. On Amazon Mechanical Turk, we tested the annotation accuracy in six different cases where tasks consisted of two different difficulty levels, arising from the identical real-world domain: four combinatory cases in which the qualification test and the actual task were the same or different from each other, as well as two other cases where workers with Masters Qualification were asked to perform the actual task only. The experimental results demonstrated the two following findings: i) Workers that were assigned to a difficult qualification test scored better annotation accuracy regardless of the difficulty of the actual task; 
ii) Workers with Masters Qualification scored better annotation accuracy on the low-difficulty task but were not as accurate as those who passed a qualification test on the high-difficulty task.

HCOMP 2020

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

masters qualification

qualification test

worker-filtering

crowdsourcing

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


technical paper

While crowdsourcing provides a convenient solution for tapping into human intelligence, a concern is the bias inherent in the data collected. Events related to the COVID-19 pandemic had an impact on people globally, and crowdworkers were no exception. Given the evidence concerning mood and stress on work, we explore how temporal events might affect crowdsourced data. We replicated an image annotation task conducted in 2018, in which workers describe people images. We expected 2020 annotations to contain more references to health, as compared to 2018 data. Overall, we find no evidence that health-related tags were used more often in 2020, but instead we find a significant increase in the use of tags related to weight (e.g., fat, chubby, overweight). This result, coupled with the “stay at home” act in effect in 2020, illustrate how crowdwork is impacted by temporal events.

Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic

We propose a crowdsourced approach for assessing political bias among news publishers. We aggregate pairwise comparisons between publishers into a relative ranking of leanings, in contrast to prior crowdsourcing approaches that request in-dependent ratings of publishers. We verify our approach by a preliminary study with 45 participants, who were asked to rank the political bias of 8 news publishers and to provide qualitative feedback on the study. We report our experimental results and describe our envisioned application scenario.

Assessing Political Bias using Crowdsourced Pairwise Comparisons

How to obtain higher quality task results with a fixed budget (labor cost) is one of the most important problems in practical crowdsourcing. We approach this problem by (1) dynamically choosing appropriate sets of tasks and workers during the worker-task assignment process and (2) introducing an ML worker that performs tasks as if it was a human worker. Our approach, named Dynamic Worker and Task Assignment with ML workers (DWTAML), chooses tasks and workers during the worker-task assignment process, based on the estimation of the reliability of workers and the confidence of task results at each moment. It extends the Dawid&Skene model and IEThresh. The results of an extensive set of experiments showed that DWTAML is effective in most cases.

Dynamic Worker-Task Assignment for High-Quality Task Results with ML Workers

Crowdsourcing offers a new form of work that makes it easy for people to work from home on their own computers. However, for some people with disabilities, such as visual impairment, problems are encountered in completing tasks. In this study, a questionnaire was used to investigate the working conditions of and problems encountered by visually impaired people on Japanese and US crowdsourcing platforms. The problems overcome by visually impaired people, difference between the visually impaired crowd workers from different countries, and possible solutions are provided herein. This research can be extended to more crowdsourcing platforms and languages in the future.

A Survey of Visually Impaired Workers in Japanese and US Crowdsourcing Platforms

In crowdsourcing, although there is a plenty of benefits in terms of cost and time to solve a complex task, there remain many challenges to reach into a consensus judgment from multiple crowd opinions. In this paper, we pay attention to a recently introduced crowd judgment model, termed as ``Constrained Crowd Judgment Analysis'' model. Unlike the traditional crowdsourced opinions (i.e., binary/multiple opinions), the option set in constrained judgment are undefined and there exist multiple components for a particular question. Moreover, there exist some constraints that should be preserved while deriving the final judgment, hence the problem becomes more difficult. Additionally, it is very difficult to rank the crowd workers as there is no ground truth present in this constrained judgment setting. In this paper, we introduce a Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) based multi-objective model for reaching consensus judgment from multiple crowd opinions as well as providing a better ranking among the crowd worker. The preliminary experimental study over a crowdsourced dataset demonstrates the effectiveness of the proposed model.

A TOPSIS-based Multi-objective Model for Constrained Crowd Judgment Analysis

Folk theories represent the users' beliefs of how an AI-driven system operate. Prior research investigated folk theories as a lens on users' reasoning about the algorithmically-curated content of news feeds on social media platforms primarily in interview studies and surveys. In this work-in-progress, we present an interactive interface for the structured capturing and crowdsourced collection of folk theories at scale.

Socially Augmented Crowdsourced Collection of Folk Theories

Extensive work has argued in favour of paying crowd workers a wage that is at least equivalent to the U.S. federal minimum wage. Meanwhile, research on collecting high quality annotations (e.g. for Natural Language Processing) suggests using qualifications such as a minimum number of previously completed tasks. If most requesters who pay fairly use this kind of minimum qualification, then workers may be forced to complete a substantial amount of poorly paid work for other requesters before they can earn a fair wage. This paper (1) explores current conventions for the threshold, (2) discusses possible alternatives, and (3) presents a study of correlation between approved work and work quality.

Qualification Labour: A Fair Wage Isn’t Enough if Workers Need to Do 5,000 Low Paid Tasks to Qualify for Your Task

This work demonstrates how semi-supervised learning and human-in-the-loop crowdsourcing can help neural machine translation (NMT) challenges common in low-resource languages. We focus on the Mande language Bambara, which has approximately 16 million primary and secondary speakers in Western Africa. Bambara is mainly spoken as opposed to written language and it has few digital resources due to its history in regions where colonial French became the language of government and industry. Thus, Bambara is a "low-resource language" and because it lacks the existing language resources (parallel digital text and labeled data) necessary for NMT, we describe a novel crowdsourcing approach to support semi-supervised NMT. We designed a crowdsourcing platform that requests the annotator to supply information when the NMT model has decision confusion. Our crowdsourcing platform was tested on evaluating translations of Malian broadcast news and Wikipedia pages in Bambara. Our initial research shows a wide variation in the quality of the translations and further work includes a more rigorous evaluation of translator skills when onboarding new annotators.

Towards a Crowdsourcing Platform for Low Resource Languages -- A Collectivist Approach

Workers on crowdsourcing platforms suffer from low working conditions, including low hourly income, the difficulty of communication with job requesters and work stress due to a lack of social belonging. Research has revealed the great potential that worker cooperation could improve overall efficiency and get them more engaged in crowd work. 
In this presentation, I will introduce my research on how to improve crowd working conditions through worker collaboration with a novel Computer-supported cooperative work (CSCW) system. Following the Design Science Research Methodology (DSRM), each sub-system will be designed to meet a specific function like peer learning and task recommendation. Controlled experiments will be conducted to evaluate the effectiveness of the function and the influence on working conditions.

Improving Crowd Working Conditions with A Novel Cooperation System

We present our work in progress platform Turkish Judge, a crowd-driven adjudication system for rejected work on Amazon Mechanical Turk. The Mechanical Turk crowdsourcing platform allows Requesters to approve or reject assignments submitted by Workers. If the work is rejected, then Workers aren't paid, and their reputation suffers. Currently, there is no built-in mechanism for Workers to appeal rejections, other than contacting Requesters directly. The time it takes Requesters to review potentially incorrectly rejected tasks means that their costs are substantially higher than the payment amount that is in dispute. As a solution to this issue, we present an automated appeals system called Turkish Judge which employs crowd workers as judges to adjudicate whether work was fairly rejected when their peers initiate an appeal.

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

Downloads

Next from HCOMP 2020

Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic

Similar lecture

Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from HCOMP 2020

Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic

Similar lecture

Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

Downloads