Netherlands

This presentation proposes a heuristic algorithm for effectively summarizing the work of novice robot operators, e.g., ones recruited through crowdsourcing platforms, in search and rescue-like tasks. Such summaries can be used for many purposes, perhaps most notably for monitoring and evaluating operator&#39;s performance in settings where information gaps preclude automatic evaluation. The underlying idea of our method is dividing the task timeline into intervals, and extracting a subset of high-scoring and low-scoring segments within, using a heuristic scoring function. This results in a short effective summary of the operator&#39;s work, based on which several other crowdworkers can evaluate her performance. The effectiveness of the proposed method was extensively evaluated and compared to a large set of alternative methods through a series of experiments in Amazon Mechanical Turk. The analysis of the results reveals that the proposed method outperforms all tested alternatives. Finally, we evaluate the performance one may achieve with the use of machine learning for predicting the operator&#39;s performance in our domain. While this approach manages to reach a performance level similar to the one achieved with summaries, it requires an order-of-magnitude greater effort for training (measured in terms of crowdworkers time).

HCOMP 2020

Effective Operator Summaries Extraction

behavioral-based summarization

collective evaluation

automatic summarization

crowdsourcing

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


This presentation proposes a heuristic algorithm for effectively summarizing the work of novice robot operators, e.g., ones recruited through crowdsourcing platforms, in search and rescue-like tasks. Such summaries can be used for many purposes, perhaps most notably for monitoring and evaluating operator's performance in settings where information gaps preclude automatic evaluation. The underlying idea of our method is dividing the task timeline into intervals, and extracting a subset of high-scoring and low-scoring segments within, using a heuristic scoring function. This results in a short effective summary of the operator's work, based on which several other crowdworkers can evaluate her performance. The effectiveness of the proposed method was extensively evaluated and compared to a large set of alternative methods through a series of experiments in Amazon Mechanical Turk. The analysis of the results reveals that the proposed method outperforms all tested alternatives. Finally, we evaluate the performance one may achieve with the use of machine learning for predicting the operator's performance in our domain. While this approach manages to reach a performance level similar to the one achieved with summaries, it requires an order-of-magnitude greater effort for training (measured in terms of crowdworkers time).

technical paper

Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic
opportunities, e.g., rural areas. However, we currently have
limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make
a living through crowd work sites. This paper examines the
challenges and advantages of rural and super rural Amazon
Mechanical Turk (MTurk) crowd workers and contrasts them
with those of workers from urban areas. Based on a survey of
421 crowd workers from differing geographic regions in the
U.S., we identified how across regions, people struggled with
being onboarded into crowd work. We uncovered that despite
the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We
also identified cultural traits, relating to time dimension and
individualism, that offer us an insight into crowd workers and
the necessary qualities for them to succeed on gig platforms.
We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools.

The Challenges of Crowd Workers in Rural and Urban America

Our paper provides and evaluates a new paradigm for collaborative human-robot operation in search and rescue-like settings with information asymmetry. In particular, we focus on settings where the human, a crowdworker in our case, is used as a sensor, providing the route-planning module with essential environmental information. In such settings, the ability to predict the expected performance of the collaborating crowdworker in real-time is instrumental for maintaining a continuously high level of performance. Through an extensive set of experiments with crowdworkers recruited and interacted through Amazon Mechanical Turk, we show that effective online prediction is indeed possible, however only if distinguishing between two subpopulations of crowdworkers, termed "operators" and "sensors", applying a different prediction model to each. Furthermore, we show that even the classification of crowdworkers to the two types can be carried out successfully in real-time, based merely on the first two minutes of collaboration. Finally, we demonstrate how the above abilities can be used for a more effective workers' recruiting process, resulting in a substantially improved overall performance.

Predicting Crowdworkers' Performance as Human-Sensors for Robot Navigation

It is becoming increasingly common for intelligent systems, such as recommender systems, to provide explanations for their generated recommendations to the users. However, we still do not have a good understanding of what types of explanations work and what factors affect the effectiveness of different types of explanations. Our work focuses on explanations for movie recommender systems. This paper presents a mixed study where we hypothesize that the type of explanation, as well as user motivation for watching movies, will affect how users respond to recommendation system explanations. Our study compares three types of explanations: i) neighbor-ratings, ii) profile-based, and iii) event-based, as well as three types of user movie-watching motivations: i) hedonic (fun and relaxation), ii) eudaimonic (inspiration and meaningfulness), and iii) educational (learning new content). We discuss the implications of the study results for the design of explanations for movie recommender systems, and future novel research directions that the study results uncover.

Understanding the Effects of Explanation Types and User Motivations on Movie Recommender System Use

On-demand emotional support is an expensive and elusive societal need that is exacerbated in difficult times -- as witnessed during the COVID-19 pandemic. Prior work in affective crowdsourcing has examined ways to overcome technical challenges for providing on-demand emotional support to end users. This can be achieved by training crowd workers to provide thoughtful and engaging on-demand emotional support. Inspired by recent advances in conversational user interface research, we investigate the efficacy of a conversational user interface for training workers to deliver psychological support to users in need. To this end, we conducted a between-subjects experimental study on Prolific, wherein a group of workers (N=200) received training on motivational interviewing via either a conversational interface or a conventional web interface. Our results indicate that training workers in a conversational interface yields both better worker performance and improves their user experience in on-demand stress management tasks.

Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy

Training workers within a task is one way of enabling novice workers to work on complex crowdsourcing tasks. Based on goal setting theory in psychology, we conduct a randomized experiment to study whether and how setting different goals---including performance goal, learning goal, and behavioral goal---when training workers for a complex crowdsourcing task affects workers' learning perception, learning gain, and post-training performance. We find that setting different goals during training significantly affects workers' learning perception, but overall does not have an effect on learning gain or post-training performance. However, higher levels of learning gain can be obtained when setting learning goals for workers who are highly learning-oriented. Additionally, giving workers a challenging behavioral goal can nudge them to adopt desirable behavior meant to improve learning and performance, though the adoption of such behavior does not lead to as much improvement as when the worker decides to take part in the behavior themselves.

Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training

In online crowd mapping, crowd workers recruited through crowdsourcing marketplaces collects geographic data. Compared to traditional mapping methods, where workers physically explore the area, the benefit of using online crowd mapping is the potential to be cost-effective and time-efficient. Previous studies have focused on mapping urban objects using street-level imagery. However, they are specifically aimed at a single type of object, and only through web platforms. To the best of our knowledge, there is still a lack of understanding on how workers perform the mapping tasks through different platforms. Aiming to fill this knowledge gap, we investigate the worker performance across web, mobile, and virtual reality platforms by designing a multi-platform system for mapping urban objects using street-level imagery with novel methods for geo-location estimation. We design a preliminary study to show the feasibility of executing online mapping tasks on three platforms. The result demonstrates that the type of task and execution platform can affect the worker performance in terms of worker accuracy, execution time, user engagement, and cognitive load.

Analyzing Workers Performance in Online Mapping Tasks Across Web, Mobile, and Virtual Reality Platforms

Although crowd work is typically completed through desktop or laptop computers by workers at their home, literature has shown that crowdsourcing is feasible through a wide array of computing devices, including smartphones and digital voice assistants. An integrated crowdsourcing platform that operates across multiple devices could provide greater flexibility to workers, but there is little understanding of crowd workers’ perceptions on uptaking crowd tasks across multiple contexts through such devices. Using a crowdsourcing survey task, we investigate workers’ willingness to accept different types of crowd tasks presented on three device types in different scenarios of varying location, time and social context. Through analysis of over 25,000 responses received from 329 crowd workers on Amazon Mechanical Turk, we show that when tasks are presented on different devices, the task acceptance rate is 80.5% on personal computers, 77.3% on smartphones and 70.7% on digital voice assistants. Our results also show how different contextual factors such as location, social con- text and time influence workers decision to accept a task on a given device. Our findings provide important insights towards the development of effective task assignment mechanisms for cross-device crowd platforms.

How Context Influences Cross-Device Task Acceptance in Crowd Work

Explaining to users why automated systems make certain mistakes is important and challenging. Researchers have proposed ways to automatically produce interpretations for deep neural network models. However, it is unclear how useful these interpretations are in helping users figure out why they are getting an error. If an interpretation effectively explains to users how the underlying deep neural network model works, people who were presented with the interpretation should be better at predicting the model’s outputs than those who were not. This paper presents an investigation on whether or not showing machine-generated visual interpretations helps users understand the incorrectly predicted labels produced by image classifiers. We showed the images and the correct labels to 150 online crowd workers and asked them to select the incorrectly predicted labels with or without showing them the machine-generated visual interpretations. The results demonstrated that displaying the visual interpretations did not increase, but rather decreased, the average guessing accuracy by roughly 10%.

How Useful Are the Machine-Generated Interpretations? A Human Evaluation on Guessing the Wrongly Predicted Labels

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called ``batch prioritization'' and ``label expansion''. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.

Batch Prioritization of Data Labeling Tasks for Training Classifiers

There are many factors that affect the quality of data received
from crowdsourcing, including cognitive biases, varying levels
of expertise, and varying subjective scales. This work
investigates how the elicitation and integration of multiple
modalities of input can enhance the quality of collective estimations.
We create a crowdsourced experiment where participants
are asked to estimate the number of dots within images
in two ways: ordinal (ranking) and cardinal (numerical) estimates.
We run our study with 300 participants and test how
the efficiency of crowdsourced computation is affected when
asking participants to provide ordinal and/or cardinal inputs
and how the accuracy of the aggregated outcome is affected
when using a variety of aggregation methods. First, we find
that more accurate ordinal and cardinal estimations can be
achieved by prompting participants to provide both cardinal
and ordinal information. Second, we present how accurate
collective numerical estimates can be achieved with significantly
fewer people when aggregating individual preferences
using optimization-based consensus aggregation models. Interestingly,
we also find that aggregating cardinal information
may yield more accurate ordinal estimates.

Downloads

Next from HCOMP 2020

The Challenges of Crowd Workers in Rural and Urban America

Similar lecture

Assessing Political Bias using Crowdsourced Pairwise Comparisons

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from HCOMP 2020

The Challenges of Crowd Workers in Rural and Urban America

Similar lecture

Assessing Political Bias using Crowdsourced Pairwise Comparisons

Downloads