Crowdsourcing is a new collaborative problem-solving paradigm, which allocates tasks that are not suitable to be solved by computers to online workers. However, prior studies neglect the quality of workers and fix the number of workers needed to complete a task, which degrades the quality of the final answer aggregated from multiple workers and easily incurs a high cost. In this paper, we propose a worker-quality adaptive task allocation framework to make a trade-off between the total monetary cost and the answer quality. We first propose a marginal likelihood estimation method to evaluate the accuracy of the workers, which eliminates the existence of correct answers and exhibits superior performance compared to the expectation maximization estimation, especially in the case of small datasets. Next, we formulate the task allocation problem that maximizes the total accuracy with a budget constraint. We devise a reverse auction-based algorithm that dynamically determines the number of participating workers based on their quality and bids without prior knowledge of the task difficulty. In addition, we employ the concept of Shapley value to fairly allocate the payment to reflect the contribution of each worker. Extensive experimental evaluations on real and synthetic datasets validate the effectiveness of our approach. Specifically, our MLE-based accuracy estimation achieves a low RMSE of 0.16 when applied to the scenario involving 10 workers and 4 tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Worker-Quality Adaptive Task Assignment in Collaborative Crowdsourcing

  • Bo Yin,
  • Sai Tang

摘要

Crowdsourcing is a new collaborative problem-solving paradigm, which allocates tasks that are not suitable to be solved by computers to online workers. However, prior studies neglect the quality of workers and fix the number of workers needed to complete a task, which degrades the quality of the final answer aggregated from multiple workers and easily incurs a high cost. In this paper, we propose a worker-quality adaptive task allocation framework to make a trade-off between the total monetary cost and the answer quality. We first propose a marginal likelihood estimation method to evaluate the accuracy of the workers, which eliminates the existence of correct answers and exhibits superior performance compared to the expectation maximization estimation, especially in the case of small datasets. Next, we formulate the task allocation problem that maximizes the total accuracy with a budget constraint. We devise a reverse auction-based algorithm that dynamically determines the number of participating workers based on their quality and bids without prior knowledge of the task difficulty. In addition, we employ the concept of Shapley value to fairly allocate the payment to reflect the contribution of each worker. Extensive experimental evaluations on real and synthetic datasets validate the effectiveness of our approach. Specifically, our MLE-based accuracy estimation achieves a low RMSE of 0.16 when applied to the scenario involving 10 workers and 4 tasks.