Worker-Quality Adaptive Task Assignment in Collaborative Crowdsourcing
摘要
Crowdsourcing is a new collaborative problem-solving paradigm, which allocates tasks that are not suitable to be solved by computers to online workers. However, prior studies neglect the quality of workers and fix the number of workers needed to complete a task, which degrades the quality of the final answer aggregated from multiple workers and easily incurs a high cost. In this paper, we propose a worker-quality adaptive task allocation framework to make a trade-off between the total monetary cost and the answer quality. We first propose a marginal likelihood estimation method to evaluate the accuracy of the workers, which eliminates the existence of correct answers and exhibits superior performance compared to the expectation maximization estimation, especially in the case of small datasets. Next, we formulate the task allocation problem that maximizes the total accuracy with a budget constraint. We devise a reverse auction-based algorithm that dynamically determines the number of participating workers based on their quality and bids without prior knowledge of the task difficulty. In addition, we employ the concept of Shapley value to fairly allocate the payment to reflect the contribution of each worker. Extensive experimental evaluations on real and synthetic datasets validate the effectiveness of our approach. Specifically, our MLE-based accuracy estimation achieves a low RMSE of 0.16 when applied to the scenario involving 10 workers and 4 tasks.