Abstract

Crowdsourcing systems aggregate decisions of many people to help users quickly identify high-quality options, such as the best answers to questions or interesting news stories. A long-standing issue in crowdsourcing is how option quality and human judgement heuristics interact to affect collective outcomes, such as the perceived popularity of options. We address this limitation by conducting a controlled experiment where subjects choose between two ranked options whose quality can be independently varied. We use this data to construct a model that quantifies how judgement heuristics and option quality combine when deciding between two options. The model reveals popularity-ranking can be unstable: unless the quality difference between the two options is sufficiently high, the higher quality option is not guaranteed to be eventually ranked on top. To rectify this instability, we create an algorithm that …

Date

October 14, 2020

Authors

Keith Burghardt, Tad Hogg, Raissa D'Souza, Kristina Lerman, Marton Posfai

Journal

Proceedings of the ACM on Human-Computer Interaction

Volume

4

Issue

CSCW2

Pages

1-20

Publisher

ACM