Abstract

Question: I have five fingers but I am not alive. What am I? Answer: a glove.
Answering such a riddle-style question is a challenging cognitive process, in that it requires complex commonsense reasoning abilities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural language understanding (NLU). However, there is currently no dataset aiming to test these abilities. In this paper, we present RIDDLESENSE 1, a new multiple-choice question answering task, which comes with the first large dataset (5.7 k examples) for answering riddlestyle commonsense questions. We systematically evaluate a wide range of models over the RIDDLESENSE challenge, and point out that there is a large gap between the bestsupervised model and human performance—suggesting intriguing future research in the direction of higher-order commonsense reasoning and linguistic creativity towards building advanced NLU systems.

Date

2021

Authors

Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, Xiang Ren

Conference

Findings of ACL 2021

Pages

1504-1515