Abstract

Free-text rationales aim to explain neural language model (LM) behavior more flexibly and intuitively via natural language. To ensure rationale quality, it is important to have metrics for measuring rationales’ faithfulness (reflects LM’s actual behavior) and plausibility (convincing to humans). All existing free-text rationale metrics are based on simulatability (association between rationale and LM’s predicted label), but there is no protocol for assessing such metrics’ reliability. To investigate this, we propose FRAME, a framework for evaluating free-text rationale simulatability metrics. FRAME is based on three axioms:(1) good metrics should yield highest scores for reference rationales, which maximize rationale-label association by construction;(2) good metrics should be appropriately sensitive to semantic perturbation of rationales; and (3) good metrics should be robust to variation in the LM’s task performance. Across …

Date

2022

Authors

Aaron Chan, Shaoliang Nie, Liang Tan, Xiaochang Peng, Hamed Firooz, Maziar Sanjabi, Xiang Ren

Journal

arXiv preprint arXiv:2207.00779