Exposing Brittleness in Reading Comprehension Systems

Thursday, November 1, 2018, 11:00 am - 12:00 pm PDTiCal
6th Floor Conf Rm- #689
This event is open to the public.
NL Seminar
Robin Jia (Stanford Univ)

Abstract: Reading comprehension systems that answer questions over a context passage can often achieve high test accuracy, but they are frustratingly brittle: they often rely heavily on superficial cues, and therefore struggle on out-of-domain inputs. In this talk, I will describe our work on understanding and challenging these systems. First, I will show how to craft adversarial reading comprehension examples by adding irrelevant distracting text to the context passage. Next, I will present the newest version of the SQuAD dataset, SQuAD 2.0, which tests whether models can distinguish answerable questions from similar but unanswerable ones. Finally, I will share some observations from our recent attempts to use reading comprehension systems as a natural language interface for building other NLP systems.

Bio: Robin Jia is a fifth-year PhD student advised by Percy Liang at Stanford University. He is an NSF Graduate Fellow, and has received Outstanding Paper and Best Short Paper Awards from EMNLP and ACL, respectively

« Return to Upcoming Events