Seminars and Events

ISI Natural Language Seminar

Prioritized training on points that are learnable, worth learning, and not yet learned

Event Details

REMINDER:

Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you’re highly encouraged to use your USC account to sign into Zoom.

If you’re an outside visitor, please inform us at (nlg-seminar-host(at)isi.edu) beforehand so we’ll be aware of your attendance and let you in.

In-person attendance will be held in CR#689, remote attendees can log on via Zoom.

For more information on the NL Seminar series and upcoming talks, please visit:

https://nlg.isi.edu/nl-seminar/ 

Training on web-scale data can take months. But much computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model’s generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select” hard”(eg high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes” easy” points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.

Speaker Bio

Bio Sören Mindermann:

Sören is a final-year PhD student in machine learning at the University of Oxford, supervised by Yarin Gal. My interests in machine learning include how it scales, causal inference and statistical modeling, as well as robustly aligning machine learning models to adopt human wishes and value.

Bio Jan Brauner:

Jan is a PhD candidate in the Centre for Doctoral Training on Intelligent and Autonomous Machines and Systems (AIMS CDT), supervised by Yarin Gal. His current research interests include AI safety and applications of AI in medicine/biomedical research.

The recording for this NL Seminar talk will be posted on our USC/ISI YouTube page within 1-2 business days: https://www.youtube.com/user/USCISI.

Subscribe here to learn more about upcoming seminars:

https://www.isi.edu/isi-seminar-series/