Abstract

Regularization is crucial to the success of many practical deep learning models, in particular in frequent scenarios where there are only a few to a moderate number of accessible training samples. In addition to weight decay, noise injection and dropout, regularization based on multi-branch architectures, such as Shake-Shake regularization, has been proven successful in many applications and attracted more and more attention. However, beyond model-based representation augmentation, it is unclear how Shake-Shake regularization helps to provide further improvement on classification tasks, let alone the baffling interaction between batch normalization and shaking. In this work, we present our investigation on Shake-Shake regularization. One of our findings illustrates the phenomenon that batch normalization in residual blocks is indispensable when shaking is applied to model branches, along with which we …

Date

2019

Authors

Che-Wei Huang, Shrikanth Narayanan

Conference

ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Pages

3322-3326

Publisher

IEEE