Model Robustness in Data-Scarce Regimes and the Effect of Frequency Perturbations

Abstract

The last decade has witnessed the meteoric rise of data-driven methods, which has been elevated to new heights thanks to the availability of powerful hardware and abundant data. Despite their swift ascension, deep learning methods are repeatedly shown to have robustness problems; they can be tricked into making errors with minor changes in the input that are invisible to us humans, or they can not withstand certain failure modes common in real-life scenarios. This thesis focuses on the robust generalization problem, where two primary aims drive our research effort.

First, inspired from the surprising lack of thorough discussions on robust generalization in data-scarce regimes, we perform an exhaustive analyses on the robustness behaviour of models trained in zero-shot learning settings. We first show that discriminative zero-shot models have distinct robustness characteristics against adversaries, such as unseen and seen classes being affected disproportionately, the effect of original model accuracy and the stark differences between how zero-shot and generalized zero-shot accuracies degrade. We also identify the unique pseudo-robustness effect caused by adversaries, where models might be falsely declared as robust. We then extend our analyses to a more practical scenario, where images are corrupted with common image corruptions. We curate and present the first three datasets for corruption robustness analyses in the zero-shot literature. Using these datasets, we provide a set of rigorous analyses with a wider range of zero-shot models to assess their robustness against corruptions. Our results show that with key augmentation choices, we can improve the performance profiles of various models. Finally, we aggregate the results of adversarial and corruption robustness behaviours of zero-shot models and conclude with a thorough comparison.

Second, inspired by the fundamental techniques in image processing, we focus on using frequency-spectra information to improve model robustness. Assuming that the true label information of an image resides in its low-frequency components, we propose HybridAugment where images are augmented by randomly swapping their high-frequency component with other images. This augmentation is implemented in tandem with existing augmentations, and enforces the network to be less reliant on high-frequency information, which is a prime reason for model robustness issues. We then propose two variants of HybridAugment, where single or multiple image settings are used to perform the augmentation. With single and multi image augmentations being used at the same time, the results are further improved. Finally, inspired by the two orthogonal frequency-centric analyses (i.e. frequency bands and phase/amplitude decomposition) and the need to unify them, we propose HybridAugment++ that performs a hierarchical augmentation in the frequency-spectra. In addition to swapping low and high-frequency components of images, HybridAugment++ also swaps phase and amplitude of random images, but does so only on the low-frequency components. HybridAugment++, with its single and paired variants working in tandem, achieves state-of-the-art results in multiple benchmark datasets, showing its effectiveness.