An information-theoretic method to automatic shortcut avoidance and domain generalization for dense prediction tasks

Document Type

Journal Article

Publication Title

IEEE Transactions on Pattern Analysis and Machine Intelligence

Publisher

IEEE

School

School of Science

RAS ID

58449

Comments

Chuah, W. Q., Tennakoon, R., Hoseinnezhad, R., Suter, D., & Bab- Hadiashar, A. (2023). An information-theoretic method to automatic shortcut avoidance and domain generalization for dense prediction tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9). https://doi.org/10.1109/TPAMI.2023.3268640

Abstract

Deep convolutional neural networks for dense prediction tasks are commonly optimized using synthetic data, as generating pixel-wise annotations for real-world data is laborious. However, the synthetically trained models do not generalize well to real-world environments. This poor “synthetic to real” (S2R) generalization we address through the lens of shortcut learning. We demonstrate that the learning of feature representations in deep convolutional networks is heavily influenced by synthetic data artifacts (shortcut attributes). To mitigate this issue, we propose an Information-Theoretic Shortcut Avoidance (ITSA) approach to automatically restrict shortcut-related information from being encoded into the feature representations. Specifically, our proposed method minimizes the sensitivity of latent features to input variations: to regularize the learning of robust and shortcut-invariant features in synthetically trained models. To avoid the prohibitive computational cost of direct input sensitivity optimization, we propose a practical yet feasible algorithm to achieve robustness. Our results show that the proposed method can effectively improve S2R generalization in multiple distinct dense prediction tasks, such as stereo matching, optical flow, and semantic segmentation. Importantly, the proposed method enhances the robustness of the synthetically trained networks and outperforms their fine-tuned counterparts (on real data) for challenging out-of-domain applications.

DOI

10.1109/TPAMI.2023.3268640

Access Rights

subscription content

Link to Full Text

Share

 
COinS