OmniVec: Learning robust representations with cross modal sharing
Majority of research in learning based methods has been towards designing and training networks for specific tasks. However, many of the learning based tasks, across modalities, share commonalities and could be potentially tackled in a joint framework. We present an approach in such direction, to learn multiple tasks, in multiple modalities, with a unified architecture. The proposed network is composed of task specific encoders, a common trunk in the middle, followed by task specific prediction heads. We first pre-train it by self-supervised masked training, followed by sequential training for the different tasks. We train the network on all major modalities, e.g.\ visual, audio, text and 3D, and report results on $22$ diverse and challenging public benchmarks. We demonstrate empirically that, using a joint network to train across modalities leads to meaningful information sharing and this allows us to achieve state-of-the-art results on most of the benchmarks. We also show generalization of the trained network on cross-modal tasks as well as unseen datasets and tasks.
PDF Abstract
ImageNet
UCF101
ModelNet
Kinetics
Places
NYUv2
HMDB51
Kinetics 400
AudioSet
MSR-VTT
iNaturalist
SUN RGB-D
S3DIS
ESC-50
ScanObjectNN
YouCook2
Oxford-IIIT Pet Dataset
Places365
ModelNet40-C
