Partially View-aligned Representation Learning with Noise-robust Contrastive Loss

In real-world applications, it is common that only a portion of data is aligned across views due to spatial, temporal, or spatiotemporal asynchronism, thus leading to socalled Partially View-aligned Problem (PVP). To solve such a less-touched problem without the help of labels, we propose simultaneously learning representation and aligning data using a noise-robust contrastive loss. In brief, for each sample from one view, our method aims to identify its within-category counterparts from other views, and thus the cross-view correspondence could be established. As the contrastive learning needs data pairs as input, we construct positive pairs using the known correspondences and negative pairs using random sampling. To alleviate or even eliminate the influence of the false negatives caused by random sampling, we propose a noise-robust contrastive loss which could adaptively prevent the false negatives from dominating the network optimization. Different from the Hungarian algorithm and its variants, our solution to PVP aims to achieve category- instead of instance-level alignment. Thanks to the higher accessibility and scalability of the category-level alignment, it is more desirable for the tasks such as clustering and classification. In addition, to the best of our knowledge, this could be the first successful attempt of enabling contrastive learning robust to noisy labels. Extensive experiments show the promising performance of our method comparing with 10 state-of-the-art multi-view approaches in the clustering and classification tasks.