Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives

1 code implementation • 6 Jun 2022 • Wei Wang, Liangzhu Ge, Jingqiao Zhang, Cheng Yang

Following SimCSE, contrastive learning based methods have achieved the state-of-the-art (SOTA) performance in learning sentence embeddings.

Attribute Contrastive Learning +5

SAS: Self-Augmentation Strategy for Language Model Pre-training

1 code implementation • 14 Jun 2021 • Yifei Xu, Jingqiao Zhang, Ru He, Liangzhu Ge, Chao Yang, Cheng Yang, Ying Nian Wu

In this paper, we propose a self-augmentation strategy (SAS) where a single network is utilized for both regular pre-training and contextualized data augmentation for the training in later epochs.

Data Augmentation Language Modelling +2

Label-Based Diversity Measure Among Hidden Units of Deep Neural Networks: A Regularization Method

no code implementations • 19 Sep 2020 • Chenguang Zhang, Yuexian Hou, Dawei Song, Liangzhu Ge, Yaoshuai Yao

In this paper, from an information theoretic perspective, we introduce a new definition of redundancy to describe the diversity of hidden units under supervised learning settings by formalizing the effect of hidden layers on the generalization capacity as the mutual information.