no code implementations • FL4NLP (ACL) 2022 • Xinwei Wu, Li Gong, Deyi Xiong
Although differential privacy (DP) can protect language models from leaking privacy, its indiscriminative protection on all data points reduces its practical utility.
1 code implementation • 28 Feb 2024 • Shaoyang Xu, Weilong Dong, Zishan Guo, Xinwei Wu, Deyi Xiong
Drawing from our findings on multilingual value alignment, we prudently provide suggestions on the composition of multilingual data for LLMs pre-training: including a limited number of dominant languages for cross-lingual alignment transfer while avoiding their excessive prevalence, and keeping a balanced distribution of non-dominant languages.
1 code implementation • 31 Oct 2023 • Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, Deyi Xiong
The ability of data memorization and regurgitation in pretrained language models, revealed in previous studies, brings the risk of data leakage.
no code implementations • 26 Sep 2023 • Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, Deyi Xiong
We also envision bridging the gap between the AI alignment research community and the researchers engrossed in the capability exploration of LLMs for both capable and safe LLMs.
no code implementations • 16 Dec 2022 • Weilong Dong, Xinwei Wu, Junzhuo Li, Shuangzhi Wu, Chao Bian, Deyi Xiong
It broadcasts the global model in the server to each client and produces pseudo data for clients so that knowledge from the global model can be explored to enhance few-shot learning of each client model.
no code implementations • 16 Dec 2022 • Junzhuo Li, Xinwei Wu, Weilong Dong, Shuangzhi Wu, Chao Bian, Deyi Xiong
Knowledge distillation (KD) has been widely used for model compression and knowledge transfer.