1 code implementation • 23 May 2022 • Emmanuel Brempong Asiedu, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, Mohammad Norouzi
We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder.