Improving Paraphrase Generation models with machine translation generated pre-training

Paraphrase generation is a fundamental and longstanding problem in the Natural Language Processing field. With the huge success of pre-trained transformers, the pre-train–fine-tune approach has become a standard choice. At the same time, popular task-agnostic pre-trainings usually require terabyte datasets and hundreds of GPUs, while available pre-trained models are limited to architecture and size. We propose a simple and efficient pre-training approach specifically for paraphrase generation, which noticeably boosts model quality and doesn't require significant computing power. We also investigate how this procedure influences the scores across different architectures and show that it helps them all.