UniLMv2 introduces a novel training procedure, PMLM, which enables efficient learning of inter-relations between corrupted tokens and context via autoencoding, as well as intra-relations between masked spans via partially autoregressive modeling, significantly advancing the capabilities of language models in diverse NLP tasks. Overview of PMLM pre-training. The model parameters are shared across the LM objectives. … Continue reading Paper – UniLMv2