FastBERT: a Self-distilling BERT with Adaptive Inference Time

3 code implementations • ACL 2020 • Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju

Pre-trained language models like BERT have proven to be highly performant.

K-BERT: Enabling Language Representation with Knowledge Graph

2 code implementations • arXiv 2019 • Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang

For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge.

Knowledge Graphs Sentence

UER: An Open-Source Toolkit for Pre-training Models

1 code implementation • IJCNLP 2019 • Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du

Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks.