ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

no code implementations • 18 May 2022 • Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, Haifeng Wang

Our method 1) introduces a self on-the-fly distillation method that can effectively distill late interaction (i. e., ColBERT) to vanilla dual-encoder, and 2) incorporates a cascade distillation process to further improve the performance with a cross-encoder teacher.

Knowledge Distillation Open-Domain Question Answering +2

ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps

no code implementations • 17 Mar 2022 • Jizhou Huang, Haifeng Wang, Yibo Sun, Yunsheng Shi, Zhengjie Huang, An Zhuo, Shikun Feng

One of the main reasons for this plateau is the lack of readily available geographic knowledge in generic PTMs.

NOTE: Solution for KDD-CUP 2021 WikiKG90M-LSC

no code implementations • 5 Jul 2021 • Weiyue Su, Zeyang Fang, Hui Zhong, Huijuan Wang, Siming Dai, Zhengjie Huang, Yunsheng Shi, Shikun Feng, Zeyu Chen

In addition to the representations, we also use various statistical probabilities among the head entities, the relations and the tail entities for the final prediction.

Feature Engineering Question Answering +2

Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification

3 code implementations • 8 Sep 2020 • Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, Yu Sun

Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification.

General Classification Node Classification +1