Search Results for author:
Found 4 papers, 3 papers with code
LEAD: Liberal Feature-based Distillation for Dense Retrieval
Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model.
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
PROD: Progressive Distillation for Dense Retrieval
It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student.
Aligning the Pretraining and Finetuning Objectives of Language Models
We found that, with objective alignment, our 768 by 3 and 512 by 3 transformer language models can reach accuracy of 83. 9%/82. 5% for concept-of-interest tagging and 73. 8%/70. 2% for acronym detection using only 200 finetuning examples per task, outperforming the 768 by 3 model pretrained without objective alignment by +4. 8%/+3. 4% and +9. 9%/+6. 3%.
