InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining

1 code implementation • 11 Oct 2023 • Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro

After instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on a wide range of zero-shot tasks.

Question Answering Reading Comprehension +2

Retrieval meets Long Context Large Language Models

no code implementations • 4 Oct 2023 • Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro

Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation.

16k 4k +4

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study

1 code implementation • 13 Apr 2023 • Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro

Thus, it is still an open question: shall we pretrain large autoregressive LMs with retrieval?

Open-Ended Question Answering Retrieval +1

Reducing Activation Recomputation in Large Transformer Models

3 code implementations • 10 May 2022 • Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro

In this paper, we show how to significantly accelerate training of large transformer models by reducing activation recomputation.

Utilizing Static Analysis and Code Generation to Accelerate Neural Networks

no code implementations • 27 Jun 2012 • Lawrence McAfee, Kunle Olukotun

In this paper, we present SONNC, a compiler for NNs that utilizes static analysis to generate optimized parallel code.

C++ code Code Generation