5 code implementations • 27 Jun 2023 • Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B.
no code implementations • NeurIPS 2014 • Shouyuan Chen, Tian Lin, Irwin King, Michael R. Lyu, Wei Chen
We also establish a general problem-dependent lower bound for the CPE problem.
no code implementations • NeurIPS 2013 • Shouyuan Chen, Michael R. Lyu, Irwin King, Zenglin Xu
For the noisy cases, we also prove error bounds for a constrained convex program for recovering the tensors.