Function Vectors in Large Language Models

no code implementations • 23 Oct 2023 • Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, David Bau

Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV).

In-Context Learning

Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)

1 code implementation • 10 May 2023 • Chantal Shaib, Millicent L. Li, Sebastian Joseph, Iain J. Marshall, Junyi Jessy Li, Byron C. Wallace

Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings.