Outlier Dimensions Encode Task-Specific Knowledge

1 code implementation • 26 Oct 2023 • William Rudman, Catherine Chen, Carsten Eickhoff

Representations from large language models (LLMs) are known to be dominated by a small subset of dimensions with exceedingly high variance.

Stable Anisotropic Regularization

1 code implementation • 30 May 2023 • William Rudman, Carsten Eickhoff

Given the success of Large Language Models (LLMs), there has been considerable interest in studying the properties of model activations.

Garden-Path Traversal in GPT-2

1 code implementation • 24 May 2022 • William Jurayj, William Rudman, Carsten Eickhoff

In recent years, large-scale transformer decoders such as the GPT-x family of models have become increasingly popular.

Decoder Language Modelling

IsoScore: Measuring the Uniformity of Embedding Space Utilization

1 code implementation • Findings (ACL) 2022 • William Rudman, Nate Gillman, Taylor Rayne, Carsten Eickhoff

We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space.

Word Embeddings