Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Pre-trained on Formally Diverse Data

no code implementations • 24 Jun 2023 • Alycia Lee, Brando Miranda, Sudharsan Sundar, Sanmi Koyejo

Current trends to pre-train capable Large Language Models (LLMs) mostly focus on scaling of model and dataset size.