Language Models
Galactica
Introduced by Taylor et al. in Galactica: A Large Language Model for ScienceGalactica is a language model which uses a Transformer architecture in a decoder-only setup with the following modifications:
- It uses GeLU activations on all model sizes
- It uses a 2048 length context window for all model sizes
- It does not use biases in any of the dense kernels or layer norms
- It uses learned positional embeddings for the model
- A vocabulary of 50k tokens was constructed using BPE. The vocabulary was generated from a randomly selected 2% subset of the training data
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Question Answering | 5 | 13.51% |
| Language Modelling | 3 | 8.11% |
| Large Language Model | 2 | 5.41% |
| Domain Adaptation | 2 | 5.41% |
| Common Sense Reasoning | 2 | 5.41% |
| Fairness | 2 | 5.41% |
| Document Classification | 1 | 2.70% |
| General Knowledge | 1 | 2.70% |
| Philosophy | 1 | 2.70% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
