Search Results for author:
Found 3 papers, 2 papers with code
Analysing The Impact of Sequence Composition on Language Model Pre-Training
In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks.
Structured Packing in LLM Training Improves Long Context Utilization
Recent developments in long-context large language models have attracted considerable attention.
Focused Transformer: Contrastive Training for Context Scaling
This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
