Search Results for author:
Found 1 papers, 0 papers with code
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
Large and sparse feed-forward layers (S-FFN) such as Mixture-of-Experts (MoE) have proven effective in scaling up Transformers model size for \textit{pretraining} large language models.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
