Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism

1 code implementation • 16 Aug 2021 • Shulun Wang, Bin Liu, Feng Liu

Softmax is widely used in neural networks for multiclass classification, gate structure and attention mechanisms.