# Attention as Activation

15 Jul 2020

Activation functions and attention mechanisms are typically treated as having different purposes and have evolved differently. However, both concepts can be formulated as a non-linear gating function... (read more)

PDF Abstract