Activation Functions

Activation functions are functions that we apply in neural networks after (typically) applying an affine transformation combining weights and input features. They are typically non-linear functions. The rectified linear unit, or ReLU, has been the most popular in the past decade, although the choice is architecture dependent and many alternatives have emerged in recent years. In this section, you will find a constantly updating list of activation functions.

METHOD YEAR PAPERS
ReLU
2000 4652
Sigmoid Activation
2000 3228
Tanh Activation
2000 3052
GELU
2016 1580
Leaky ReLU
2014 373
Swish
2017 70
PReLU
2015 46
GLU
2016 35
Maxout
2013 31
Softplus
2000 25
ELU
2015 19
SELU
2017 13
ReLU6
2017 11
Mish
2019 10
Softsign Activation
2000 10
Hard Swish
2019 10
CReLU
2016 5
Hard Sigmoid
2015 4
RReLU
2015 3
KAF
2017 3
SReLU
2015 2
modReLU
2015 2
Hermite Activation
2019 2
E-swish
2018 2
SiLU
2017 1
PELU
2016 1
ELiSH
2018 1
HardELiSH
2018 1
SERLU
2018 1
ARiA
2018 1
m-arcsinh
2020 1
Lecun's Tanh
1998 0
Hardtanh Activation
2000 0