Swish

Introduced by Ramachandran et al. in Searching for Activation Functions

Swish is an activation function, $f(x) = x \cdot \text{sigmoid}(\beta x)$, where $\beta$ a learnable parameter. Nearly all implementations do not use the learnable parameter $\beta$, in which case the activation function is $x\sigma(x)$ ("Swish-1").

The function $x\sigma(x)$ is exactly the SiLU, which was introduced by other authors before the swish. See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the same activation function was experimented with later.

Source: Searching for Activation Functions

Latest Papers

PAPER DATE
EfficientHRNet: Efficient Scaling for Lightweight High-Resolution Multi-Person Pose Estimation
Christopher NeffAneri ShethSteven FurgursonHamed Tabkhi
2020-07-16
NVAE: A Deep Hierarchical Variational Autoencoder
Arash VahdatJan Kautz
2020-07-08
Improving accuracy and speeding up Document Image Classification through parallel systems
Javier FerrandoJuan Luis DominguezJordi TorresRaul GarciaDavid GarciaDaniel GarridoJordi CortadaMateo Valero
2020-06-16
Rethinking Pre-training and Self-training
| Barret ZophGolnaz GhiasiTsung-Yi LinYin CuiHanxiao LiuEkin D. CubukQuoc V. Le
2020-06-11
Deep Learning based Diagnosis of COVID-19 usingChest CT-scan Images
| Talha AnwarSeemab Zakir
2020-05-20
AI Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Etiology on Chest CT
| Harrison X. BaiRobin WangZeng XiongBen HsiehKen ChangKasey HalseyThi My Linh TranJi Whae ChoiDong-Cui WangLin-Bo ShiJi MeiXiao-Long JiangIan PanQiu-Hua ZengPing-Feng HuYi-Hui LiFei-Xian FuRaymond Y. HuangRonnie SebroQi-Zhi YuMichael K. AtalayWei-Hua Liao
2020-04-27
YOLOv4: Optimal Speed and Accuracy of Object Detection
| Alexey BochkovskiyChien-Yao WangHong-Yuan Mark Liao
2020-04-23
LSQ+: Improving low-bit quantization through learnable offsets and better initialization
| Yash BhalgatJinwon LeeMarkus NagelTijmen BlankevoortNojun Kwak
2020-04-20
DriftNet: Aggressive Driving Behavior Classification using 3D EfficientNet Architecture
Alam NoorBilel BenjdiraAdel AmmarAnis Koubaa
2020-04-18
Video Face Manipulation Detection Through Ensemble of CNNs
| Nicolò BonettiniEdoardo Daniele CannasSara MandelliLuca BondiPaolo BestaginiStefano Tubaro
2020-04-16
An Evaluation of DNN Architectures for Page Segmentation of Historical Newspapers
| Bernhard LieblManuel Burghardt
2020-04-15
Towards an Effective and Efficient Deep Learning Model for COVID-19 Patterns Detection in X-ray Images
| Eduardo LuzPedro Lopes SilvaRodrigo SilvaLudmila SilvaGladston MoreiraDavid Menotti
2020-04-12
Analysis on DeepLabV3+ Performance for Automatic Steel Defects Detection
Zheng NieJiachen XuShengchang Zhang
2020-04-09
Evolving Normalization-Activation Layers
| Hanxiao LiuAndrew BrockKaren SimonyanQuoc V. Le
2020-04-06
NBDT: Neural-Backed Decision Trees
| Alvin WanLisa DunlapDaniel HoJihan YinScott LeeHenry JinSuzanne PetrykSarah Adel BargalJoseph E. Gonzalez
2020-04-01
Designing Network Design Spaces
| Ilija RadosavovicRaj Prateek KosarajuRoss GirshickKaiming HePiotr Dollár
2020-03-30
Circumventing Outliers of AutoAugment with Knowledge Distillation
Longhui WeiAn XiaoLingxi XieXin ChenXiaopeng ZhangQi Tian
2020-03-25
Multi-Plateau Ensemble for Endoscopic Artefact Segmentation and Detection
| Suyog JadhavUdbhav BambaArnav ChavanRishabh TiwariAryan Raj
2020-03-23
Meta Pseudo Labels
Hieu PhamQizhe XieZihang DaiQuoc V. Le
2020-03-23
Fixing the train-test resolution discrepancy: FixEfficientNet
| Hugo TouvronAndrea VedaldiMatthijs DouzeHervé Jégou
2020-03-18
Gimme Signals: Discriminative signal encoding for multimodal activity recognition
| Raphael MemmesheimerNick TheisenDietrich Paulus
2020-03-13
Learned Threshold Pruning
Kambiz AzarianYash BhalgatJinwon LeeTijmen Blankevoort
2020-02-28
MaxUp: A Simple Way to Improve Generalization of Neural Network Training
Chengyue GongTongzheng RenMao YeQiang Liu
2020-02-20
A closer look at network resolution for efficient network design
Anonymous
2020-01-01
Learning Neural Activations
| Fayyaz ul Amir Afsar MinhasAmina Asif
2019-12-27
Attention-Based Face AntiSpoofing of RGB Images, using a Minimal End-2-End Neural Network
Ali GhofraniRahil Mahdian ToroghiSeyed Mojtaba Tabatabaie
2019-12-18
SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
| Xianzhi DuTsung-Yi LinPengchong JinGolnaz GhiasiMingxing TanYin CuiQuoc V. LeXiaodan Song
2019-12-10
CSPNet: A New Backbone that can Enhance Learning Capability of CNN
| Chien-Yao WangHong-Yuan Mark LiaoI-Hau YehYueh-Hua WuPing-Yang ChenJun-Wei Hsieh
2019-11-27
Structured Multi-Hashing for Model Compression
Elad EbanYair Movshovitz-AttiasHao WuMark SandlerAndrew PoonYerlan IdelbayevMiguel A. Carreira-Perpinan
2019-11-25
Adversarial Examples Improve Image Recognition
| Cihang XieMingxing TanBoqing GongJiang WangAlan YuilleQuoc V. Le
2019-11-21
Fast Sparse ConvNets
| Erich ElsenMarat DukhanTrevor GaleKaren Simonyan
2019-11-21
EfficientDet: Scalable and Efficient Object Detection
| Mingxing TanRuoming PangQuoc V. Le
2019-11-20
Experimental Exploration of Compact Convolutional Neural Network Architectures for Non-temporal Real-time Fire Detection
Ganesh Samarth C. A.Neelanjan BhowmikToby P. Breckon
2019-11-20
Self-training with Noisy Student improves ImageNet classification
| Qizhe XieMinh-Thang LuongEduard HovyQuoc V. Le
2019-11-11
Identification of primary angle-closure on AS-OCT images with Convolutional Neural Networks
Chenglang YuanCheng BianHongjian KangShu LiangKai MaYefeng Zheng
2019-10-23
ICPS-net: An End-to-End RGB-based Indoor Camera Positioning System using deep convolutional neural networks
Ali GhofraniRahil Mahdian ToroghiSayed Mojtaba Tabatabaie
2019-10-14
RandAugment: Practical automated data augmentation with a reduced search space
| Ekin D. CubukBarret ZophJonathon ShlensQuoc V. Le
2019-09-30
MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
| Taojiannan YangSijie ZhuChen ChenShen YanMi ZhangAndrew Willis
2019-09-27
K-TanH: Efficient TanH For Deep Learning
Abhisek KunduAlex HeineckeDhiraj KalamkarSudarshan SrinivasanEric C. QinNaveen K. MellempudiDipankar DasKunal BanerjeeBharat KaulPradeep Dubey
2019-09-17
Mish: A Self Regularized Non-Monotonic Neural Activation Function
| Diganta Misra
2019-08-23
Effect of Activation Functions on the Training of Overparametrized Neural Nets
Abhishek PanigrahiAbhishek ShettyNavin Goyal
2019-08-16
MixConv: Mixed Depthwise Convolutional Kernels
| Mingxing TanQuoc V. Le
2019-07-22
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
| Mingxing TanQuoc V. Le
2019-05-28
FSD: Feature Skyscraper Detector for Stem End and Blossom End of Navel Orange
Xiaoye SunGongyan LiShaoyun Xu
2019-05-24
CondConv: Conditionally Parameterized Convolutions for Efficient Inference
| Brandon YangGabriel BenderQuoc V. LeJiquan Ngiam
2019-04-10
LiSHT: Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks
| Swalpa Kumar RoySuvojit MannaShiv Ram DubeyBidyut B. Chaudhuri
2019-01-01
Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
| Hock Hung ChiengNoorhaniza WahidPauline OngSai Raj Kishore Perla
2018-12-15
Probabilistic DL Reasoning with Pinpointing Formulas: A Prolog-based Approach
Riccardo ZeseGiuseppe CotaEvelina LammaElena BellodiFabrizio Riguzzi
2018-09-17
Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)
G. ZhangH. Li
2018-07-26
ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets
Narendra PatwardhanMadhura IngalhalikarRahee Walambe
2018-05-22
Mean Field Theory of Activation Functions in Deep Neural Networks
| Mirco MilletaríThiparat ChotibutPaolo E. Trevisanutto
2018-05-22
On the Selection of Initialization and Activation Function for Deep Neural Networks
Soufiane HayouArnaud DoucetJudith Rousseau
2018-05-21
E-swish: Adjusting Activations to Different Network Depths
| Eric Alcaide
2018-01-22
Searching for Activation Functions
| Prajit RamachandranBarret ZophQuoc V. Le
2017-10-16

Categories