A Parametric Rectified Linear Unit, or PReLU, is an activation function that generalizes the traditional rectified unit with a slope for negative values. Formally:

$$f\left(y_{i}\right) = y_{i} \text{ if } y_{i} \ge 0$$ $$f\left(y_{i}\right) = a_{i}y_{i} \text{ if } y_{i} \leq 0$$

The intuition is that different layers may require different types of nonlinearity. Indeed the authors find in experiments with convolutional neural networks that PReLus for the initial layer have more positive slopes, i.e. closer to linear. Since the filters of the first layers are Gabor-like filters such as edge or texture detectors, this shows a circumstance where positive and negative responses of filters are respected. In contrast the authors find deeper layers have smaller coefficients, suggesting the model becomes more discriminative at later layers (while it wants to retain more information at earlier layers).

Source: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Latest Papers

PAPER DATE
A Crowdsourced Open-Source Kazakh Speech Corpus and Initial Speech Recognition Baseline
Yerbolat KhassanovSaida MussakhojayevaAlmas MirzakhmetovAlen AdiyevMukhamet NurpeiissovHuseyin Atakan Varol
2020-09-22
Reinforced Wasserstein Training for Severity-Aware Semantic Segmentation in Autonomous Driving
Xiaofeng LiuYimeng ZhangXiongchang LiuSong BaiSite LiJane You
2020-08-11
Frame-To-Frame Consistent Semantic Segmentation
| Manuel RebolPatrick Knöbelreiter
2020-08-03
Funnel Activation for Visual Recognition
| Ningning MaXiangyu ZhangJian Sun
2020-07-23
Journey Towards Tiny Perceptual Super-Resolution
Royson LeeŁukasz DudziakMohamed AbdelfattahStylianos I. VenierisHyeji KimHongkai WenNicholas D. Lane
2020-07-08
Deep Learning-based Aerial Image Segmentation with Open Data for Disaster Impact Assessment
Ananya GuptaSimon WatsonHujun Yin
2020-06-10
Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
Vijay Pandey
2020-06-04
Perceptual Extreme Super Resolution Network with Receptive Field Block
Taizhang ShangQiuju DaiShengchen ZhuTong YangYandong Guo
2020-05-26
Arbitrary Scale Super-Resolution for Brain MRI Images
Chuan TanJin ZhuPietro Lio'
2020-04-05
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
| Zechun LiuZhiqiang ShenMarios SavvidesKwang-Ting Cheng
2020-03-07
EndoL2H: Deep Super-Resolution for Capsule Endoscopy
| Yasin AlmaliogluKutsev Bengisu OzyorukAbdulkadir GokceKagan IncetanGuliz Irem GokcelerMuhammed Ali SimsekKivanc AraratRichard J. ChenNicholas J. DurrFaisal MahmoodMehmet Turan
2020-02-13
Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant
Shayan AziznejadHarshit GuptaJoaquim CamposMichael Unser
2020-01-17
An Application of Generative Adversarial Networks for Super Resolution Medical Imaging
Rewa SoodBinit TopiwalaKarthik ChoutaguntaRohit SoodMirabela Rusu
2019-12-19
Anisotropic Super Resolution in Prostate MRI using Super Resolution Generative Adversarial Networks
Rewa SoodMirabela Rusu
2019-12-19
ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
| Tomoki HayashiRyuichi YamamotoKatsuki InoueTakenori YoshimuraShinji WatanabeTomoki TodaKazuya TakedaYu ZhangXu Tan
2019-10-24
Comparison of UNet, ENet, and BoxENet for Segmentation of Mast Cells in Scans of Histological Slices
Alexander KarimovArtem RazumovRuslana ManbatchurinaKsenia SimonovaIrina DonetsAnastasia VlasovaYulia KhramtsovaKonstantin Ushenin
2019-09-15
Learning Lightweight Lane Detection CNNs by Self Attention Distillation
| Yuenan HouZheng MaChunxiao LiuChen Change Loy
2019-08-02
Image Super-Resolution Using a Wavelet-based Generative Adversarial Network
Qi ZhangHuafeng WangSichen Yang
2019-07-24
Boosting Resolution and Recovering Texture of micro-CT Images with Deep Learning
Ying Da WangRyan T. ArmstrongPeyman Mostaghimi
2019-07-15
Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text
Murali Karthick BaskarShinji WatanabeRamon AstudilloTakaaki HoriLukáš BurgetJan Černocký
2019-04-30
DENet: A Universal Network for Counting Crowd with Varying Densities and Scales
Lei LiuJie JiangWenjing JiaSaeed AmirgholipourMichelle ZeibotsXiangjian He
2019-04-17
SRGAN: Training Dataset Matters
Nao TakanoGita Alaghband
2019-03-24
Real time backbone for semantic segmentation
Zhengeng YangHongshan YuQiang FuWei SunWenyan JiaMingui SunZhi-Hong Mao
2019-03-16
SREdgeNet: Edge Enhanced Single Image Super Resolution using Dense Edge Detection Network and Feature Merge Network
Kwanyoung KimSe Young Chun
2018-12-18
C3: Concentrated-Comprehensive Convolution and its application to semantic segmentation
| Hyojin ParkYoungjoon YooGeonseok SeoDongyoon HanSangdoo YunNojun Kwak
2018-12-12
DSNet for Real-Time Driving Scene Semantic Segmentation
Wenfu WangZhijie Pan
2018-12-06
Efficient Semantic Segmentation for Visual Bird's-eye View Interpretation
Timo SämannKarl AmendeStefan MilzChristian WittMartin SimonJohannes Petzold
2018-11-29
ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network
| Sachin MehtaMohammad RastegariLinda ShapiroHannaneh Hajishirzi
2018-11-28
Bi-GANs-ST for Perceptual Image Super-resolution
Xiaotong LuoRong ChenYuan XieYanyun QuCuihua Li
2018-11-01
SNAP: A semismooth Newton algorithm for pathwise optimization with optimal local convergence rate and oracle properties
Jian HuangYuling JiaoXiliang LuYueyong ShiQinglong Yang
2018-10-09
Super-Resolution via Conditional Implicit Maximum Likelihood Estimation
Ke LiShichong PengJitendra Malik
2018-10-02
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
| Xintao WangKe YuShixiang WuJinjin GuYihao LiuChao DongChen Change LoyYu QiaoXiaoou Tang
2018-09-01
Wide Activation for Efficient and Accurate Image Super-Resolution
| Jiahui YuYuchen FanJianchao YangNing XuZhaowen WangXinchao WangThomas Huang
2018-08-27
A Dataset of Laryngeal Endoscopic Images with Comparative Study on Convolution Neural Network Based Semantic Segmentation
| Max-Heinrich LavesJens BickerLüder A. KahrsTobias Ortmaier
2018-07-16
Mapping Road Lanes Using Laser Remission and Deep Neural Networks
Raphael V. CarneiroRafael C. NascimentoRânik GuidoliniVinicius B. CardosoThiago Oliveira-SantosClaudine BadueAlberto F. De Souza
2018-04-27
Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
| Xintao WangKe YuChao DongChen Change Loy
2018-04-09
ESPnet: End-to-End Speech Processing Toolkit
Shinji WatanabeTakaaki HoriShigeki KaritaTomoki HayashiJiro NishitobaYuya UnnoNelson Enrique Yalta SoplinJahn HeymannMatthew WiesnerNanxin ChenAdithya RenduchintalaTsubasa Ochiai
2018-03-30
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
| Sachin MehtaMohammad RastegariAnat CaspiLinda ShapiroHannaneh Hajishirzi
2018-03-19
Enhancing Batch Normalized Convolutional Networks using Displaced Rectifier Linear Units: A Systematic Comparative Study
David MacêdoCleber ZanchettinAdriano L. I. OliveiraTeresa Ludermir
2018-01-01
Computationally efficient cardiac views projection using 3D Convolutional Neural Networks
Matthieu LeJesse Lieman-SifryFelix LauSean SallAlbert HsiaoDaniel Golden
2017-11-03
Fast Scene Understanding for Autonomous Driving
| Davy NevenBert De BrabandereStamatios GeorgoulisMarc ProesmansLuc Van Gool
2017-08-08
FastVentricle: Cardiac Segmentation with ENet
Jesse Lieman-SifryMatthieu LeFelix LauSean SallDaniel Golden
2017-04-13
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
| Christian LedigLucas TheisFerenc HuszarJose CaballeroAndrew CunninghamAlejandro AcostaAndrew AitkenAlykhan TejaniJohannes TotzZehan WangWenzhe Shi
2016-09-15
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
| Adam PaszkeAbhishek ChaurasiaSangpil KimEugenio Culurciello
2016-06-07
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
| Kaiming HeXiangyu ZhangShaoqing RenJian Sun
2015-02-06

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories