Weight Normalization is a normalization method for training neural networks. It is inspired by batch normalization, but it is a deterministic method that does not share batch normalization's property of adding noise to the gradients. It reparameterizes each weight vector $\textbf{w}$ in terms of a parameter vector $\textbf{v}$ and a scalar parameter $g$ and to perform stochastic gradient descent with respect to those parameters instead. Weight vectors are expressed in terms of the new parameters using:

$$ \textbf{w} = \frac{g}{\Vert\textbf{v}\Vert}\textbf{v}$$

where $\textbf{v}$ is a $k$-dimensional vector, $g$ is a scalar, and $\Vert\textbf{v}\Vert$ denotes the Euclidean norm of $\textbf{v}$. This reparameterization has the effect of fixing the Euclidean norm of the weight vector $\textbf{w}$: we now have $\Vert\textbf{w}\Vert = g$, independent of the parameters $\textbf{v}$.

Source: Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Latest Papers

PAPER DATE
SpeedySpeech: Efficient Neural Speech Synthesis
Jan VainerOndřej Dušek
2020-08-09
Learning from a Complementary-label Source Domain: Theory and Algorithms
Yiyang ZhangFeng LiuZhen FangBo YuanGuangquan ZhangJie Lu
2020-08-04
VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network
Jinhyeok YangJunmo LeeYoungik KimHoonyoung ChoInjung Kim
2020-07-30
Clarinet: A One-step Approach Towards Budget-friendly Unsupervised Domain Adaptation
| Yiyang ZhangFeng LiuZhen FangBo YuanGuangquan ZhangJie Lu
2020-07-29
NVAE: A Deep Hierarchical Variational Autoencoder
Arash VahdatJan Kautz
2020-07-08
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
Jeremiah Zhe LiuZi LinShreyas PadhyDustin TranTania Bedrax-WeissBalaji Lakshminarayanan
2020-06-17
New Interpretations of Normalization Methods in Deep Learning
Jiacheng SunXiangyong CaoHanwen LiangWeiran HuangZewei ChenZhenguo Li
2020-06-16
Adversarial representation learning for private speech generation
| David EricssonAdam ÖstbergEdvin Listo ZecJohn MartinssonOlof Mogren
2020-06-16
Weighted Optimization: better generalization by smoother interpolation
Yuege XieRachel WardHolger RauhutHung-Hsu Chou
2020-06-15
SE-MelGAN -- Speaker Agnostic Rapid Speech Enhancement
Luka ChkhetianiLevan Bejanidze
2020-06-13
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
Yonatan DuklerQuanquan GuGuido Montúfar
2020-06-11
CLARINET: A RISC-V Based Framework for Posit Arithmetic Empiricism
Riya JainNiraj SharmaFarhad MerchantSachin PatkarRainer Leupers
2020-05-30
Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
| Geng YangShan YangKai LiuPeng FangWei ChenLei Xie
2020-05-11
Single-Side Domain Generalization for Face Anti-Spoofing
| Yunpei JiaJie ZhangShiguang ShanXilin Chen
2020-04-29
Parallel Neural Text-to-Speech
Kainan PengWei PingZhao SongKexin Zhao
2020-01-01
Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
Brendan RuffTaylor BeckJoscha Bach
2019-11-29
Implicit Regularization of Normalization Methods
Xiaoxia WuEdgar DobribanTongzheng RenShanshan WuZhiyuan LiSuriya GunasekarRachel WardQiang Liu
2019-11-18
Understanding the Disharmony between Weight Normalization Family and Weight Decay: $ε-$shifted $L_2$ Regularizer
Li XiangChen ShuoXia YanYang Jian
2019-11-14
Systematic Comparison of the Influence of Different Data Preprocessing Methods on the Performance of Gait Classifications Using Machine Learning
Johannes BurdackFabian HorstSven GiesselbachIbrahim HassanSabrina DaffnerWolfgang I. Schöllhorn
2019-11-11
Inherent Weight Normalization in Stochastic Neural Networks
| Georgios DetorakisSourav DuttaAbhishek KhannaMatthew JerrySuman DattaEmre Neftci
2019-10-27
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
| Kundan KumarRithesh KumarThibault de BoissiereLucas GestinWei Zhen TeohJose SoteloAlexandre de BrebissonYoshua BengioAaron Courville
2019-10-08
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
| Yuhang LiXin DongWei Wang
2019-09-28
Adversarial Lipschitz Regularization
| Dávid Terjék
2019-07-12
Multi-Speaker End-to-End Speech Synthesis
Jihyun ParkKexin ZhaoKainan PengWei Ping
2019-07-09
Weight Normalization based Quantization for Deep Neural Network Compression
Wen-Pu CaiWu-Jun Li
2019-07-01
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets
| Devansh ArpitVictor CamposYoshua Bengio
2019-06-05
Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
| Kenta OonoTaiji Suzuki
2019-05-27
Non-Autoregressive Neural Text-to-Speech
| Kainan PengWei PingZhao SongKexin Zhao
2019-05-21
Neural source-filter waveform models for statistical parametric speech synthesis
Xin WangShinji TakakiJunichi Yamagishi
2019-04-27
FloWaveNet : A Generative Flow for Raw Audio
Sungwon KimSang-gil LeeJongyoon SongSungroh Yoon
2018-11-06
WaveGlow: A Flow-based Generative Network for Speech Synthesis
| Ryan PrengerRafael ValleBryan Catanzaro
2018-10-31
Fine-tuning on Clean Data for End-to-End Speech Translation: FBK @ IWSLT 2018
Mattia Antonino Di GangiRoberto DessìRoldano CattoniMatteo NegriMarco Turchi
2018-10-16
The RWTH Aachen University English-German and German-English Unsupervised Neural Machine Translation Systems for WMT 2018
Miguel Gra{\c{c}}aYunsu KimJulian SchamperJiahui GengHermann Ney
2018-10-01
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
| Wei PingKainan PengJitong Chen
2018-07-19
Orthogonal Weight Normalization: Solution to Optimization overMultiple Dependent Stiefel Manifolds in Deep Neural Networks
| Lei HuangXianglong LiuBo LangAdams Wei YuYongliang WangBo Li
2018-02-02
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima
Simon S. DuJason D. LeeYuandong TianBarnabas PoczosAarti Singh
2017-12-03
Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
| Wei PingKainan PengAndrew GibianskySercan O. ArikAjay KannanSharan NarangJonathan RaimanJohn Miller
2017-10-20
Projection Based Weight Normalization for Deep Neural Networks
| Lei HuangXianglong LiuBo LangBo Li
2017-10-06
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
Igor GitmanBoris Ginsburg
2017-09-24
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
| Lei HuangXianglong LiuBo LangAdams Wei YuYongliang WangBo Li
2017-09-16
Churn Identification in Microblogs using Convolutional Neural Networks with Structured Logical Knowledge
Mourad GridachHatem HaddadHala Mulki
2017-09-01
On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
| Sitao XiangHao Li
2017-04-13
Multiplicative LSTM for sequence modelling
Ben KrauseLiang LuIain MurraySteve Renals
2016-09-26
Improved Techniques for Training GANs
| Tim SalimansIan GoodfellowWojciech ZarembaVicki CheungAlec RadfordXi Chen
2016-06-10
Density estimation using Real NVP
| Laurent DinhJascha Sohl-DicksteinSamy Bengio
2016-05-27
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
| Tim SalimansDiederik P. Kingma
2016-02-25

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories