Search Results for author:
Found 6 papers, 0 papers with code
The Nonlinearity Coefficient -- A Practical Guide to Neural Architecture Design
We argue that the NLC is the most powerful scalar statistic for architecture design specifically and neural network analysis in general.
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
Via an extensive empirical study, we show that the NLC is a powerful predictor of test error and that attaining a right-sized NLC is essential for optimal performance.
Gradients explode - Deep Networks are shallow - ResNet explained
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.
The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.
Nonparametric Neural Networks
Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch.
Stability Selection for Structured Variable Selection
Recently, a meta-algorithm called Stability Selection was proposed that can provide reliable finite-sample control of the number of false positives.
