Search Results for author:
Found 8 papers, 1 papers with code
MLP-Mixer as a Wide and Sparse MLP
Multi-layer perceptron (MLP) is a fundamental component of deep learning that has been extensively employed for various problems.
Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias
In this study, we first reveal that a specific finite-difference computation, composed of both gradient ascent and descent steps, reduces the computational cost of GR.
Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case
Lastly, we can replace each weight with a Haar orthogonal random matrix independent of the Jacobian of the activation function using this key fact.
Layer-Wise Interpretation of Deep Neural Networks Using Identity Initialization
The interpretability of neural networks (NNs) is a challenging but essential topic for transparency in the decision-making process using machine learning.
Selective Forgetting of Deep Networks at a Finer Level than Samples
Such forgetting is crucial also in a practical sense since the deployed DNNs may be trained on the data with outliers, poisoned by attackers, or with leaked/sensitive information.
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
We investigate the spectral distribution of the conditional FIM, which is the FIM given a single sample, by focusing on fully-connected networks achieving dynamical isometry.
Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks.
Cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents
The method is based on the spectral distribution instead of the traditional likelihood.
