Search Results for author:
Found 7 papers, 2 papers with code
Curvature-corrected learning dynamics in deep neural networks
However, block-diagonal approximations of natural gradient, which are widely used in most second order methods (e. g. K-FAC), significantly distort the dynamics to follow highly divergent paths, destroying weight balance across layers.
Discovering Symmetry Group Structures via Implicit Orthogonality Bias
We introduce the HyperCube network, a novel approach for autonomously discovering symmetry group structures within data.
ISAAC Newton: Input-based Approximate Curvature for Newton's Method
We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons.
The Missing Invariance Principle Found -- the Reciprocal Twin of Invariant Risk Minimization
Further, we introduce a simplified, practical version of the MRI formulation called MRI-v1.
Few-Shot Multi-task Learning via Implicit regularization
However, FSL problems proves to be significantly more challenging and require more compute expensive process to optimize.
EXACT ANALYSIS OF CURVATURE CORRECTED LEARNING DYNAMICS IN DEEP LINEAR NETWORKS
Deep neural networks exhibit complex learning dynamics due to the highly non-convex loss landscape, which causes slow convergence and vanishing gradient problems.
Gradient Descent for Spiking Neural Networks
Much of studies on neural computation are based on network models of static neurons that produce analog output, despite the fact that information processing in the brain is predominantly carried out by dynamic neurons that produce discrete pulses called spikes.
