no code implementations • ICML 2020 • Dongsung Huh
However, block-diagonal approximations of natural gradient, which are widely used in most second order methods (e. g. K-FAC), significantly distort the dynamics to follow highly divergent paths, destroying weight balance across layers.
no code implementations • 26 Feb 2024 • Dongsung Huh
We introduce the HyperCube network, a novel approach for autonomously discovering symmetry group structures within data.
1 code implementation • 1 May 2023 • Felix Petersen, Tobias Sutter, Christian Borgelt, Dongsung Huh, Hilde Kuehne, Yuekai Sun, Oliver Deussen
We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons.
1 code implementation • 29 May 2022 • Dongsung Huh, Avinash Baidya
Further, we introduce a simplified, practical version of the MRI formulation called MRI-v1.
no code implementations • 29 Sep 2021 • Dongsung Huh
However, FSL problems proves to be significantly more challenging and require more compute expensive process to optimize.
no code implementations • 25 Sep 2019 • Dongsung Huh
Deep neural networks exhibit complex learning dynamics due to the highly non-convex loss landscape, which causes slow convergence and vanishing gradient problems.
no code implementations • NeurIPS 2018 • Dongsung Huh, Terrence J. Sejnowski
Much of studies on neural computation are based on network models of static neurons that produce analog output, despite the fact that information processing in the brain is predominantly carried out by dynamic neurons that produce discrete pulses called spikes.