Search Results for author:
Found 5 papers, 0 papers with code
ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Comparing with linear neural networks, we show that a ReLU activated wide neural network at random initialization has a larger angle separation for similar data in the feature space of model gradient, and has a smaller condition number for NTK.
Cut your Losses with Squentropy
We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy.
Limitations of Neural Collapse for Understanding Generalization in Deep Learning
We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property).
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement
We apply a fast kernel method for mask-based single-channel speech enhancement.
