Search Results for author:
Found 10 papers, 2 papers with code
Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networks
Our results show that neural networks extract information from higher-order correlations in the spiked cumulant model efficiently, and reveal a large gap in the amount of data required by neural networks and random features to learn from higher-order cumulants.
Mapping of attention mechanisms to a generalized Potts model
In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word.
Optimal transfer protocol by incremental layer defrosting
The simplest transfer learning protocol is based on ``freezing" the feature-extractor layers of a network pre-trained on a data-rich source task, and then adapting only the last layers to a data-poor target task.
Bias-inducing geometries: an exactly solvable data model with fairness implications
Then, we consider a novel mitigation strategy based on a matched inference approach, consisting in the introduction of coupled learning models.
Gaussian Universality of Perceptrons with Random Labels
We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance.
Probing transfer learning with a model of synthetic correlated datasets
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task.
Generalisation error in learning with random features and the hidden manifold model
In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model.
Critical initialisation in continuous approximations of binary neural networks
Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.
On the role of synaptic stochasticity in training low-precision neural networks
Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes.
Learning may need only a few bits of synaptic precision
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states.
