1 code implementation • 28 Aug 2023 • Samuel Chun-Hei Lam, Justin Sirignano, Konstantinos Spiliopoulos
Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity.
no code implementations • 14 Feb 2023 • Benjamin J. Zhang, Youssef M. Marzouk, Konstantinos Spiliopoulos
We show that in continuous time, when a transport map is applied to Langevin dynamics, the result is a Riemannian manifold Langevin dynamics (RMLD) with metric defined by the transport map.
1 code implementation • 2 Sep 2022 • Jiahui Yu, Konstantinos Spiliopoulos
A given layer $i$ with $N_{i}$ hidden units is allowed to be normalized by $1/N_{i}^{\gamma_{i}}$ with $\gamma_{i}\in[1/2, 1]$ and we study the effect of the choice of the $\gamma_{i}$ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set.
no code implementations • 18 Aug 2021 • Benjamin J. Zhang, Youssef M. Marzouk, Konstantinos Spiliopoulos
We introduce a novel geometry-informed irreversible perturbation that accelerates convergence of the Langevin algorithm for Bayesian computation.
no code implementations • 18 May 2021 • Justin Sirignano, Jonathan MacArt, Konstantinos Spiliopoulos
Recent research has used deep learning to develop partial differential equation (PDE) models in science and engineering.
1 code implementation • 20 Nov 2020 • Jiahui Yu, Konstantinos Spiliopoulos
In addition, we show that to leading order in $N$, the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches the mean field normalization.
no code implementations • 13 Nov 2019 • Justin Sirignano, Konstantinos Spiliopoulos
In addition, we study the convergence of the limit differential equation to the stationary solution.
no code implementations • 9 Jul 2019 • Justin Sirignano, Konstantinos Spiliopoulos
We analyze single-layer neural networks with the Xavier initialization in the asymptotic regime of large numbers of hidden units and large numbers of stochastic gradient descent training steps.
no code implementations • 11 Mar 2019 • Justin Sirignano, Konstantinos Spiliopoulos
The limit procedure is valid for any number of hidden layers and it naturally also describes the limiting behavior of the training loss.
no code implementations • 5 Dec 2018 • Konstantinos Spiliopoulos
We use relative entropy ideas to analyze the behavior of the algorithm as a function of the threshold parameter and of the size of the data.
no code implementations • 28 Aug 2018 • Justin Sirignano, Konstantinos Spiliopoulos
We rigorously prove a central limit theorem for neural network models with a single hidden layer.
no code implementations • 11 Oct 2017 • Justin Sirignano, Konstantinos Spiliopoulos
Stochastic gradient descent in continuous time (SGDCT) provides a computationally efficient method for the statistical learning of continuous-time models, which are widely used in science, engineering, and finance.
8 code implementations • 24 Aug 2017 • Justin Sirignano, Konstantinos Spiliopoulos
The algorithm is tested on a class of high-dimensional free boundary PDEs, which we are able to accurately solve in up to $200$ dimensions.
no code implementations • 17 Nov 2016 • Justin Sirignano, Konstantinos Spiliopoulos
Stochastic gradient descent in continuous time (SGDCT) provides a computationally efficient method for the statistical learning of continuous-time models, which are widely used in science, engineering, and finance.