Search Results for author:
Found 30 papers, 11 papers with code
Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically
Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures without explicitly encoding any structural bias.
Guiding Language Models of Code with Global Context using Monitors
We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it.
Ranked #1 on
Code Completion
on DotPrompts
In-Context Learning through the Bayesian Prism
One of the main discoveries in this line of research has been that for several function classes, such as linear regression, transformers successfully generalize to new functions in the class.
A Theory of Emergent In-Context Learning as Implicit Structure Induction
Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations.
Towards a Mathematics Formalisation Assistant using Large Language Models
Mathematics formalisation is the task of writing mathematics (i. e., definitions, theorem statements, proofs) in natural language, as found in books and papers, into a formal language that can then be checked for correctness by a program.
When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks
On analyzing the task, we find that identifying the target location in the grid world is the main challenge for the models.
Revisiting the Compositional Generalization Abilities of Neural Sequence Models
Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences.
Learning and Generalization in Overparameterized Normalizing Flows
In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using stochastic gradient descent with a sufficiently small learning rate and suitable initialization.
Learning and Generalization in RNNs
In contrast to the previous work that could only deal with functions of sequences that are sums of functions of individual tokens in the sequence, we allow general functions.
Analyzing the Nuances of Transformers' Polynomial Simplification Abilities
To understand Transformers' abilities in such tasks in a fine-grained manner, we deviate from traditional end-to-end settings, and explore a step-wise polynomial simplification task.
Are NLP Models really able to Solve Simple Math Word Problems?
Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs.
Ranked #1 on
Math Word Problem SolvingΩ
on MAWPS
Do Transformers Understand Polynomial Simplification?
For a polynomial which is not necessarily in this normal form, a sequence of simplification steps is applied to reach the fully simplified (i. e., in the normal form) polynomial.
Learning and Generalization in Univariate Overparameterized Normalizing Flows
In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using Stochastic Gradient Descent (SGD).
On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages
We find that while recurrent models generalize nearly perfectly if the lengths of the training and test strings are from the same range, they perform poorly if the test strings are longer.
On the Ability and Limitations of Transformers to Recognize Formal Languages
Our analysis also provides insights on the role of self-attention mechanism in modeling certain behaviors and the influence of positional encoding schemes on the learning and generalization abilities of the model.
Robust Identifiability in Linear Structural Equation Models of Causal Inference
First, for a large and well-studied class of LSEMs, namely ``bow free'' models, we provide a sufficient condition on model parameters under which robust identifiability holds, thereby removing the restriction of paths required by prior work.
On the Computational Power of Transformers and its Implications in Sequence Modeling
Transformers are being used extensively across several sequence modeling tasks.
Non-Gaussianity of Stochastic Gradient Noise
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training?
Effect of Activation Functions on the Training of Overparametrized Neural Nets
In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks.
Universality Patterns in the Training of Neural Networks
This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.)
Stability of Linear Structural Equation Models of Causal Inference
First we prove that under a sufficient condition, for a certain sub-class of $\LSEM$ that are \emph{bow-free} (Brito and Pearl (2002)), the parameter recovery is stable.
Non-Gaussian Component Analysis using Entropy Methods
NGCA is also related to dimension reduction and to other data analysis problems such as ICA.
Depth separation and weight-width trade-offs for sigmoidal neural networks
We show a similar separation between the expressive power of depth-2 and depth-3 sigmoidal neural networks over a large class of input distributions, as long as the weights are polynomially bounded.
Learnability of Learned Neural Networks
This paper explores the simplicity of learned neural networks under various settings: learned on real vs random data, varying size/architecture and using large minibatch size vs small minibatch size.
Heavy-Tailed Analogues of the Covariance Matrix for ICA
Like the current state-of-the-art, the new algorithm is based on the centroid body (a first moment analogue of the covariance matrix).
Heavy-tailed Independent Component Analysis
Independent component analysis (ICA) is the problem of efficiently recovering a matrix $A \in \mathbb{R}^{n\times n}$ from i. i. d.
The More, the Merrier: the Blessing of Dimensionality for Learning Large Gaussian Mixtures
The problem of learning this map can be efficiently solved using some recent results on tensor decompositions and Independent Component Analysis (ICA), thus giving an algorithm for recovering the mixture.
Fourier PCA and Robust Tensor Decomposition
Fourier PCA is Principal Component Analysis of a matrix obtained from higher order derivatives of the logarithm of the Fourier transform of a distribution. We make this method algorithmic by developing a tensor decomposition method for a pair of tensors sharing the same vectors in rank-$1$ decompositions.
Efficient learning of simplices
We also show a direct connection between the problem of learning a simplex and ICA: a simple randomized reduction to ICA from the problem of learning a simplex.
Thompson Sampling for Contextual Bandits with Linear Payoffs
Thompson Sampling is one of the oldest heuristics for multi-armed bandit problems.
