Search Results for author:
Found 12 papers, 4 papers with code
Discovering modular solutions that generalize compositionally
This allows us to relate the problem of compositional generalization to that of identification of the underlying modules.
Gated recurrent neural networks discover attention
In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers.
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
Solving Static Permutation Mastermind using $O(n \log n)$ Queries
Permutation Mastermind is a version of the classical mastermind game in which the number of positions $n$ is equal to the number of colors $k$, and repetition of colors is not allowed, neither in the codeword nor in the queries.
Combinatorics Probability
Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
Finally, we evaluate our approach empirically on MNIST and reinforcement learning tasks and show that it considerably improves the gradient estimation of ES at no extra computational cost.
Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses
Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs).
Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning
In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs.
The linear hidden subset problem for the (1+1) EA with scheduled and adaptive mutation rates
For the first setup, we give a schedule that achieves a runtime of $(1\pm o(1))\beta n \ln n$, where $\beta \approx 3. 552$, which is an asymptotic improvement over the runtime of the static setup.
When Does Hillclimbing Fail on Monotone Functions: An entropy compression argument
Hillclimbing is an essential part of any optimization algorithm.
Approximating Real-Time Recurrent Learning with Random Kronecker Factors
Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling.
Fast-Slow Recurrent Neural Networks
Processing sequential data of variable length is a major challenge in a wide range of applications, such as speech recognition, language modeling, generative image modeling and machine translation.
Ranked #10 on
Language Modelling
on Penn Treebank (Character Level)
Drift Analysis and Evolutionary Algorithms Revisited
One of the easiest randomized greedy optimization algorithms is the following evolutionary algorithm which aims at maximizing a boolean function $f:\{0, 1\}^n \to {\mathbb R}$.
