Search Results for author:
Found 95 papers, 36 papers with code
Language Imbalance Can Boost Cross-lingual Generalisation
In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages.
On the Effect of (Near) Duplicate Subwords in Language Modelling
In this paper, we study the impact of near duplicate subwords on LM training efficiency.
Hallmarks of Optimization Trajectories in Neural Networks and LLMs: The Lengths, Bends, and Dead Ends
We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich structure of parameters contained within their optimization trajectories.
Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning
In this work, we find empirical evidence that learning rate transfer can be attributed to the fact that under $\mu$P and its depth extension, the largest eigenvalue of the training loss Hessian (i. e. the sharpness) is largely independent of the width and depth of the network for a sustained period of training time.
A Language Model's Guide Through Latent Space
Concept guidance has emerged as a cheap and simple way to control the behavior of language models by probing their hidden representations for concept vectors and using them to perturb activations at inference time.
Towards Meta-Pruning via Optimal Transport
Structural pruning of neural networks conventionally relies on identifying and discarding less important neurons, a practice often resulting in significant accuracy loss that necessitates subsequent fine-tuning efforts.
How Good is a Single Basin?
The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles.
Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures
Abstract reasoning is a cornerstone of human intelligence, and replicating it with artificial intelligence (AI) presents an ongoing challenge.
Disentangling Linear Mode-Connectivity
Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes.
LIME: Localized Image Editing via Attention Regularization in Diffusion Models
Diffusion models (DMs) have gained prominence due to their ability to generate high-quality, varied images, with recent advancements in text-to-image generation.
Recurrent Distance Filtering for Graph Representation Learning
Graph neural networks based on iterative one-hop message passing have been shown to struggle in harnessing the information from distant nodes effectively.
Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalization
In this study, we investigate how neural networks exhibit shape bias during training on synthetic datasets, serving as an indicator of the synthetic data quality.
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
This leads to the notion of a `compute-optimal' model, i. e. a model that allocates a given level of compute during training optimally to maximize performance.
Simplifying Transformer Blocks
A simple design recipe for deep Transformers is to compose identical building blocks.
Transformer Fusion with Optimal Transport
We flesh out an abstraction for layer alignment, that can generalize to arbitrary architectures -- in principle -- and we apply this to the key ingredients of Transformers such as multi-head self-attention, layer-normalization, and residual connections, and we discuss how to handle them via various ablation studies.
Towards guarantees for parameter isolation in continual learning
Deep learning has proved to be a successful paradigm for solving many challenges in machine learning.
The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute
We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours.
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention model with skip connections in the proportional limit of infinite-depth-and-width.
Scaling MLPs: A Tale of Inductive Bias
We show that the performance of MLPs drastically improves with scale (95% on CIFAR10, 82% on CIFAR100, 58% on ImageNet ReaL), highlighting that lack of inductive bias can indeed be compensated.
Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore.
The Hessian perspective into the Nature of Convolutional Neural Networks
While Convolutional Neural Networks (CNNs) have long been investigated and applied, as well as theorized, we aim to provide a slightly different perspective into their nature -- through the perspective of their Hessian maps.
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
Training models to apply linguistic knowledge and visual concepts from 2D images to 3D world understanding is a promising direction that researchers have only recently started to explore.
Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model's performances on old tasks drop dramatically after being optimized for a new task.
Random Teachers are Good Teachers
In this work, we investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation.
Cosmology from Galaxy Redshift Surveys with PointNet
In this work, we aim to improve upon two-point statistics by employing a \textit{PointNet}-like neural network to regress the values of the cosmological parameters directly from point cloud data.
The Curious Case of Benign Memorization
While such a memorization capacity seems worrisome, in this work we show that under training protocols that include \textit{data augmentation}, neural networks learn to memorize entirely random labels in a benign way, i. e. they learn embeddings that lead to highly non-trivial performance under nearest neighbour probing.
Decoding a Neural Retriever's Latent Space for Query Suggestion
Neural retrieval models have superseded classic bag-of-words methods such as BM25 as the retrieval framework of choice.
Mastering Spatial Graph Prediction of Road Networks
Accurately predicting road networks from satellite images requires a global understanding of the network topology.
OpenFilter: A Framework to Democratize Research Access to Social Media AR Filters
Augmented Reality or AR filters on selfies have become very popular on social media platforms for a variety of applications, including marketing, entertainment and aesthetics.
How Tempering Fixes Data Augmentation in Bayesian Neural Networks
While data augmentation has been empirically recognized as one of the main drivers of this effect, a theoretical account of its role, on the other hand, is largely missing.
Phenomenology of Double Descent in Finite-Width Neural Networks
`Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized.
Generalization Through The Lens Of Leave-One-Out Error
Despite the tremendous empirical success of deep learning models to solve various learning tasks, our theoretical understanding of their generalization ability is very limited.
FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control
Generating music with deep neural networks has been an area of active research in recent years.
On the effectiveness of Randomized Signatures as Reservoir for Learning Rough Dynamics
Many finance, physics, and engineering phenomena are modeled by continuous-time dynamical systems driven by highly irregular (stochastic) inputs.
Boosting Search Engines with Interactive Agents
This paper presents first successful steps in designing search agents that learn meta-strategies for iterative query refinement in information-seeking tasks.
How to Query Language Models?
We apply this approach of querying by example to the LAMA probe and obtain substantial improvements of up to 37. 8% for BERT-large on the T-REx data when providing only 10 demonstrations--even outperforming a baseline that queries the model with up to 40 paraphrases of the question.
Analytic Insights into Structure and Rank of Neural Network Hessian Maps
Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks.
Precise characterization of the prior predictive distribution of deep ReLU networks
Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture.
Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
The dataset curation hypothesis of Aitchison (2020): we show empirically that the CPE does not arise in a real curated data set but can be produced in a controlled experiment with varying curation strength.
Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks.
Uniform Convergence, Adversarial Spheres and a Simple Remedy
By considering a specific dataset, it was observed that a neural network completely misclassifies a projection of the training data (adversarial set), rendering any existing generalization bound based on uniform convergence vacuous.
Learning Generative Models of Textured 3D Meshes from Real-World Images
Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections.
Generative Minimization Networks: Training GANs Without Competition
Among known problems experienced by practitioners is the lack of convergence guarantees or convergence to a non-optimum cycle.
SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German
Swiss German is a dialect continuum whose natively acquired dialects significantly differ from the formal variety of the language.
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order optimizers.
Batch normalization provably avoids ranks collapse for randomly initialised deep networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
Convolutional Generation of Textured 3D Meshes
A key contribution of our work is the encoding of the mesh and texture as 2D representations, which are semantically aligned and can be easily modeled by a 2D convolutional GAN.
BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
Measuring the quality of a generated sequence against a set of references is a central problem in many learning frameworks, be it to compute a score, to assign a reward, or to perform discrimination.
Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used.
Controlling Style and Semantics in Weakly-Supervised Image Generation
We propose a weakly-supervised approach for conditional image generation of complex scenes where a user has fine control over objects appearing in the scene.
Mixing of Stochastic Accelerated Gradient Descent
We study the mixing properties for stochastic accelerated gradient descent (SAGD) on least-squares regression.
Adversarial Training Generalizes Data-dependent Spectral Norm Regularization
We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks.
LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games
We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme.
Autoregressive Text Generation Beyond Feedback Loops
Autoregressive state transitions, where predictions are conditioned on past predictions, are the predominant choice for both deterministic and stochastic sequential models.
Cosmological N-body simulations: a challenge for scalable generative models
Our results show that the proposed model produces samples of high visual quality, although the statistical analysis reveals that capturing rare features in the data poses significant problems for the generative models.
Cosmological constraints with deep learning from KiDS-450 weak lensing maps
We present the cosmological results with a CNN from the KiDS-450 tomographic weak lensing dataset, constraining the total matter density $\Omega_m$, the fluctuation amplitude $\sigma_8$, and the intrinsic alignment amplitude $A_{\rm{IA}}$.
Cosmology and Nongalactic Astrophysics
Adversarial Training is a Form of Data-dependent Operator Norm Regularization
We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks.
Escaping Flat Areas via Function-Preserving Structural Network Modifications
We develop our method for fully-connected as well as convolutional layers.
Evaluating GANs via Duality
Generative Adversarial Networks (GANs) have shown great results in accurately modeling complex distributions, but their training is known to be difficult due to instabilities caused by a challenging minimax optimization problem.
The Odds are Odd: A Statistical Test for Detecting Adversarial Examples
We investigate conditions under which test statistics exist that can reliably detect examples, which have been adversarially manipulated in a white-box attack.
A domain agnostic measure for monitoring and evaluating GANs
Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training.
Learning and Evaluating Sparse Interpretable Sentence Embeddings
Previous research on word embeddings has shown that sparse representations, which can be either learned on top of existing dense embeddings or obtained through model constraints during training time, have the benefit of increased interpretability properties: to some degree, each dimension can be understood by a human and associated with a recognizable feature in the data.
End-to-End Neural Entity Linking
Entity Linking (EL) is an essential task for semantic text understanding and information extraction.
Ranked #1 on
Entity Linking
on OKE-2015
Cosmological constraints from noisy convergence maps through deep learning
We find that, for a shape noise level corresponding to 8. 53 galaxies/arcmin$^2$ and the smoothing scale of $\sigma_s = 2. 34$ arcmin, the network is able to generate 45% tighter constraints.
Cosmology and Nongalactic Astrophysics
A Distributed Second-Order Algorithm You Can Trust
Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years.
Deep State Space Models for Unconditional Word Generation
Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models.
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks.
Zero-Shot Dual Machine Translation
Our method can obtain improvements also on the setting where a small amount of parallel data for the zero-shot language pair is available.
Hyperbolic Neural Networks
However, the representational power of hyperbolic geometry is not yet on par with Euclidean geometry, mostly because of the absence of corresponding hyperbolic neural network layers.
Adversarially Robust Training through Structured Gradient Regularization
We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations.
Local Saddle Point Optimization: A Curvature Exploitation Approach
Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems.
Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
Learning graph representations via low-dimensional embeddings that preserve relevant network properties is an important class of problems in machine learning.
Ranked #1 on
Link Prediction
on WordNet
Escaping Saddles with Stochastic Gradients
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions.
Fast cosmic web simulations with generative adversarial networks
Computational models of the underlying physical processes, such as classical N-body simulations, are extremely resource intensive, as they track the action of gravity in an expanding universe using billions of particles as tracers of the cosmic matter distribution.
The best defense is a good offense: Countering black box attacks by predicting slightly wrong labels
Black-Box attacks on machine learning models occur when an attacker, despite having no access to the inner workings of a model, can successfully craft an attack by means of model theft.
Semantic Interpolation in Implicit Models
In implicit models, one often interpolates between sampled points in latent space.
Parametrizing filters of a CNN with a GAN
It is commonly agreed that the use of relevant invariances as a good statistical bias is important in machine-learning.
Flexible Prior Distributions for Deep Generative Models
We consider the problem of training generative models with deep neural networks as generators, i. e. to map latent codes to data points.
Generator Reversal
We consider the problem of training generative models with deep neural networks as generators, i. e. to map latent codes to data points.
Learning Aerial Image Segmentation from Online Maps
We adapt a state-of-the-art CNN architecture for semantic segmentation of buildings and roads in aerial images, and compare its performance when using different training data sets, ranging from manually labeled, pixel-accurate ground truth of the same city to automatic training data derived from OpenStreetMap data from distant locations.
Cosmological model discrimination with Deep Learning
We find that our implementation of DCNN outperforms the skewness and kurtosis statistics, especially for high noise levels.
Accelerated Dual Learning by Homotopic Initialization
Gradient descent and coordinate descent are well understood in terms of their asymptotic behavior, but less so in a transient regime often used for approximations in machine learning.
An Online Learning Approach to Generative Adversarial Networks
We consider the problem of training generative models with a Generative Adversarial Network (GAN).
Stabilizing Training of Generative Adversarial Networks through Regularization
Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters.
Deep Joint Entity Disambiguation with Local Neural Attention
We propose a novel deep learning model for joint document-level entity disambiguation, which leverages learned neural representations.
Ranked #4 on
Entity Disambiguation
on WNED-CWEB
Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification
This paper presents a novel approach for multi-lingual sentiment classification in short texts.
A Semi-supervised Framework for Image Captioning
We here propose a novel way of using such textual data by artificially generating missing visual information.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilingual character-level translation even surpasses the models specifically trained on that language pair alone, both in terms of BLEU score and human judgment.
DynaNewton - Accelerating Newton's Method for Machine Learning
Solutions on this path are tracked such that the minimizer of the previous objective is guaranteed to be within the quadratic convergence region of the next objective to be optimized.
Starting Small -- Learning with Adaptive Sample Sizes
For many machine learning problems, data is abundant and it may be prohibitive to make multiple passes through the full training set.
Probabilistic Bag-Of-Hyperlinks Model for Entity Linking
We demonstrate the accuracy of our approach on a wide range of benchmark datasets, showing that it matches, and in many cases outperforms, existing state-of-the-art methods.
Variance Reduced Stochastic Gradient Descent with Neighbors
As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms.
A Variance Reduced Stochastic Newton Method
Quasi-Newton methods are widely used in practise for convex loss minimization problems.
Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning.
Probabilistic Latent Semantic Analysis
Probabilistic Latent Semantic Analysis is a novel statistical technique for the analysis of two-mode and co-occurrence data, which has applications in information retrieval and filtering, natural language processing, machine learning from text, and in related areas.
