Search Results for author:
Found 16 papers, 10 papers with code
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs).
Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model
Stepwise inference protocols, such as scratchpads and chain-of-thought, help language models solve complex problems by decomposing them into a sequence of simpler subproblems.
FoMo Rewards: Can we cast foundation models as reward functions?
We explore the viability of casting foundation models as generic reward functions for reinforcement learning.
Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy.
Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks
Transformers trained on huge text corpora exhibit a remarkable set of capabilities, e. g., performing basic arithmetic.
In-Context Learning Dynamics with Random Binary Sequences
Large language models (LLMs) trained on huge corpora of text datasets demonstrate intriguing capabilities, achieving state-of-the-art performance on tasks they were not explicitly trained for.
Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
Motivated by this, we perform a controlled study for understanding compositional generalization in conditional diffusion models in a synthetic setting, varying different attributes of the training data and measuring the model's ability to generate samples out-of-distribution.
Mechanistic Mode Connectivity
We study neural network loss landscapes through the lens of mode connectivity, the observation that minimizers of neural networks retrieved via training on a dataset are connected via simple paths of low loss.
What shapes the loss landscape of self-supervised learning?
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL).
Analyzing Data-Centric Properties for Graph Contrastive Learning
Overall, our work rigorously contextualizes, both empirically and theoretically, the effects of data-centric properties on augmentation strategies and learning paradigms for graph SSL.
Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering
Federated learning is generally used in tasks where labels are readily available (e. g., next word prediction).
Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices
Unsupervised graph representation learning is critical to a wide range of applications where labels may be scarce or expensive to procure.
Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning.
How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation
Catastrophic forgetting undermines the effectiveness of deep neural networks (DNNs) in scenarios such as continual learning and lifelong learning.
A Gradient Flow Framework For Analyzing Network Pruning
We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and is therefore appropriate for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models.
OrthoReg: Robust Network Pruning Using Orthonormality Regularization
To address this issue, we propose OrthoReg, a principled regularization strategy that enforces orthonormality on a network's filters to reduce inter-filter correlation, thereby allowing reliable, efficient determination of group importance estimates, improved trainability of pruned networks, and efficient, simultaneous pruning of large groups of filters.
