Search Results for author:
Found 16 papers, 10 papers with code
WatChat: Explaining perplexing programs by debugging mental models
Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code.
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
In this work, we propose Hydra heads, a sequentially dependent, drop-in replacement for standard draft heads that significantly improves speculation accuracy.
How to guess a gradient
We study how to narrow the gap in optimization performance between methods that calculate exact gradients and those that use directional derivatives.
Striped Attention: Faster Ring Attention for Causal Transformers
In experiments running Striped Attention on A100 GPUs and TPUv4s, we are able to achieve up to 1. 45x end-to-end throughput improvements over the original Ring Attention algorithm on causal transformer training at a sequence length of 256k.
The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning
We study two natural scaling techniques -- weight pruning and simply training a smaller or larger model, which we refer to as dense scaling -- and their effects on two core capabilities of LLMs: (a) recalling facts presented during pre-training and (b) processing information presented in-context during inference.
Differentiating Metropolis-Hastings to Optimize Intractable Densities
We develop an algorithm for automatic differentiation of Metropolis-Hastings samplers, allowing us to differentiate through probabilistic inference, even if the model has discrete components within it.
Designing Perceptual Puzzles by Differentiating Probabilistic Programs
We design new visual illusions by finding "adversarial examples" for principled models of human perception -- specifically, for probabilistic models, which treat vision as Bayesian inference.
Differentiable Vector Graphics Rasterization for Editing and Learning
We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content.
Neural Kernels Without Tangents
We investigate the connections between neural networks and simple building blocks in kernel space.
Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments.
DiffTaichi: Differentiable Programming for Physical Simulation
We present DiffTaichi, a new differentiable programming language tailored for building high-performance differentiable physical simulators.
Gradient Descent: The Ultimate Optimizer
This allows us to easily apply the method to other optimizers and hyperparameters (e. g. momentum coefficients).
Programming Heterogeneous Systems from an Image Processing DSL
We address this problem by extending the image processing language, Halide, so users can specify which portions of their applications should become hardware accelerators, and then we provide a compiler that uses this code to automatically create the accelerator along with the "glue" code needed for the user's application to access this hardware.
Software Engineering
A Systematic Approach to Blocking Convolutional Neural Networks
Convolutional Neural Networks (CNNs) are the state of the art solution for many computer vision problems, and many researchers have explored optimized implementations.
Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging
Many graphics and vision problems can be expressed as non-linear least squares optimizations of objective functions over visual data, such as images and meshes.
