Search Results for author:
Found 54 papers, 19 papers with code
Weight subcloning: direct initialization of transformers using larger pretrained ones
The usual practice of transfer learning overcomes this challenge by initializing the model with weights of a pretrained model of the same size and specification to increase the convergence and training speed.
Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models
Motivated by this, we ask the following important question, "How can we leverage the knowledge from a large VFM to train a small task-specific model for a new target task with limited labeled training data?
Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications
While these models can achieve high-quality lip articulation for speakers in the training set, they are unable to capture the full and diverse distribution of 3D facial motions that accompany speech in the real world.
HUGS: Human Gaussian Splats
We achieve state-of-the-art rendering quality with a rendering speed of 60 FPS while being ~100x faster to train over previous work.
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2. 9% average performance improvement on 38 evaluation benchmarks compared to the previous best.
TiC-CLIP: Continual Training of CLIP Models
We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models: TiC-DataComp, TiC-YFCC, and TiC-Redcaps.
Novel-View Acoustic Synthesis from 3D Reconstructed Rooms
Given audio recordings from 2-4 microphones and the 3D geometry and material of a scene containing multiple unknown sound sources, we estimate the sound anywhere in the scene.
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
By applying our method to SAM and CLIP, we obtain SAM-CLIP: a unified model that combines the capabilities of SAM and CLIP into a single vision transformer.
CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
While CLIP is scalable, promptable, and robust to distribution shifts on image classification tasks, it lacks object localization capabilities.
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications.
Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models
In this paper, we propose a new strategy for adapting ASR models to new target domains without any text or speech from those domains.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
VISION Datasets: A Benchmark for Vision-based InduStrial InspectiON
We introduce the VISION Datasets, a diverse collection of 14 industrial inspection datasets, uniquely poised to meet these challenges.
Pointersect: Neural Rendering with Cloud-Ray Intersection
Specifically, we train a set transformer that, given a small number of local neighbor points along a light ray, provides the intersection point, the surface normal, and the material blending weights, which are used to render the outcome of this light ray.
Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis
Here, we ask two fundamental questions about this strategy: when is synthetic data effective for personalization, and why is it effective in those cases?
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
FaceLit: Neural 3D Relightable Faces
We propose a generative framework, FaceLit, capable of generating a 3D face that can be rendered at various user-defined lighting conditions and views, learned purely from 2D images in-the-wild without any manual annotation.
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization
To this end, we introduce a novel token mixing operator, RepMixer, a building block of FastViT, that uses structural reparameterization to lower the memory access cost by removing skip-connections in the network.
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
Models pretrained on ImageNet+ and fine-tuned on CIFAR-100+, Flowers-102+, and Food-101+, reach up to 3. 4% improved accuracy.
FastFill: Efficient Compatible Model Update
The task of retrieving the most similar data from a gallery set to a given query data is performed through a similarity comparison on features.
RangeAugment: Efficient Online Augmentation with Range Learning
To answer the open question on the importance of magnitude ranges for each augmentation operation, we introduce RangeAugment that allows us to efficiently learn the range of magnitudes for individual as well as composite augmentation operations.
I see what you hear: a vision-inspired method to localize words
This paper explores the possibility of using visual object detection techniques for word localization in speech data.
APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations
Recent advances in learning aligned multimodal representations have been primarily driven by training large neural networks on massive, noisy paired-modality datasets.
MobileOne: An Improved One millisecond Mobile Backbone
Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device.
Ranked #586 on
Image Classification
on ImageNet
NeuMan: Neural Human Radiance Field from a Single Video
Photorealistic rendering and reposing of humans is important for enabling augmented reality experiences.
Forward Compatible Training for Large-Scale Embedding Retrieval Systems
To avoid the cost of backfilling, BCT modifies training of the new model to make its representations compatible with those of the old model.
Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition
With recent advances in speech synthesis, synthetic data is becoming a viable alternative to real data for training speech recognition models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Data Incubation -- Synthesizing Missing Data for Handwriting Recognition
We use the framework to optimize data synthesis and demonstrate significant improvement on handwriting recognition over a model trained on real data only.
Token Pooling in Vision Transformers
Token Pooling is a simple and effective operator that can benefit many architectures.
Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
However, under an unsupervised-style setting, typical training algorithms for controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but unpaired samples are given during inference.
Instance-Level Task Parameters: A Robust Multi-task Weighting Framework
When applied to datasets where one or more tasks can have noisy annotations, the proposed method learns to prioritize learning from clean labels for a given task, e. g. reducing surface estimation errors by up to 60%.
Optimize what matters: Training DNN-HMM Keyword Spotting Model Using End Metric
The DNN, in prior methods, is trained independent of the HMM parameters to minimize the cross-entropy loss between the predicted and the ground-truth state probabilities.
Ranked #2 on
Keyword Spotting
on hey Siri
SapAugment: Learning A Sample Adaptive Policy for Data Augmentation
Our policy adapts the augmentation parameters based on the training loss of the data samples.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Subject-Aware Contrastive Learning for Biosignals
Datasets for biosignals, such as electroencephalogram (EEG) and electrocardiogram (ECG), often have noisy labels and have limited number of subjects (<100).
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution
We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples.
Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis
We present a method to generate speech from input text and a style vector that is extracted from a reference speech signal in an unsupervised manner, i. e., no style annotation, such as speaker information, is required.
Least squares binary quantization of neural networks
We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed least squares quantization algorithms.
Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum
To the best of our knowledge, our work is the first curriculum learning method to show gains on large scale image classification and detection tasks.
OPTIMAL BINARY QUANTIZATION FOR DEEP NEURAL NETWORKS
We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed optimal quantization algorithms.
MVX-Net: Multimodal VoxelNet for 3D Object Detection
Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data.
Ranked #7 on
3D Object Detection
on DAIR-V2X-I
Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
In this work, we propose and evaluate the stochastic preconditioned nonlinear conjugate gradient algorithm for large scale DNN training tasks.
Divide, Denoise, and Defend against Adversarial Attacks
Improving the robustness of neural networks against these attacks is important, especially for security-critical applications.
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Accurate detection of objects in 3D point clouds is a central problem in many applications, such as autonomous navigation, housekeeping robots, and augmented/virtual reality.
Attentional Network for Visual Object Detection
We propose augmenting deep neural networks with an attention mechanism for the visual object detection task.
Learning from Simulated and Unsupervised Images through Adversarial Training
With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations.
Ranked #3 on
Image-to-Image Translation
on Cityscapes Labels-to-Photo
(Per-class Accuracy metric)
Coupled Generative Adversarial Networks
We propose coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images.
Gaussian Conditional Random Field Network for Semantic Segmentation
In contrast to the existing approaches that use discrete Conditional Random Field (CRF) models, we propose to use a Gaussian CRF model for the task of semantic segmentation.
A Multi-Stream Bi-Directional Recurrent Neural Network for Fine-Grained Action Detection
We present a multi-stream bi-directional recurrent neural network for fine-grained action detection.
Action Recognition In Videos
Fine-Grained Action Detection
+2
Global-Local Face Upsampling Network
In our deep network architecture the global and local constraints that define a face can be efficiently modeled and learned end-to-end using training data.
Robust Face Alignment Using a Mixture of Invariant Experts
Face alignment is particularly challenging when there are large variations in pose (in-plane and out-of-plane rotations) and facial expression.
Deep Gaussian Conditional Random Field Network: A Model-based Deep Network for Discriminative Denoising
We propose a novel deep network architecture for image\\ denoising based on a Gaussian Conditional Random Field (GCRF) model.
Layered Interpretation of Street View Images
We propose a layered street view model to encode both depth and semantic information on street view images for autonomous driving.
Deep Hierarchical Parsing for Semantic Segmentation
We propose to tackle this problem by including the classification loss of the internal nodes of the random parse trees in the original RCPN loss function.
Efficient Upsampling of Natural Images
1) For the edge layer, we use a nonparametric approach by constructing a dictionary of patches from a given image, and synthesize edge regions in a higher-resolution version of the image.
Recursive Context Propagation Network for Semantic Scene Labeling
Then a top-down propagation of the aggregated information takes place that enhances the contextual information of each local feature.
Joint Geodesic Upsampling of Depth Images
We propose an algorithm utilizing geodesic distances to upsample a low resolution depth image using a registered high resolution color image.
