Search Results for author:
Found 32 papers, 9 papers with code
Neural Architecture Search: Insights from 1000 Papers
Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas.
Natural Language Understanding
Neural Architecture Search
+2
What Makes Convolutional Models Great on Long Sequence Modeling?
We focus on the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by S4 that are sufficient to make up an effective global convolutional model: 1) The parameterization of the convolutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length.
Ranked #6 on
Long-range modeling
on LRA
Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints
In this work, we study the more challenging open-domain setting consisting of low frequency user prompt patterns (or broad prompts, e. g., prompt about 93rd academy awards) and demonstrate the effectiveness of character-based language models.
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting.
LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
Results show that the perplexity of 16-layer GPT-2 and Transformer-XL can be achieved with up to 1. 5x, 2. 5x faster runtime and 1. 2x, 2. 0x lower peak memory utilization.
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (K=3 for typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e. g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training.
Ranking Convolutional Architectures by their Feature Extraction Capabilities
We propose a simple but powerful method which we call FEAR, for ranking architectures in any search space.
FEAR: A Simple Lightweight Method to Rank Architectures
We propose a simple but powerful method which we call FEAR, for ranking architectures in any search space.
Ranking Architectures by Feature Extraction Capabilities
By training different architectures in the search space to the same training or validation error and subsequently comparing the usefulness of the features extracted on the task-dataset of interest by freezing most of the architecture we obtain quick estimates of the relative performance.
MultiPoint: Cross-spectral registration of thermal and optical aerial imagery
This model is then deployed for fast and accurate online interest point detection.
Reparameterized Variational Divergence Minimization for Stable Imitation
While recent state-of-the-art results for adversarial imitation-learning algorithms are encouraging, recent works exploring the imitation learning from observation (ILO) setting, where trajectories \textit{only} contain expert observations, have not been met with the same success.
A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks
Learning to align these different instruction sets is challenging because: a) different recipes vary in their order of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text instructions.
Efficient Forward Architecture Search
We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers.
Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations
We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting modules that comprise the system.
Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments.
Deep Reinforcement Learning of Universal Policies with Diverse Environment Summaries
Deep reinforcement learning has enabled robots to complete complex tasks in simulation.
Learn-to-Score: Efficient 3D Scene Exploration by Predicting View Utility
We propose to learn a better utility function that predicts the usefulness of future viewpoints.
Discovering Blind Spots in Reinforcement Learning
Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments.
Anytime Neural Network: a Versatile Trade-off Between Computation and Accuracy
We present an approach for anytime predictions in deep neural networks (DNNs).
Log-DenseNet: How to Sparsify a DenseNet
Skip connections are increasingly utilized by deep neural networks to improve accuracy and cost-efficiency.
Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
Experimentally, the adaptive weights induce more competitive anytime predictions on multiple recognition data-sets and models than non-adaptive approaches including weighing all losses equally.
Safety-Aware Algorithms for Adversarial Contextual Bandit
To address this problem, we first study online convex programming in the full information setting where in each round the learner receives an adversarial convex loss and a convex constraint.
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
Developing and testing algorithms for autonomous vehicles in real world is an expensive and time consuming process.
Submodular Trajectory Optimization for Aerial 3D Scanning
Drones equipped with cameras are emerging as a powerful tool for large-scale aerial 3D scanning, but existing automatic flight planners do not exploit all available information about the scene, and can therefore produce inaccurate and incomplete 3D models.
Flight Dynamics-based Recovery of a UAV Trajectory using Ground Cameras
Our main contribution is a new bundle adjustment procedure which in addition to optimizing the camera poses, regularizes the point trajectory using a prior based on motion dynamics (or specifically flight dynamics).
Learning to Gather Information via Imitation
The budgeted information gathering problem - where a robot with a fixed fuel budget is required to maximize the amount of information gathered from the world - appears in practice across a wide range of applications in autonomous exploration and inspection with mobile robots.
Risk-Aware Algorithms for Adversarial Contextual Bandits
To address this problem, we first study the full information setting where in each round the learner receives an adversarial convex loss and a convex constraint.
No-Regret Replanning under Uncertainty
This paper explores the problem of path planning under uncertainty.
Predicting Multiple Structured Visual Interpretations
We present a simple approach for producing a small number of structured visual outputs which have high recall, for a variety of tasks including monocular pose estimation and semantic scene segmentation.
Vision and Learning for Deliberative Monocular Cluttered Flight
Cameras provide a rich source of information while being passive, cheap and lightweight for small and medium Unmanned Aerial Vehicles (UAVs).
Knapsack Constrained Contextual Submodular List Prediction with Application to Multi-document Summarization
We study the problem of predicting a set or list of options under knapsack constraint.
Learning Policies for Contextual Submodular Prediction
Many prediction domains, such as ad placement, recommendation, trajectory prediction, and document summarization, require predicting a set or list of options.
