Search Results for author:
Found 235 papers, 108 papers with code
Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning
Inductive biases are crucial in disentangled representation learning for narrowing down an underspecified solution set.
Disentangling Length from Quality in Direct Preference Optimization
A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO).
Yell At Your Robot: Improving On-the-Fly from Language Corrections
In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections.
Efficient Data Collection for Robotic Manipulation via Compositional Generalization
Recent works on large-scale robotic data collection typically vary a wide range of environmental factors during data collection, such as object types and table textures.
Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities.
A Critical Evaluation of AI Feedback for Aligning Large Language Models
RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model.
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations.
RLVF: Learning from Verbal Feedback without Overgeneralization
The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences.
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e. g., candidate robot actions, localizations, or trajectories).
Universal Neural Functionals
A challenging problem in many modern machine learning tasks is to process weight-space features, i. e., to transform or extract information from the weights and gradients of a neural network.
Clarify: Improving Model Robustness With Natural Language Corrections
Additionally, we use Clarify to find and rectify 31 novel hard subpopulations in the ImageNet dataset, improving minority-split accuracy from 21. 1% to 28. 7%.
SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning
We posit that a significant challenge to widespread adoption of robotic RL, as well as further development of robotic RL methods, is the comparative inaccessibility of such methods.
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
AutoFT: Learning an Objective for Robust Fine-Tuning
Given a task, AutoFT searches for a fine-tuning procedure that enhances out-of-distribution (OOD) generalization.
MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks.
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation
We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection.
Fine-tuning Language Models for Factuality
The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines.
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Generalization remains one of the most important desiderata for robust robot learning systems.
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels.
Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment
We provide theoretical analysis of our selection mechanism and demonstrate that ROAM enables a robot to adapt rapidly to changes in dynamics both in simulation and on a real Go1 quadruped, even successfully moving forward with roller skates on its feet.
Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
We aim to enable this paradigm in robotic reinforcement learning, allowing a robot to learn a new task with little human effort by leveraging data and models from the Internet.
Contrastive Preference Learning: Learning from Human Feedback without RL
Thus, learning a reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase.
An Emulator for Fine-Tuning Large Language Models using Small Language Models
To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?"
Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
Can we leverage offline RL to recover better policies from online interaction?
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data.
Robot Parkour Learning
Parkour is a grand challenge for legged locomotion that requires robots to overcome various obstacles rapidly in complex environments.
BridgeData V2: A Dataset for Robot Learning at Scale
By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Waypoint-Based Imitation Learning for Robotic Manipulation
AWE can be combined with any BC algorithm, and we find that AWE can increase the success rate of state-of-the-art algorithms by up to 25% in simulation and by 4-28% on real-world bimanual manipulation tasks, reducing the decision making horizon by up to a factor of 10.
Contrastive Example-Based Control
In this paper, we propose a method for offline, example-based control that learns an implicit model of multi-step transitions, rather than a reward function.
Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations
Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation.
PIGEON: Predicting Image Geolocations
We train two models for evaluations on street-level data and general-purpose image geolocalization; the first model, PIGEON, is trained on data from the game of Geoguessr and is capable of placing over 40% of its guesses within 25 kilometers of the target location globally.
Ranked #1 on
Photo geolocation estimation
on YFCC4k
Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
Towards an answer to this question, we study imitation learning policies in simulation and on a real robot language-conditioned manipulation task to quantify the difficulty of generalization to different (sets of) factors.
Polybot: Training One Policy Across Robots While Embracing Variability
Reusing large datasets is crucial to scale vision-based robotic manipulators to everyday scenarios due to the high cost of collecting robotic datasets.
Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts
Effective machine learning models learn both robust features that directly determine the outcome of interest (e. g., an object with wheels is more likely to be a car), and shortcut features (e. g., an object on a road is more likely to be a car).
Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning
Specifically, we design an office navigation environment, where the agent's goal is to find a particular office, and office locations differ in different buildings (i. e., tasks).
Conservative Prediction via Data-Driven Confidence Minimization
Taking inspiration from this result, we present data-driven confidence minimization (DCM), which minimizes confidence on an uncertainty dataset containing examples that the model is likely to misclassify at test time.
Train Offline, Test Online: A Real Robot Learning Benchmark
Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF).
Disentanglement via Latent Quantization
In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space.
Meta-Learning Online Adaptation of Language Models
We meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of-date base question-answering model's ability to answer questions about a document after a single weighted gradient step.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions.
A Control-Centric Benchmark for Video Prediction
Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics.
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Fine manipulation tasks, such as threading cable ties or slotting a battery, are notoriously difficult for robots because they require precision, careful coordination of contact forces, and closed-loop visual feedback.
Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets
Concretely, we propose a simple approach that uses a small amount of downstream expert data to selectively query relevant behaviors from an offline, unlabeled dataset (including many sub-optimal behaviors).
Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale.
Open-World Object Manipulation using Pre-trained Vision-Language Models
This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information.
Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
In this work, we propose MEDAL++, a novel design for self-improving robotic systems: given a small set of expert demonstrations at the start, the robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
Language-Driven Representation Learning for Robotics
First, we demonstrate that existing representations yield inconsistent results across these tasks: masked autoencoding approaches pick up on low-level spatial features at the cost of high-level semantics, while contrastive learning approaches capture the opposite.
Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features
Transfer learning with a small amount of target data is an effective and common approach to adapting a pre-trained model to distribution shifts.
Improving Domain Generalization with Domain Relations
Distribution shift presents a significant challenge in machine learning, where models often underperform during the test stage when faced with a different distribution than the one they were trained on.
Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts
Some robust training algorithms (e. g., Group DRO) specialize to group shifts and require group information on all training points.
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
In this paper, we identify a property of the structure of an LLM's probability function that is useful for such detection.
A Survey of Meta-Reinforcement Learning
Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible.
NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
Expert demonstrations are a rich source of supervision for training visual robotic manipulation policies, but imitation learning methods often require either a large number of demonstrations or expensive online expert supervision to learn reactive closed-loop behaviors.
RT-1: Robotics Transformer for Real-World Control at Scale
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance.
Learning Options via Compression
Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks.
Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models
A growing ecosystem of large, open-source foundation models has reduced the labeled data and technical expertise necessary to apply machine learning to many new problems.
Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
Temporal shifts -- distribution shifts arising from the passage of time -- often occur gradually and have the additional structure of timestamp metadata.
Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference
While large pre-trained language models are powerful, their predictions often lack logical consistency across test inputs.
Giving Feedback on Interactive Student Programs with Meta-Exploration
However, teaching and giving feedback on such software is time-consuming -- standard approaches require instructors to manually grade student-implemented interactive programs.
Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations
We study this multi-domain long-tailed learning problem and aim to produce a model that generalizes well across all classes and domains.
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task.
When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world.
You Only Live Once: Single-Life Reinforcement Learning
We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within a single episode without interventions, utilizing its prior experience while contending with some form of novelty.
Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens.
Knowledge-Driven New Drug Recommendation
However, newly approved drugs do not have much historical prescription data and cannot leverage existing drug recommendation methods.
C-Mixup: Improving Generalization in Regression
In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks.
Learning to Reason With Relational Abstractions
Large language models have recently shown promising progress in mathematical reasoning when fine-tuned with human-generated sequences walking through a sequence of solution steps.
Offline Reinforcement Learning at Multiple Frequencies
In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies.
Memory-Based Model Editing at Scale
We find that only SERAC achieves high performance on all three problems, consistently outperforming existing approaches to model editing by a significant margin.
Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning
In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by ~20%.
A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning
Prior works have considered an alternating approach where a forward policy learns to solve the task and the backward policy learns to reset the environment, but what initial state distribution should the backward policy reset the agent to?
Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models
We present a data-efficient framework for solving sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.
R3M: A Universal Visual Representation for Robot Manipulation
We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks.
Do Deep Networks Transfer Invariances Across Classes?
Based on this analysis, we show how a generative approach for learning the nuisance transformations can help transfer invariances across classes and improve performance on a set of imbalanced image classification benchmarks.
Ranked #21 on
Long-tail Learning
on CIFAR-10-LT (ρ=100)
Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
Offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions.
Vision-Based Manipulators Need to Also See from Their Hands
We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations.
Policy Architectures for Compositional Generalization in Control
Many tasks in control, robotics, and planning can be specified using desired goal configurations for various entities in the environment.
Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
As ERM models can be good spurious attribute predictors, CNC works by (1) using a trained ERM model's outputs to identify samples with the same class but dissimilar spurious features, and (2) training a robust model with contrastive learning to learn similar representations for same-class samples.
Robust Policy Learning over Multiple Uncertainty Sets
Reinforcement learning (RL) agents need to be robust to variations in safety-critical environments.
Diversify and Disambiguate: Learning From Underspecified Data
Many datasets are underspecified: there exist multiple equally viable solutions to a given task.
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning.
How to Leverage Unlabeled Data in Offline Reinforcement Learning
One natural solution is to learn a reward function from the labeled data and use it to label the unlabeled data.
Fully Online Meta-Learning Without Task Boundaries
In this paper, we study how meta-learning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future.
Improving Out-of-Distribution Robustness via Selective Augmentation
Machine learning algorithms typically assume that training and test examples are drawn from the same distribution.
Autonomous Reinforcement Learning: Formalism and Benchmarking
In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
CoMPS: Continual Meta Policy Search
Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly.
MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments.
Information is Power: Intrinsic Control via Information Capture
We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model.
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases.
Meta-learning with an Adaptive Task Scheduler
In ATS, for the first time, we design a neural scheduler to decide which meta-training tasks to use next by predicting the probability being sampled for each candidate task, and train the scheduler to optimize the generalization capacity of the meta-model to unseen tasks.
Fast Model Editing at Scale
To enable easy post-hoc editing at scale, we propose Model Editor Networks using Gradient Decomposition (MEND), a collection of small auxiliary editing networks that use a single desired input-output pair to make fast, local edits to a pre-trained model's behavior.
MEMO: Test Time Robustness via Adaptation and Augmentation
We study the problem of test time robustification, i. e., using the test input to improve model robustness.
Data Sharing without Rewards in Multi-Task Offline Reinforcement Learning
However, these benefits come at a cost -- for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks.
FitVid: High-Capacity Pixel-Level Video Prediction
Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for a variety of visual perception tasks.
Test Time Robustification of Deep Models via Adaptation and Augmentation
We study the problem of test time robustification, i. e., using the test input to improve model robustness.
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets
Robot learning holds the promise of learning policies that generalize broadly.
A Workflow for Offline Model-Free Robotic Reinforcement Learning
To this end, we devise a set of metrics and conditions that can be tracked over the course of offline training, and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve final performance.
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks.
Model-based Reinforcement Learning
reinforcement-learning
+1
Lifelong Robotic Reinforcement Learning by Retaining Experiences
Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills.
Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation.
Efficiently Identifying Task Groupings for Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Bayesian Embeddings for Few-Shot Open World Recognition
As autonomous decision-making agents move from narrow operating environments to unstructured worlds, learning systems must move from a closed-world formulation to an open-world and few-shot setting in which agents continuously learn new classes from small amounts of information.
Autonomous Reinforcement Learning via Subgoal Curricula
Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents.
ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
High-quality computer science education is limited by the difficulty of providing instructor feedback to students at scale.
Differentiable Annealed Importance Sampling and the Perils of Gradient Noise
To this end, we propose Differentiable AIS (DAIS), a variant of AIS which ensures differentiability by abandoning the Metropolis-Hastings corrections.
Just Train Twice: Improving Group Robustness without Training Group Information
Standard training via empirical risk minimization (ERM) can produce models that achieve high accuracy on average but low accuracy on certain groups, especially in the presence of spurious correlations between the input and label.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Visual Adversarial Imitation Learning using Variational Models
We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions.
FitVid: Overfitting in Pixel-Level Video Prediction
There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions.
Ranked #6 on
Video Generation
on BAIR Robot Pushing
Intrinsic Control of Variational Beliefs in Dynamic Partially-Observed Visual Environments
We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model.
Meta-Learning with Fewer Tasks through Task Interpolation
Meta-learning enables algorithms to quickly learn a newly encountered task with just a few labeled examples by transferring previously learned knowledge.
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks.
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data.
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.
Discriminator Augmented Model-Based Reinforcement Learning
By planning through a learned dynamics model, model-based reinforcement learning (MBRL) offers the prospect of good performance with little environment interaction.
Model-based Reinforcement Learning
reinforcement-learning
+1
FEW-SHOTLEARNING WITH WEAK SUPERVISION
Few-shot meta-learning methods aim to learn the common structure shared across a set of tasks to facilitate learning new tasks with small amounts of data.
Variational Model-Based Imitation Learning in High-Dimensional Observation Spaces
We consider the problem setting of imitation learning where the agent is provided a fixed dataset of demonstrations.
Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction.
Ranked #1 on
Video Prediction
on Cityscapes 128x128
Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms
Our results show that the proposed method can successfully adapt a trained policy to different robotic platforms with novel physical parameters and the superiority of our meta-learning algorithm compared to state-of-the-art methods for the introduced few-shot policy adaptation problem.
COMBO: Conservative Offline Model-Based Policy Optimization
We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model.
How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains.
On Trade-offs of Image Prediction in Visual Model-Based Reinforcement Learning
In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning.
Model-based Reinforcement Learning
reinforcement-learning
+1
Variable-Shot Adaptation for Incremental Meta-Learning
Few-shot meta-learning methods consider the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
Information Transfer in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Model-Based Visual Planning with Self-Supervised Functional Distances
In our experiments, we find that our method can successfully learn models that perform a variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning to open and close a drawer using a real-world robot.
Offline Reinforcement Learning from Images with Latent Space Models
In this work, we build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces.
Variable-Shot Adaptation for Online Meta-Learning
Few-shot meta-learning methods consider the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild.
Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Continual Learning of Control Primitives : Skill Discovery via Reset-Games
First, in real world settings, when an agent attempts a tasks and fails, the environment must somehow "reset" so that the agent can attempt the task again.
Learning Latent Representations to Influence Multi-Agent Interaction
We propose a reinforcement learning-based framework for learning latent representations of an agent's policy, where the ego agent identifies the relationship between its behavior and the other agent's future strategy.
Reinforcement Learning with Videos: Combining Offline Observations with Interaction
In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans?
Continual Learning of Control Primitives: Skill Discovery via Reset-Games
Reinforcement learning has the potential to automate the acquisition of behavior in complex settings, but in order for it to be successfully deployed, a number of practical challenges must be addressed.
Measuring and Harnessing Transference in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration.
One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
While reinforcement learning algorithms can learn effective policies for complex tasks, these policies are often brittle to even minor task variations, especially when variations are not explicitly provided during training.
Learning to be Safe: Deep RL with a Safety Critic
Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself.
MELD: Meta-Reinforcement Learning from Images via Latent State Models
Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks.
Batch Exploration with Examples for Scalable Robotic Reinforcement Learning
Concretely, we propose an exploration technique, Batch Exploration with Examples (BEE), that explores relevant regions of the state-space, guided by a modest number of human provided images of important states.
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment.
Offline Meta-Reinforcement Learning with Advantage Weighting
That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task.
Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
Learning a new task often requires both exploring to gather task-relevant information and exploiting this information to solve the task.
Goal-Aware Prediction: Learning to Model What Matters
In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task.
Adaptive Risk Minimization: Learning to Adapt to Domain Shift
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
Meta-Learning Symmetries by Reparameterization
We present a method for learning and encoding equivariances into networks by learning corresponding parameter sharing patterns from data.
Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations.
Deep Reinforcement Learning amidst Lifelong Non-Stationarity
As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives.
Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning
In principle, meta-reinforcement learning approaches can exploit this shared structure, but in practice, they fail to adapt to new environments when adaptation requires targeted exploration (e. g., exploring the cabinets to find ingredients in a new kitchen).
Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments, but most robot learning systems today are deployed as fixed policies which do not adapt after deployment.
Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions.
MOPO: Model-based Offline Policy Optimization
We also characterize the trade-off between the gain and risk of leaving the support of the batch data.
Model Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
Watch, Try, Learn: Meta-Learning from Demonstrations and Rewards
Imitation learning allows agents to learn complex behaviors from demonstrations.
Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments.
Weakly-Supervised Reinforcement Learning for Controllable Behavior
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks.
OmniTact: A Multi-Directional High Resolution Touch Sensor
We compare with a state-of-the-art tactile sensor that is only sensitive on one side, as well as a state-of-the-art multi-directional tactile sensor, and find that OmniTact's combination of high-resolution and multi-directional sensing is crucial for reliably inserting the electrical connector and allows for higher accuracy in the state estimation task.
Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
Learning adaptable policies is crucial for robots to operate autonomously in our complex and quickly changing world.
Scalable Multi-Task Imitation Learning with Autonomous Improvement
In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation.
Gradient Surgery for Multi-Task Learning
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge.
Learning Predictive Models From Observation and Interaction
Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works, and then use this learned model to plan coordinated sequences of actions to bring about desired outcomes.
Continuous Meta-Learning without Tasks
In this work, we enable the application of generic meta-learning algorithms to settings where this task segmentation is unavailable, such as continual online learning with a time-varying task.
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche.
Unsupervised Curricula for Visual Meta-Reinforcement Learning
In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.
Meta-Learning without Memorization
If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes.
Entity Abstraction in Visual Model-Based Reinforcement Learning
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before.
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors.
Ranked #1 on
Meta-Learning
on ML10
RoboNet: Large-Scale Multi-Robot Learning
This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to collect impractically large amounts of data for each separate experiment?
Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents
We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure?
Goal-Conditioned Video Prediction
Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.
SMiRL: Surprise Minimizing RL in Entropic Environments
All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis.
Unsupervised Pre-training
Unsupervised Reinforcement Learning
Mint: Matrix-Interleaving for Multi-Task Learning
Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data.
Consistent Meta-Reinforcement Learning via Model Identification and Experience Relabeling
Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large.
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
Critically, our model can infer rewards for new, structurally-similar tasks from a single demonstration.
Ranked #1 on
MuJoCo Games
on Sawyer Pusher
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects.
Meta-Learning with Implicit Gradients
By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer.
Training an Interactive Helper
In this paper, we investigate if, and how, a "helper" agent can be trained to interactively adapt their behavior to maximize the reward of another agent, whom we call the "prime" agent, without observing their reward or receiving explicit demonstrations.
Learning to Interactively Learn and Assist
Most importantly, we find that our approach produces an agent that is capable of learning interactively from a human user, without a set of explicit demonstrations or a reward function, and achieving significantly better performance cooperatively with a human than a human performing the task alone.
Language as an Abstraction for Hierarchical Deep Reinforcement Learning
We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations.
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
Imitation learning allows agents to learn complex behaviors from demonstrations.
Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning
A significant challenge for the practical application of reinforcement learning toreal world problems is the need to specify an oracle reward function that correctly defines a task.
Learning to Reinforcement Learn by Imitation
Meta-reinforcement learning aims to learn fast reinforcement learning (RL) procedures that can be applied to new tasks or environments.
Reasoning About Physical Interactions with Object-Centric Models
Object-based factorizations provide a useful level of abstraction for interacting with the world.
End-to-End Robotic Reinforcement Learning without Reward Engineering
In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task.
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects.
Guided Meta-Policy Search
Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch.
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging.
NoRML: No-Reward Meta Learning
To this end, we introduce a method that allows for self-adaptation of learned policies: No-Reward Meta Learning (NoRML).
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions.
Ranked #15 on
Video Generation
on BAIR Robot Pushing
Model-Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
Ranked #12 on
Atari Games 100k
on Atari 100k
Online Meta-Learning
Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch.
Unsupervised Visuomotor Control through Distributional Planning Networks
While reinforcement learning (RL) has the potential to enable robots to autonomously acquire a wide range of skills, in practice, RL usually requires manual, per-task engineering of reward functions, especially in real world settings where aspects of the environment needed to compute progress are not directly accessible.
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Object-based factorizations provide a useful level of abstraction for interacting with the world.
