no code implementations • 6 Mar 2024 • Xiao Ma, Sumit Patidar, Iain Haughton, Stephen James
This paper introduces Hierarchical Diffusion Policy (HDP), a hierarchical agent for multi-task robotic manipulation.
no code implementations • 25 Sep 2023 • Jiangliu Wang, Jianbo Jiao, Yibing Song, Stephen James, Zhan Tong, Chongjian Ge, Pieter Abbeel, Yun-hui Liu
This work aims to improve unsupervised audio-visual pre-training.
no code implementations • 31 Aug 2023 • Amber Xie, Youngwoon Lee, Pieter Abbeel, Stephen James
Contact is at the core of robotic manipulation.
1 code implementation • 23 Aug 2023 • Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, Pieter Abbeel
Using learned reward functions (LRFs) as a means to solve sparse-reward reinforcement learning (RL) tasks has yielded some steady progress in task-complexity through the years.
1 code implementation • 5 Feb 2023 • Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, Pieter Abbeel
In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation.
no code implementations • 3 Nov 2022 • Kai Chen, Stephen James, Congying Sui, Yun-hui Liu, Pieter Abbeel, Qi Dou
To further improve the performance of the stereo framework, StereoPose is equipped with a parallax attention module for stereo feature fusion and an epipolar loss for improving the stereo-view consistency of network predictions.
1 code implementation • 25 Oct 2022 • John So, Amber Xie, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali Agha-mohammadi, Pieter Abbeel, Stephen James
In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data.
1 code implementation • 6 Oct 2022 • Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, Trevor Darrell
Finally, we train a 307M parameter vision transformer on a massive collection of 4. 5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.
1 code implementation • 5 Oct 2022 • Wilson Yan, Danijar Hafner, Stephen James, Pieter Abbeel
To generate accurate videos, algorithms have to understand the spatial and temporal dependencies in the world.
no code implementations • 15 Sep 2022 • Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, Pieter Abbeel
Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics.
no code implementations • 28 Jun 2022 • Younggyo Seo, Danijar Hafner, Hao liu, Fangchen Liu, Stephen James, Kimin Lee, Pieter Abbeel
Yet the current approaches typically train a single model end-to-end for learning both visual representations and dynamics, making it difficult to accurately model the interaction between robots and small objects.
Model-based Reinforcement Learning Reinforcement Learning (RL) +1
1 code implementation • 8 Jun 2022 • Wilson Yan, Ryo Okumura, Stephen James, Pieter Abbeel
In this work, we present Patch-based Object-centric Video Transformer (POVT), a novel region-based video generation architecture that leverages object-centric information to efficiently model temporal dynamics in videos.
no code implementations • 7 Jun 2022 • Zhao Mandi, Pieter Abbeel, Stephen James
From these findings, we advocate for evaluating future meta-RL methods on more challenging tasks and including multi-task pretraining with fine-tuning as a simple, yet strong baseline.
1 code implementation • 26 Apr 2022 • Stephen James, Pieter Abbeel
Coarse-to-fine Q-attention enables sample-efficient robot manipulation by discretizing the translation space in a coarse-to-fine manner, where the resolution gradually increases at each layer in the hierarchy.
no code implementations • 14 Apr 2022 • Kai Chen, Rui Cao, Stephen James, Yichuan Li, Yun-hui Liu, Pieter Abbeel, Qi Dou
To continuously improve the quality of pseudo labels, we iterate the above steps by taking the trained student model as a new teacher and re-label real data using the refined teacher model.
1 code implementation • 4 Apr 2022 • Stephen James, Pieter Abbeel
We propose Learned Path Ranking (LPR), a method that accepts an end-effector goal pose, and learns to rank a set of goal-reaching paths generated from an array of path generating methods, including: path planning, Bezier curve sampling, and a learned policy.
2 code implementations • 25 Mar 2022 • Younggyo Seo, Kimin Lee, Stephen James, Pieter Abbeel
Our framework consists of two phases: we pre-train an action-free latent video prediction model, and then utilize the pre-trained representations for efficiently learning action-conditional world models on unseen environments.
1 code implementation • 22 Feb 2022 • Kentaro Wada, Stephen James, Andrew J. Davison
Robots need the capability of placing objects in arbitrary, specific poses to rearrange the world and achieve various valuable tasks.
1 code implementation • 11 Feb 2022 • Kentaro Wada, Stephen James, Andrew J. Davison
We evaluate our methods using the YCB objects in both simulation and the real world, achieving safe object extraction from piles.
1 code implementation • 8 Feb 2022 • Stephen James, Pieter Abbeel
We propose a new policy parameterization for representing 3D rotations during reinforcement learning.
1 code implementation • 7 Feb 2022 • Shikun Liu, Stephen James, Andrew J. Davison, Edward Johns
Unlike previous methods where task relationships are assumed to be fixed, Auto-Lambda is a gradient-based meta learning framework which explores continuous, dynamic task relationships via task-specific weightings, and can optimise any choice of combination of tasks through the formulation of a meta-loss; where the validation loss automatically influences task weightings throughout training.
Ranked #3 on Robot Manipulation on RLBench (Succ. Rate (10 tasks, 100 demos/task) metric)
no code implementations • 29 Sep 2021 • Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, Pieter Abbeel
Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics.
1 code implementation • CVPR 2022 • Stephen James, Kentaro Wada, Tristan Laidlow, Andrew J. Davison
We present a coarse-to-fine discretisation method that enables the use of discrete reinforcement learning approaches in place of unstable and data-inefficient actor-critic methods in continuous robotics domains.
Ranked #7 on Robot Manipulation on RLBench
1 code implementation • 31 May 2021 • Stephen James, Andrew J. Davison
Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks.
Ranked #1 on Robot Task Planning on RLBench
1 code implementation • 1 May 2021 • Alexandru-Iosif Toma, Hussein Ali Jaafar, Hao-Ya Hsueh, Stephen James, Daniel Lenton, Ronald Clark, Sajad Saeedi
We propose waypoint planning networks (WPN), a hybrid algorithm based on LSTMs with a local kernel - a classic algorithm such as A*, and a global kernel using a learned algorithm.
no code implementations • ICCV 2021 • Zoe Landgraf, Raluca Scona, Tristan Laidlow, Stephen James, Stefan Leutenegger, Andrew J. Davison
At test time, our model can generate 3D shape and instance segmentation from a single depth view, probabilistically sampling proposals for the occluded region from the learned latent space.
2 code implementations • 15 Feb 2021 • Daniel Lenton, Stephen James, Ronald Clark, Andrew J. Davison
Spatial memory, or the ability to remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments.
1 code implementation • 4 Feb 2021 • Daniel Lenton, Fabio Pardo, Fabian Falck, Stephen James, Ronald Clark
We introduce Ivy, a templated Deep Learning (DL) framework which abstracts existing DL frameworks.
no code implementations • 1 Jan 2021 • Stephen James, Andrew Davison
Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks.
no code implementations • ICLR 2021 • Daniel James Lenton, Stephen James, Ronald Clark, Andrew Davison
With our broad demonstrations, we show that ESMN represents a useful and general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures.
1 code implementation • CVPR 2020 • Kentaro Wada, Edgar Sucar, Stephen James, Daniel Lenton, Andrew J. Davison
Robots and other smart devices need efficient object-based scene representations from their on-board vision systems to reason about contact, physics and occlusion.
1 code implementation • 4 Nov 2019 • Alessandro Bonardi, Stephen James, Andrew J. Davison
But is there a way to remove the need for real world human demonstrations during training?
2 code implementations • 26 Sep 2019 • Stephen James, Zicong Ma, David Rovick Arrojo, Andrew J. Davison
We present a challenging new benchmark and learning-environment for robot learning: RLBench.
1 code implementation • 26 Jun 2019 • Stephen James, Marc Freese, Andrew J. Davison
PyRep is a toolkit for robot learning research, built on top of the virtual robotics experimentation platform (V-REP).
no code implementations • CVPR 2019 • Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, Konstantinos Bousmalis
Using domain adaptation methods to cross this "reality gap" requires a large amount of unlabelled real-world data, whilst domain randomization alone can waste modeling power.
3 code implementations • 8 Oct 2018 • Stephen James, Michael Bloesch, Andrew J. Davison
Despite this, most robot learning approaches have focused on learning a single task, from scratch, with a limited notion of generalisation, and no way of leveraging the knowledge to learn other tasks more efficiently.
1 code implementation • 20 Jun 2018 • Jan Matas, Stephen James, Andrew J. Davison
Moreover, due to the large amount of data needed to learn these end-to-end solutions, an emerging trend is to learn control policies in simulation and then transfer them over to the real world.
1 code implementation • 7 Jul 2017 • Stephen James, Andrew J. Davison, Edward Johns
End-to-end control for robot manipulation and grasping is emerging as an attractive alternative to traditional pipelined approaches.
no code implementations • 13 Sep 2016 • Stephen James, Edward Johns
Building upon the recent success of deep Q-networks, we present an approach which uses 3D simulations to train a 7-DOF robotic arm in a control task without any prior knowledge.