Search Results for author:
Found 68 papers, 15 papers with code
A Note on Stability in Asynchronous Stochastic Approximation without Communication Delays
In this paper, we study asynchronous stochastic approximation algorithms without communication delays.
Iterative Option Discovery for Planning, by Planning
Discovering useful temporal abstractions, in the form of options, is widely thought to be key to applying reinforcement learning and planning to increasingly complex domains.
Value-aware Importance Weighting for Off-policy Reinforcement Learning
Importance sampling is a central idea underlying off-policy prediction in reinforcement learning.
Maintaining Plasticity in Deep Continual Learning
If deep-learning systems are applied in a continual learning setting, then it is well known that they may fail to remember earlier examples.
On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs
We show two average-reward off-policy control algorithms, Differential Q-learning (Wan, Naik, & Sutton 2021a) and RVI Q-learning (Abounadi Bertsekas & Borkar 2001), converge in weakly communicating MDPs.
The Alberta Plan for AI Research
Herein we describe our approach to artificial intelligence research, which we call the Alberta Plan.
Doubly-Asynchronous Value Iteration: Making Value Iteration Asynchronous in Actions
However, Asynchronous VI still requires a maximization over the entire action space, making it impractical for domains with large action space.
Toward Discovering Options that Achieve Faster Planning
In a variant of the classic four-room domain, we show that 1) a higher objective value is typically associated with fewer number of elementary planning operations used by the option-value iteration algorithm to obtain a near-optimal value function, 2) our algorithm achieves an objective value that matches it achieved by two human-designed options 3) the amount of computation used by option-value iteration with options discovered by our algorithm matches it with the human-designed options, 4) the options produced by our algorithm also make intuitive sense--they seem to move to and terminate at the entrances of rooms.
The Quest for a Common Model of the Intelligent Decision Maker
It is time to recognize and build on the convergence of multiple diverse disciplines on a substantive common model of the intelligent agent.
A History of Meta-gradient: Gradient Methods for Meta-learning
The history of meta-learning methods based on gradient descent is reviewed, focusing primarily on methods that adapt step-size (learning rate) meta-parameters.
Reward-Respecting Subtasks for Model-Based Reinforcement Learning
Each subtask is solved to produce an option, and then a model of the option is learned and made available to the planning process.
Model-based Reinforcement Learning
reinforcement-learning
+1
Learning Agent State Online with Recurrent Generate-and-Test
Learning continually and online from a continuous stream of data is challenging, especially for a reinforcement learning agent with sequential data.
Average-Reward Learning and Planning with Options
We extend the options framework for temporal abstraction in reinforcement learning from discounted Markov decision processes (MDPs) to average-reward MDPs.
An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment
In the Rooms task, the product of importance sampling ratios can be as large as $2^{14}$ and can sometimes be two.
Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former.
An Empirical Comparison of Off-policy Prediction Learning Algorithms on the Collision Task
In the middle tier, the five Gradient-TD algorithms and Off-policy TD($\lambda$) were more sensitive to the bootstrapping parameter.
Planning with Expectation Models for Control
Our algorithms and experiments are the first to treat MBRL with expectation models in a general setting.
Does the Adam Optimizer Exacerbate Catastrophic Forgetting?
Catastrophic forgetting remains a severe hindrance to the broad application of artificial neural networks (ANNs), however, it continues to be a poorly understood phenomenon.
Average-Reward Off-Policy Policy Evaluation with Function Approximation
We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function.
Incremental Policy Gradients for Online Reinforcement Learning Control
Our emphasis is on the first approach in this work, detailing an incremental policy gradient update which neither waits until the end of the episode, nor relies on learning estimates of the return.
Understanding the Pathologies of Approximate Policy Evaluation when Combined with Greedification in Reinforcement Learning
We demonstrate analytically and experimentally that such pathological behaviours can impact a wide range of RL and dynamic programming algorithms; such behaviours can arise both with and without bootstrapping, and with linear function approximation as well as with more complex parameterized functions like neural networks.
Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
In this article we argue for the domain of voice document editing and for the methods of model-based reinforcement learning.
Model-based Reinforcement Learning
reinforcement-learning
+1
Inverse Policy Evaluation for Value-based Sequential Decision-making
In this work, we explore the use of \textit{inverse policy evaluation}, the process of solving for a likely policy given a value function, for deriving behavior from a value function.
Learning and Planning in Average-Reward Markov Decision Processes
We introduce learning and planning algorithms for average-reward MDPs, including 1) the first general proven-convergent off-policy model-free control algorithm without reference states, 2) the first proven-convergent off-policy model-free prediction algorithm, and 3) the first off-policy learning algorithm that converges to the actual value function rather than to the value function plus an offset.
Learning Sparse Representations Incrementally in Deep Reinforcement Learning
Sparse representations have been shown to be useful in deep reinforcement learning for mitigating catastrophic interference and improving the performance of agents in terms of cumulative reward.
Discounted Reinforcement Learning Is Not an Optimization Problem
Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks.
Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning
We explore fixed-horizon temporal difference (TD) methods, reinforcement learning algorithms for a new kind of value function that predicts the sum of rewards over a $\textit{fixed}$ number of future time steps.
Planning with Expectation Models
In particular, we 1) show that planning with an expectation model is equivalent to planning with a distribution model if the state value function is linear in state features, 2) analyze two common parametrization choices for approximating the expectation: linear and non-linear expectation models, 3) propose a sound model-based policy evaluation algorithm and present its convergence results, and 4) empirically demonstrate the effectiveness of the proposed planning algorithm.
Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning
In this paper, we examine an instance of meta-learning in which feature relevance is learned by adapting step size parameters of stochastic gradient descent---building on a variety of prior work in stochastic approximation, machine learning, and artificial neural networks.
Should All Temporal Difference Learning Use Emphasis?
ETD was proposed mainly to address convergence issues of conventional Temporal Difference (TD) learning under off-policy training but it is different from conventional TD learning even under on-policy training.
Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target
Our results show that (1) using off-policy correction can have an adverse effect on the performance of Sarsa and $Q(\sigma)$; (2) increasing the backup length $n$ consistently improved performance across all the different algorithms; and (3) the performance of Sarsa and $Q$-learning was more robust to the effect of the target network update frequency than the performance of Tree Backup, $Q(\sigma)$, and Retrace in this particular task.
Online Off-policy Prediction
The ability to learn behavior-contingent predictions online and off-policy has long been advocated as a key capability of predictive-knowledge learning systems but remained an open algorithmic challenge for decades.
Predicting Periodicity with Temporal Difference Learning
Temporal difference (TD) learning is an important approach in reinforcement learning, as it combines ideas from dynamic programming and Monte Carlo methods in a way that allows for online and incremental model-free learning.
Per-decision Multi-step Temporal Difference Learning with Control Variates
Multi-step temporal difference (TD) learning is an important approach in reinforcement learning, as it unifies one-step TD learning with Monte Carlo methods in a way where intermediate algorithms can outperform either extreme.
Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling
We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful.
Two geometric input transformation methods for fast online reinforcement learning with neural nets
We apply neural nets with ReLU gates in online reinforcement learning.
TIDBD: Adapting Temporal-difference Step-sizes Through Stochastic Meta-descent
In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning.
Reactive Reinforcement Learning in Asynchronous Environments
The relationship between a reinforcement learning (RL) agent and an asynchronous environment is often ignored.
Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods
This paper investigates estimating the variance of a temporal-difference learning agent's update target.
A Deeper Look at Experience Replay
Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay.
Communicative Capital for Prosthetic Agents
This work presents an overarching perspective on the role that machine intelligence can play in enhancing human abilities, especially those that have been diminished due to injury or illness.
A First Empirical Study of Emphatic Temporal Difference Learning
In this paper we present the first empirical study of the emphatic temporal-difference learning algorithm (ETD), comparing it with conventional temporal-difference learning, in particular, with linear TD(0), on on-policy and off-policy variations of the Mountain Car problem.
GQ($λ$) Quick Reference and Implementation Guide
This document should serve as a quick reference for and guide to the implementation of linear GQ($\lambda$), a gradient-based off-policy temporal-difference learning algorithm.
Policy Iterations for Reinforcement Learning Problems in Continuous Time and Space -- Fundamental Theory and Methods
Policy iteration (PI) is a recursive process of policy evaluation and improvement for solving an optimal decision-making/control problem, or in other words, a reinforcement learning (RL) problem.
On Generalized Bellman Equations and Temporal-Difference Learning
As to its soundness, using Markov chain theory, we prove the ergodicity of the joint state-trace process under nonrestrictive conditions, and we show that associated with our scheme is a generalized Bellman equation (for the policy to be evaluated) that depends on both the evolution of $\lambda$ and the unique invariant probability measure of the state-trace process.
Multi-step Reinforcement Learning: A Unifying Algorithm
These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance.
Multi-step Off-policy Learning Without Importance Sampling Ratios
We show that an explicit use of importance sampling ratios can be eliminated by varying the amount of bootstrapping in TD updates in an action-dependent manner.
Learning Representations by Stochastic Meta-Gradient Descent in Neural Networks
In this paper, we introduce a new incremental learning algorithm called crossprop, which learns incoming weights of hidden units based on the meta-gradient descent approach, that was previously introduced by Sutton (1992) and Schraudolph (1999) for learning step-sizes.
Face valuing: Training user interfaces with facial expressions and reinforcement learning
The primary objective of the current work is to demonstrate that a learning agent can reduce the amount of explicit feedback required for adapting to the user's preferences pertaining to a task by learning to perceive a value of its behavior from the human user, particularly from the user's facial expressions---we call this face valuing.
True Online Temporal-Difference Learning
Our results suggest that the true online methods indeed dominate the regular methods.
Learning to Predict Independent of Span
If predictions are made at a high rate or span over a large amount of time, substantial computation can be required to store all relevant observations and to update all predictions when the outcome is finally observed.
True Online Emphatic TD($λ$): Quick Reference and Implementation Guide
This document is a guide to the implementation of true online emphatic TD($\lambda$), a model-free temporal-difference algorithm for learning to make long-term predictions which combines the emphasis idea (Sutton, Mahmood & White 2015) and the true-online idea (van Seijen & Sutton 2014).
Emphatic Temporal-Difference Learning
Emphatic algorithms are temporal-difference learning algorithms that change their effective state distribution by selectively emphasizing and de-emphasizing their updates on different time steps.
An Empirical Evaluation of True Online TD(λ)
Our results confirm the strength of true online TD({\lambda}): 1) for sparse feature vectors, the computational overhead with respect to TD({\lambda}) is minimal; for non-sparse features the computation time is at most twice that of TD({\lambda}), 2) across all domains/representations the learning speed of true online TD({\lambda}) is often better, but never worse than that of TD({\lambda}), and 3) true online TD({\lambda}) is easier to use, because it does not require choosing between trace types, and it is generally more stable with respect to the step-size.
Temporal-Difference Networks
We introduce a generalization of temporal-difference (TD) learning to networks of interrelated predictions.
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning
In this paper we introduce the idea of improving the performance of parametric temporal-difference (TD) learning algorithms by selectively emphasizing or de-emphasizing their updates on different time steps.
Weighted importance sampling for off-policy learning with linear function approximation
Second, we show that these benefits extend to a new weighted-importance-sampling version of off-policy LSTD(lambda).
Universal Option Models
We prove that the UOM of an option can construct a traditional option model given a reward function, and the option-conditional return is computed directly by a single dot-product of the UOM with the reward function.
Temporal-Difference Learning to Assist Human Decision Making during the Control of an Artificial Limb
In the present work, we explore the use of temporal-difference learning and GVFs to predict when users will switch their control influence between the different motor functions of a robot arm.
Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
Our main results are to prove that linear Dyna-style planning converges to a unique solution independent of the generating distribution, under natural conditions.
Off-Policy Actor-Critic
Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning.
Multi-timescale Nexting in a Reinforcement Learning Robot
The term "nexting" has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense.
Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation
We introduce the first temporal-difference learning algorithms that converge with smooth value function approximators, such as neural networks.
Multi-Step Dyna Planning for Policy Evaluation and Control
We extend Dyna planning architecture for policy evaluation and control in two significant aspects.
A Convergent O(n) Temporal-difference Algorithm for Off-policy Learning with Linear Function Approximation
We introduce the first temporal-difference learning algorithm that is stable with linear function approximation and off-policy training, for any finite Markov decision process, target policy, and exciting behavior policy, and whose complexity scales linearly in the number of parameters.
A computational model of hippocampal function in trace conditioning
For trace conditioning, with no contiguity between stimulus and reward, these long-latency temporal elements are vital to learning adaptively timed responses.
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
In particular, we show that options may be used interchangeably with primitive actions in planning methods such as dynamic programming and in learning methods such as Q-learning.
Learning to Predict by the Methods of Temporal Differences
This article introduces a class of incremental learning procedures specialized for prediction that is, for using past experience with an incompletely known system to predict its future behavior.
