Fully Parameterized Quantile Function for Distributional Reinforcement Learning

NeurIPS 2019  ·  Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, Tie-Yan Liu ·

Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.

PDF Abstract NeurIPS 2019 PDF NeurIPS 2019 Abstract

Results from the Paper
Edit

Ranked #3 on Atari Games on Atari 2600 Skiing (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data 		Result Benchmark
Atari Games Atari 2600 Alien FQF Score 16754.6 # 9
Compare
Atari Games Atari 2600 Amidar FQF Score 3165.3 # 6
Compare
Atari Games Atari 2600 Asterix FQF Score 578388.5 # 7
Compare
Atari Games Atari 2600 Asteroids FQF Score 4553.0 # 15
Compare
Atari Games Atari 2600 Battle Zone FQF Score 87928.6 # 8
Compare
Atari Games Atari 2600 Berzerk FQF Score 12422.2 # 8
Compare
Atari Games Atari 2600 Bowling FQF Score 102.3 # 11
Compare
Atari Games Atari 2600 Breakout FQF Score 854.2 # 6
Compare
Atari Games Atari 2600 Chopper Command FQF Score 876460.0 # 7
Compare
Atari Games Atari 2600 Crazy Climber FQF Score 223470.6 # 7
Compare
Atari Games Atari 2600 Fishing Derby FQF Score 52.7 # 12
Compare
Atari Games Atari 2600 Frostbite Fearlessmrx Score 214060 # 5
Compare
Atari Games Atari 2600 Gravitar FQF Score 1406.0 # 20
Compare
Atari Games Atari 2600 HERO FQF Score 30926.2 # 14
Compare
Atari Games Atari 2600 Ice Hockey FQF Score 17.3 # 11
Compare
Atari Games Atari 2600 James Bond FQF Score 87291.7 # 5
Compare
Atari Games Atari 2600 Kung-Fu Master FQF Score 111138.5 # 8
Compare
Atari Games Atari 2600 Ms. Pacman FQF Score 7631.9 # 11
Compare
Atari Games Atari 2600 Phoenix FQF Score 174077.5 # 11
Compare
Atari Games Atari 2600 River Raid FQF Score 23560.7 # 10
Compare
Atari Games Atari 2600 Robotank FQF Score 75.7 # 9
Compare
Atari Games Atari 2600 Skiing FQF Score -9085.3 # 3
Compare
Atari Games Atari 2600 Space Invaders FQF Score 46498.3 # 8
Compare
Atari Games Atari 2600 Star Gunner FQF Score 131981.2 # 12
Compare
Atari Games Atari 2600 Wizard of Wor FQF Score 44782.6 # 9
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy