Deep Reinforcement Learning with Double Q-learning

22 Sep 2015Hado van HasseltArthur GuezDavid Silver

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented... (read more)

PDF Abstract

Code


Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Atari Games Atari 2600 Alien Prior+Duel hs Score 823.7 # 33
Atari Games Atari 2600 Alien DDQN (tuned) hs Score 1033.4 # 29
Atari Games Atari 2600 Alien DQN hs Score 634.0 # 35
Atari Games Atari 2600 Alien DQN noop Score 1620.0 # 23
Atari Games Atari 2600 Amidar DQN hs Score 178.4 # 32
Atari Games Atari 2600 Amidar DQN noop Score 978.0 # 19
Atari Games Atari 2600 Amidar Prior+Duel hs Score 238.4 # 26
Atari Games Atari 2600 Amidar DDQN (tuned) hs Score 169.1 # 35
Atari Games Atari 2600 Assault DQN noop Score 4280.4 # 22
Atari Games Atari 2600 Assault DQN hs Score 3489.3 # 26
Atari Games Atari 2600 Assault Prior+Duel hs Score 10950.6 # 10
Atari Games Atari 2600 Assault DDQN (tuned) hs Score 6060.8 # 17
Atari Games Atari 2600 Asterix DDQN (tuned) hs Score 16837 # 25
Atari Games Atari 2600 Asterix DQN hs Score 3170.5 # 33
Atari Games Atari 2600 Asterix DQN noop Score 4359 # 30
Atari Games Atari 2600 Asterix Prior+Duel hs Score 364200 # 6
Atari Games Atari 2600 Asteroids Prior+Duel hs Score 1021.9 # 32
Atari Games Atari 2600 Asteroids DQN hs Score 1458.7 # 27
Atari Games Atari 2600 Asteroids DQN noop Score 1364.5 # 28
Atari Games Atari 2600 Asteroids DDQN (tuned) hs Score 1193.2 # 29
Atari Games Atari 2600 Atlantis DDQN (tuned) hs Score 319688.0 # 26
Atari Games Atari 2600 Atlantis DQN hs Score 292491.0 # 28
Atari Games Atari 2600 Atlantis DQN noop Score 279987.0 # 29
Atari Games Atari 2600 Atlantis Prior+Duel hs Score 423252.0 # 20
Atari Games Atari 2600 Bank Heist DDQN (tuned) hs Score 886.0 # 23
Atari Games Atari 2600 Bank Heist DQN noop Score 455.0 # 28
Atari Games Atari 2600 Bank Heist DQN hs Score 312.7 # 31
Atari Games Atari 2600 Bank Heist Prior+Duel hs Score 1004.6 # 18
Atari Games Atari 2600 Battle Zone DDQN (tuned) hs Score 24740.0 # 25
Atari Games Atari 2600 Battle Zone DQN noop Score 29900.0 # 19
Atari Games Atari 2600 Battle Zone DQN hs Score 23750.0 # 26
Atari Games Atari 2600 Battle Zone Prior+Duel hs Score 30650.0 # 18
Atari Games Atari 2600 Beam Rider DQN hs Score 9743.2 # 25
Atari Games Atari 2600 Beam Rider DQN noop Score 8627.5 # 26
Atari Games Atari 2600 Beam Rider DDQN (tuned) hs Score 17417.2 # 15
Atari Games Atari 2600 Beam Rider Prior+Duel hs Score 37412.2 # 5
Atari Games Atari 2600 Berzerk DQN hs Score 493.4 # 31
Atari Games Atari 2600 Berzerk Prior+Duel hs Score 2178.6 # 9
Atari Games Atari 2600 Berzerk DDQN (tuned) hs Score 1011.1 # 21
Atari Games Atari 2600 Berzerk DQN noop Score 585.6 # 29
Atari Games Atari 2600 Bowling DQN noop Score 50.4 # 23
Atari Games Atari 2600 Bowling DDQN (tuned) hs Score 69.6 # 13
Atari Games Atari 2600 Bowling DQN hs Score 56.5 # 20
Atari Games Atari 2600 Bowling Prior+Duel hs Score 50.4 # 23
Atari Games Atari 2600 Boxing DQN hs Score 70.3 # 25
Atari Games Atari 2600 Boxing Prior+Duel hs Score 79.2 # 19
Atari Games Atari 2600 Boxing DDQN (tuned) hs Score 73.5 # 22
Atari Games Atari 2600 Boxing DQN noop Score 88.0 # 18
Atari Games Atari 2600 Breakout DQN hs Score 354.5 # 28
Atari Games Atari 2600 Breakout DDQN (tuned) hs Score 368.9 # 24
Atari Games Atari 2600 Breakout Prior+Duel hs Score 354.6 # 27
Atari Games Atari 2600 Breakout DQN noop Score 385.5 # 21
Atari Games Atari 2600 Centipede DDQN (tuned) hs Score 3853.5 # 30
Atari Games Atari 2600 Centipede Prior+Duel hs Score 5570.2 # 20
Atari Games Atari 2600 Centipede DQN hs Score 3973.9 # 29
Atari Games Atari 2600 Centipede DQN noop Score 4657.7 # 23
Atari Games Atari 2600 Chopper Command DDQN (tuned) hs Score 3495.0 # 32
Atari Games Atari 2600 Chopper Command DQN noop Score 6126.0 # 21
Atari Games Atari 2600 Chopper Command DQN hs Score 5017.0 # 25
Atari Games Atari 2600 Chopper Command Prior+Duel hs Score 8058.0 # 16
Atari Games Atari 2600 Crazy Climber Prior+Duel hs Score 127853.0 # 18
Atari Games Atari 2600 Crazy Climber DDQN (tuned) hs Score 113782.0 # 26
Atari Games Atari 2600 Crazy Climber DQN noop Score 110763.0 # 28
Atari Games Atari 2600 Crazy Climber DQN hs Score 98128.0 # 31
Atari Games Atari 2600 Demon Attack Prior+Duel hs Score 73371.3 # 14
Atari Games Atari 2600 Demon Attack DDQN (tuned) hs Score 69803.4 # 18
Atari Games Atari 2600 Demon Attack DQN noop Score 12149.4 # 30
Atari Games Atari 2600 Demon Attack DQN hs Score 12550.7 # 29
Atari Games Atari 2600 Double Dunk DDQN (tuned) hs Score -0.3 # 19
Atari Games Atari 2600 Double Dunk Prior+Duel hs Score -10.7 # 27
Atari Games Atari 2600 Double Dunk DQN hs Score -6.0 # 24
Atari Games Atari 2600 Double Dunk DQN noop Score -6.6 # 25
Atari Games Atari 2600 Enduro Prior+Duel hs Score 2223.9 # 9
Atari Games Atari 2600 Enduro DQN noop Score 729.0 # 22
Atari Games Atari 2600 Enduro DDQN (tuned) hs Score 1216.6 # 19
Atari Games Atari 2600 Enduro DQN hs Score 626.7 # 24
Atari Games Atari 2600 Fishing Derby DDQN (tuned) hs Score 3.2 # 27
Atari Games Atari 2600 Fishing Derby DQN hs Score -1.6 # 29
Atari Games Atari 2600 Fishing Derby DQN noop Score -4.9 # 31
Atari Games Atari 2600 Fishing Derby Prior+Duel hs Score 17.0 # 21
Atari Games Atari 2600 Freeway DQN hs Score 26.9 # 25
Atari Games Atari 2600 Freeway DQN noop Score 30.8 # 15
Atari Games Atari 2600 Freeway Prior+Duel hs Score 28.2 # 22
Atari Games Atari 2600 Freeway DDQN (tuned) hs Score 28.8 # 21
Atari Games Atari 2600 Frostbite Prior+Duel hs Score 4038.4 # 12
Atari Games Atari 2600 Frostbite DQN noop Score 797.4 # 27
Atari Games Atari 2600 Frostbite DQN hs Score 496.1 # 30
Atari Games Atari 2600 Frostbite DDQN (tuned) hs Score 1448.1 # 24
Atari Games Atari 2600 Gopher DDQN (tuned) hs Score 15253.0 # 20
Atari Games Atari 2600 Gopher DQN noop Score 8777.4 # 25
Atari Games Atari 2600 Gopher DQN hs Score 8190.4 # 28
Atari Games Atari 2600 Gopher Prior+Duel hs Score 105148.4 # 6
Atari Games Atari 2600 Gravitar DQN noop Score 473.0 # 22
Atari Games Atari 2600 Gravitar DQN hs Score 298.0 # 33
Atari Games Atari 2600 Gravitar DDQN (tuned) hs Score 200.5 # 39
Atari Games Atari 2600 Gravitar Prior+Duel hs Score 167.0 # 40
Atari Games Atari 2600 HERO DQN hs Score 14992.9 # 27
Atari Games Atari 2600 HERO DQN noop Score 20437.8 # 22
Atari Games Atari 2600 HERO DDQN (tuned) hs Score 14892.5 # 28
Atari Games Atari 2600 HERO Prior+Duel hs Score 15459.2 # 25
Atari Games Atari 2600 Ice Hockey DQN hs Score -1.6 # 17
Atari Games Atari 2600 Ice Hockey DQN noop Score -1.9 # 19
Atari Games Atari 2600 Ice Hockey Prior+Duel hs Score 0.5 # 10
Atari Games Atari 2600 Ice Hockey DDQN (tuned) hs Score -2.5 # 21
Atari Games Atari 2600 James Bond DDQN (tuned) hs Score 573.0 # 24
Atari Games Atari 2600 James Bond Prior+Duel hs Score 585.0 # 22
Atari Games Atari 2600 James Bond DQN noop Score 768.5 # 18
Atari Games Atari 2600 James Bond DQN hs Score 697.5 # 19
Atari Games Atari 2600 Kangaroo DQN noop Score 7259.0 # 18
Atari Games Atari 2600 Kangaroo DQN hs Score 4496.0 # 20
Atari Games Atari 2600 Kangaroo DDQN (tuned) hs Score 11204.0 # 14
Atari Games