Dueling Network Architectures for Deep Reinforcement Learning

20 Nov 2015  ·  Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, Nando de Freitas ·

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien Duel noop Score 4461.4 # 20
Atari Games Atari 2600 Alien Prior+Duel hs Score 823.7 # 42
Atari Games Atari 2600 Alien Prior+Duel noop Score 3941.0 # 23
Atari Games Atari 2600 Alien Duel hs Score 1486.5 # 33
Atari Games Atari 2600 Alien DDQN (tuned) noop Score 3747.7 # 24
Atari Games Atari 2600 Amidar DDQN (tuned) noop Score 1793.3 # 13
Atari Games Atari 2600 Amidar Prior+Duel noop Score 2296.8 # 10
Atari Games Atari 2600 Amidar Prior+Duel hs Score 238.4 # 34
Atari Games Atari 2600 Amidar Duel noop Score 2354.5 # 9
Atari Games Atari 2600 Amidar Duel hs Score 172.7 # 43
Atari Games Atari 2600 Assault Prior+Duel noop Score 11477.0 # 15
Atari Games Atari 2600 Assault Duel hs Score 3994.8 # 31
Atari Games Atari 2600 Assault Duel noop Score 4621.0 # 29
Atari Games Atari 2600 Assault DDQN (tuned) noop Score 5393.2 # 28
Atari Games Atari 2600 Assault Prior+Duel hs Score 10950.6 # 17
Atari Games Atari 2600 Asterix Duel hs Score 15840.0 # 35
Atari Games Atari 2600 Asterix Prior+Duel hs Score 364200.0 # 11
Atari Games Atari 2600 Asterix Duel noop Score 28188.0 # 23
Atari Games Atari 2600 Asterix Prior+Duel noop Score 375080.0 # 10
Atari Games Atari 2600 Asterix DDQN (tuned) noop Score 17356.5 # 32
Atari Games Atari 2600 Asteroids DDQN (tuned) noop Score 734.7 # 42
Atari Games Atari 2600 Asteroids Duel hs Score 2035.4 # 26
Atari Games Atari 2600 Asteroids Prior+Duel noop Score 1192.7 # 37
Atari Games Atari 2600 Asteroids Duel noop Score 2837.7 # 22
Atari Games Atari 2600 Atlantis DDQN (tuned) noop Score 106056.0 # 38
Atari Games Atari 2600 Atlantis Prior+Duel noop Score 395762.0 # 28
Atari Games Atari 2600 Atlantis Duel hs Score 445360.0 # 26
Atari Games Atari 2600 Atlantis Duel noop Score 382572.0 # 29
Atari Games Atari 2600 Bank Heist Prior+Duel noop Score 1503.1 # 6
Atari Games Atari 2600 Bank Heist Duel hs Score 1129.3 # 20
Atari Games Atari 2600 Bank Heist Duel noop Score 1611.9 # 5
Atari Games Atari 2600 Bank Heist DDQN (tuned) noop Score 1030.6 # 24
Atari Games Atari 2600 Battle Zone Prior+Duel noop Score 35520.0 # 19
Atari Games Atari 2600 Battle Zone Duel noop Score 37150.0 # 18
Atari Games Atari 2600 Battle Zone DDQN (tuned) noop Score 31700.0 # 22
Atari Games Atari 2600 Battle Zone Duel hs Score 31320.0 # 24
Atari Games Atari 2600 Beam Rider DDQN (tuned) noop Score 13772.8 # 27
Atari Games Atari 2600 Beam Rider Prior+Duel noop Score 30276.5 # 14
Atari Games Atari 2600 Beam Rider Duel noop Score 12164.0 # 30
Atari Games Atari 2600 Beam Rider Duel hs Score 14591.3 # 25
Atari Games Atari 2600 Berzerk DDQN (tuned) noop Score 1225.4 # 24
Atari Games Atari 2600 Berzerk Duel noop Score 1472.6 # 20
Atari Games Atari 2600 Berzerk Prior+Duel noop Score 3409.0 # 11
Atari Games Atari 2600 Berzerk Duel hs Score 910.6 # 29
Atari Games Atari 2600 Bowling DDQN (tuned) noop Score 68.1 # 20
Atari Games Atari 2600 Bowling Duel noop Score 65.5 # 22
Atari Games Atari 2600 Bowling Prior+Duel noop Score 46.7 # 34
Atari Games Atari 2600 Bowling Duel hs Score 65.7 # 21
Atari Games Atari 2600 Boxing Duel hs Score 77.3 # 32
Atari Games Atari 2600 Boxing Duel noop Score 99.4 # 16
Atari Games Atari 2600 Boxing Prior+Duel noop Score 98.9 # 19
Atari Games Atari 2600 Boxing DDQN (tuned) noop Score 91.6 # 29
Atari Games Atari 2600 Breakout DDQN (tuned) noop Score 418.5 # 26
Atari Games Atari 2600 Breakout Prior+Duel noop Score 366.0 # 33
Atari Games Atari 2600 Breakout Duel noop Score 345.3 # 38
Atari Games Atari 2600 Breakout Duel hs Score 411.6 # 27
Atari Games Atari 2600 Centipede DDQN (tuned) noop Score 5409.4 # 29
Atari Games Atari 2600 Centipede Prior+Duel noop Score 7687.5 # 23
Atari Games Atari 2600 Centipede Duel noop Score 7561.4 # 25
Atari Games Atari 2600 Centipede Duel hs Score 4881.0 # 30
Atari Games Atari 2600 Chopper Command Duel hs Score 3784.0 # 36
Atari Games Atari 2600 Chopper Command Duel noop Score 11215.0 # 19
Atari Games Atari 2600 Chopper Command DDQN (tuned) noop Score 5809.0 # 29
Atari Games Atari 2600 Chopper Command Prior+Duel noop Score 13185.0 # 17
Atari Games Atari 2600 Crazy Climber DDQN (tuned) noop Score 117282.0 # 32
Atari Games Atari 2600 Crazy Climber Duel hs Score 124566.0 # 28
Atari Games Atari 2600 Crazy Climber Duel noop Score 143570.0 # 18
Atari Games Atari 2600 Crazy Climber Prior+Duel noop Score 162224.0 # 14
Atari Games Atari 2600 Defender Prior+Duel hs Score 34415.0 # 19
Atari Games Atari 2600 Defender Prior+Duel noop Score 41324.5 # 17
Atari Games Atari 2600 Defender Duel noop Score 42214.0 # 16
Atari Games Atari 2600 Demon Attack Prior+Duel noop Score 72878.6 # 23
Atari Games Atari 2600 Demon Attack DDQN (tuned) noop Score 58044.2 # 32
Atari Games Atari 2600 Demon Attack Duel noop Score 60813.3 # 31
Atari Games Atari 2600 Demon Attack Duel hs Score 56322.8 # 33
Atari Games Atari 2600 Double Dunk Duel hs Score -0.8 # 26
Atari Games Atari 2600 Double Dunk Duel noop Score 0.1 # 22
Atari Games Atari 2600 Double Dunk Prior+Duel noop Score -12.5 # 26
Atari Games Atari 2600 Double Dunk DDQN (tuned) noop Score -5.5 # 26
Atari Games Atari 2600 Enduro Duel hs Score 2077.4 # 18
Atari Games Atari 2600 Enduro Prior+Duel noop Score 2306.4 # 11
Atari Games Atari 2600 Enduro Duel noop Score 2258.2 # 12
Atari Games Atari 2600 Enduro DDQN (tuned) noop Score 1211.8 # 28
Atari Games Atari 2600 Fishing Derby Prior+Duel noop Score 41.3 # 17
Atari Games Atari 2600 Fishing Derby DDQN (tuned) noop Score 15.5 # 30
Atari Games Atari 2600 Fishing Derby Duel hs Score -4.1 # 36
Atari Games Atari 2600 Fishing Derby Duel noop Score 46.4 # 13
Atari Games Atari 2600 Freeway DDQN (tuned) noop Score 33.3 # 17
Atari Games Atari 2600 Freeway Duel noop Score 0.0 # 57
Atari Games Atari 2600 Freeway Prior+Duel noop Score 33.0 # 19
Atari Games Atari 2600 Freeway Duel hs Score 0.2 # 53
Atari Games Atari 2600 Frostbite Duel noop Score 4672.8 # 16
Atari Games Atari 2600 Frostbite Prior+Duel noop Score 7413.0 # 13
Atari Games Atari 2600 Frostbite DDQN (tuned) noop Score 1683.3 # 32
Atari Games Atari 2600 Frostbite Duel hs Score 2332.4 # 29
Atari Games Atari 2600 Gopher Duel noop Score 15718.4 # 26
Atari Games Atari 2600 Gopher Prior+Duel noop Score 104368.2 # 11
Atari Games Atari 2600 Gopher Duel hs Score 20051.4 # 23
Atari Games Atari 2600 Gopher DDQN (tuned) noop Score 14840.8 # 28
Atari Games Atari 2600 Gravitar Duel noop Score 588.0 # 27
Atari Games Atari 2600 Gravitar DDQN (tuned) noop Score 412.0 # 38
Atari Games Atari 2600 Gravitar Duel hs Score 297.0 # 45
Atari Games Atari 2600 Gravitar Prior+Duel noop Score 238.0 # 50
Atari Games Atari 2600 HERO DDQN (tuned) noop Score 20130.2 # 30
Atari Games Atari 2600 HERO Duel hs Score 15207.9 # 33
Atari Games Atari 2600 HERO Prior+Duel noop Score 21036.5 # 25
Atari Games Atari 2600 HERO Duel noop Score 20818.2 # 28
Atari Games Atari 2600 Ice Hockey Prior+Duel noop Score -0.4 # 20
Atari Games Atari 2600 Ice Hockey Duel noop Score 0.5 # 17
Atari Games Atari 2600 Ice Hockey DDQN (tuned) noop Score -2.7 # 20
Atari Games Atari 2600 Ice Hockey Duel hs Score -1.3 # 20
Atari Games Atari 2600 James Bond Prior+Duel noop Score 812.0 # 26
Atari Games Atari 2600 James Bond Duel noop Score 1312.5 # 21
Atari Games Atari 2600 James Bond DDQN (tuned) noop Score 1358.0 # 20
Atari Games Atari 2600 James Bond Duel hs Score 835.5 # 25
Atari Games Atari 2600 Kangaroo Prior+Duel noop Score 1792.0 # 34
Atari Games Atari 2600 Kangaroo Duel hs Score 10334.0 # 25
Atari Games Atari 2600 Kangaroo DDQN (tuned) noop Score 12992.0 # 18
Atari Games Atari 2600 Kangaroo Duel noop Score 14854.0 # 8
Atari Games Atari 2600 Krull Prior+Duel noop Score 10374.4 # 18
Atari Games Atari 2600 Krull DDQN (tuned) noop Score 7920.5 # 31
Atari Games Atari 2600 Krull Duel hs Score 8051.6 # 30
Atari Games Atari 2600 Krull Duel noop Score 11451.9 # 11
Atari Games Atari 2600 Kung-Fu Master DDQN (tuned) noop Score 29710.0 # 33
Atari Games Atari 2600 Kung-Fu Master Prior+Duel noop Score 48375.0 # 17
Atari Games Atari 2600 Kung-Fu Master Duel noop Score 34294.0 # 28
Atari Games Atari 2600 Kung-Fu Master Duel hs Score 24288.0 # 37
Atari Games Atari 2600 Montezuma's Revenge Duel hs Score 22.0 # 38
Atari Games Atari 2600 Ms. Pacman Duel noop Score 6283.5 # 15
Atari Games Atari 2600 Ms. Pacman DDQN (tuned) noop Score 2711.4 # 29
Atari Games Atari 2600 Ms. Pacman Duel hs Score 2250.6 # 33
Atari Games Atari 2600 Ms. Pacman Prior+Duel noop Score 3327.3 # 26
Atari Games Atari 2600 Name This Game Prior+Duel noop Score 15572.5 # 15
Atari Games Atari 2600 Name This Game Duel noop Score 11971.1 # 24
Atari Games Atari 2600 Name This Game Duel hs Score 11185.1 # 26
Atari Games Atari 2600 Name This Game DDQN (tuned) noop Score 10616.0 # 28
Atari Games Atari 2600 Phoenix Prior+Duel hs Score 63597.0 # 13
Atari Games Atari 2600 Pong Prior+Duel noop Score 20.9 # 17
Atari Games Atari 2600 Pong Duel hs Score 18.8 # 38
Atari Games Atari 2600 Pong DDQN (tuned) noop Score 20.9 # 17
Atari Games Atari 2600 Pong Duel noop Score 21.0 # 1
Atari Games Atari 2600 Private Eye Prior+Duel noop Score 206.0 # 34
Atari Games Atari 2600 Private Eye DDQN (tuned) noop Score 129.7 # 40
Atari Games Atari 2600 Private Eye Duel hs Score 292.6 # 29
Atari Games Atari 2600 Private Eye Duel noop Score 103.0 # 42
Atari Games Atari 2600 Q*Bert DDQN (tuned) noop Score 15088.5 # 28
Atari Games Atari 2600 Q*Bert Prior+Duel noop Score 18760.3 # 21
Atari Games Atari 2600 Q*Bert Duel noop Score 19220.3 # 20
Atari Games Atari 2600 Q*Bert Duel hs Score 14175.8 # 32
Atari Games Atari 2600 River Raid Duel hs Score 16569.4 # 17
Atari Games Atari 2600 River Raid DDQN (tuned) noop Score 14884.5 # 20
Atari Games Atari 2600 River Raid Prior+Duel noop Score 20607.6 # 12
Atari Games Atari 2600 River Raid Duel noop Score 21162.6 # 11
Atari Games Atari 2600 Road Runner Prior+Duel noop Score 62151.0 # 14
Atari Games Atari 2600 Road Runner DDQN (tuned) noop Score 44127.0 # 29
Atari Games Atari 2600 Road Runner Duel noop Score 69524.0 # 12
Atari Games Atari 2600 Road Runner Duel hs Score 58549.0 # 16
Atari Games Atari 2600 Robotank Duel noop Score 65.3 # 14
Atari Games Atari 2600 Robotank DDQN (tuned) noop Score 65.1 # 15
Atari Games Atari 2600 Robotank Duel hs Score 62.0 # 22
Atari Games Atari 2600 Robotank Prior+Duel noop Score 27.5 # 33
Atari Games Atari 2600 Seaquest Prior+Duel noop Score 931.6 # 47
Atari Games Atari 2600 Seaquest DDQN (tuned) noop Score 16452.7 # 18
Atari Games Atari 2600 Seaquest Duel hs Score 37361.6 # 12
Atari Games Atari 2600 Seaquest Duel noop Score 50254.2 # 11
Atari Games Atari 2600 Space Invaders Prior+Duel noop Score 15311.5 # 16
Atari Games Atari 2600 Space Invaders Duel noop Score 6427.3 # 18
Atari Games Atari 2600 Space Invaders Duel hs Score 5993.1 # 19
Atari Games Atari 2600 Space Invaders DDQN (tuned) noop Score 2525.5 # 33
Atari Games Atari 2600 Star Gunner DDQN (tuned) noop Score 60142.0 # 27
Atari Games Atari 2600 Star Gunner Duel hs Score 90804.0 # 17
Atari Games Atari 2600 Star Gunner Duel noop Score 89238.0 # 18
Atari Games Atari 2600 Star Gunner Prior+Duel noop Score 125117.0 # 15
Atari Games Atari 2600 Tennis Duel hs Score 4.4 # 17
Atari Games Atari 2600 Tennis Duel noop Score 5.1 # 16
Atari Games Atari 2600 Tennis Prior+Duel noop Score 0.0 # 20
Atari Games Atari 2600 Tennis DDQN (tuned) noop Score -22.8 # 29
Atari Games Atari 2600 Time Pilot Duel noop Score 11666.0 # 21
Atari Games Atari 2600 Time Pilot DDQN (tuned) noop Score 8339.0 # 27
Atari Games Atari 2600 Time Pilot Duel hs Score 6601.0 # 32
Atari Games Atari 2600 Time Pilot Prior+Duel noop Score 7553.0 # 30
Atari Games Atari 2600 Tutankham Prior+Duel noop Score 245.9 # 17
Atari Games Atari 2600 Tutankham DDQN (tuned) noop Score 218.4 # 21
Atari Games Atari 2600 Tutankham Duel noop Score 211.4 # 23
Atari Games Atari 2600 Tutankham Duel hs Score 48.0 # 41
Atari Games Atari 2600 Up and Down DDQN (tuned) noop Score 22972.2 # 28
Atari Games Atari 2600 Up and Down Duel hs Score 24759.2 # 27
Atari Games Atari 2600 Up and Down Prior+Duel noop Score 33879.1 # 24
Atari Games Atari 2600 Up and Down Duel noop Score 44939.6 # 23
Atari Games Atari 2600 Venture Duel hs Score 200.0 # 28
Atari Games Atari 2600 Venture DDQN (tuned) noop Score 98.0 # 32
Atari Games Atari 2600 Venture Prior+Duel noop Score 48.0 # 37
Atari Games Atari 2600 Venture Duel noop Score 497.0 # 22
Atari Games Atari 2600 Video Pinball Duel noop Score 98209.5 # 33
Atari Games Atari 2600 Video Pinball Duel hs Score 110976.2 # 32
Atari Games Atari 2600 Video Pinball DDQN (tuned) noop Score 309941.9 # 23
Atari Games Atari 2600 Video Pinball Prior+Duel noop Score 479197.0 # 17
Atari Games Atari 2600 Wizard of Wor Duel hs Score 7054.0 # 27
Atari Games Atari 2600 Wizard of Wor DDQN (tuned) noop Score 7492.0 # 25
Atari Games Atari 2600 Wizard of Wor Duel noop Score 7855.0 # 24
Atari Games Atari 2600 Wizard of Wor Prior+Duel noop Score 12352.0 # 17
Atari Games Atari 2600 Zaxxon Duel hs Score 10164.0 # 26
Atari Games Atari 2600 Zaxxon Prior+Duel noop Score 13886.0 # 19
Atari Games Atari 2600 Zaxxon Duel noop Score 12944.0 # 21
Atari Games Atari 2600 Zaxxon DDQN (tuned) noop Score 10163.0 # 27

Methods