Dueling Network Architectures for Deep Reinforcement Learning

20 Nov 2015Ziyu WangTom SchaulMatteo HesselHado van HasseltMarc LanctotNando de Freitas

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Atari Games Atari 2600 Alien DDQN (tuned) noop Score 3747.7 # 16
Atari Games Atari 2600 Alien Prior+Duel noop Score 3941.0 # 15
Atari Games Atari 2600 Alien Duel noop Score 4461.4 # 13
Atari Games Atari 2600 Alien Duel hs Score 1486.5 # 25
Atari Games Atari 2600 Alien Prior+Duel hs Score 823.7 # 33
Atari Games Atari 2600 Amidar DDQN (tuned) noop Score 1793.3 # 10
Atari Games Atari 2600 Amidar Duel hs Score 172.7 # 34
Atari Games Atari 2600 Amidar Prior+Duel noop Score 2296.8 # 8
Atari Games Atari 2600 Amidar Prior+Duel hs Score 238.4 # 26
Atari Games Atari 2600 Amidar Duel noop Score 2354.5 # 7
Atari Games Atari 2600 Assault Duel hs Score 3994.8 # 23
Atari Games Atari 2600 Assault Prior+Duel noop Score 11477.0 # 8
Atari Games Atari 2600 Assault Prior+Duel hs Score 10950.6 # 10
Atari Games Atari 2600 Assault DDQN (tuned) noop Score 5393.2 # 20
Atari Games Atari 2600 Assault Duel noop Score 4621.0 # 21
Atari Games Atari 2600 Asterix Prior+Duel hs Score 364200.0 # 6
Atari Games Atari 2600 Asterix Duel noop Score 28188 # 15
Atari Games Atari 2600 Asterix DDQN (tuned) noop Score 17356.5 # 23
Atari Games Atari 2600 Asterix Duel hs Score 15840 # 26
Atari Games Atari 2600 Asterix Prior+Duel noop Score 375080 # 5
Atari Games Atari 2600 Asteroids Duel hs Score 2035.4 # 20
Atari Games Atari 2600 Asteroids DDQN (tuned) noop Score 734.7 # 35
Atari Games Atari 2600 Asteroids Prior+Duel noop Score 1192.7 # 30
Atari Games Atari 2600 Asteroids Duel noop Score 2837.7 # 16
Atari Games Atari 2600 Atlantis Duel noop Score 382572.0 # 22
Atari Games Atari 2600 Atlantis DDQN (tuned) noop Score 106056.0 # 31
Atari Games Atari 2600 Atlantis Prior+Duel noop Score 395762.0 # 21
Atari Games Atari 2600 Atlantis Duel hs Score 445360.0 # 19
Atari Games Atari 2600 Bank Heist Duel hs Score 1129.3 # 14
Atari Games Atari 2600 Bank Heist Duel noop Score 1611.9 # 3
Atari Games Atari 2600 Bank Heist Prior+Duel noop Score 1503.1 # 4
Atari Games Atari 2600 Bank Heist DDQN (tuned) noop Score 1030.6 # 17
Atari Games Atari 2600 Battle Zone DDQN (tuned) noop Score 31700.0 # 15
Atari Games Atari 2600 Battle Zone Prior+Duel noop Score 35520.0 # 12
Atari Games Atari 2600 Battle Zone Duel noop Score 37150.0 # 11
Atari Games Atari 2600 Battle Zone Duel hs Score 31320.0 # 17
Atari Games Atari 2600 Beam Rider Duel noop Score 12164.0 # 22
Atari Games Atari 2600 Beam Rider Duel hs Score 14591.3 # 17
Atari Games Atari 2600 Beam Rider DDQN (tuned) noop Score 13772.8 # 19
Atari Games Atari 2600 Beam Rider Prior+Duel noop Score 30276.5 # 9
Atari Games Atari 2600 Berzerk Duel hs Score 910.6 # 22
Atari Games Atari 2600 Berzerk Prior+Duel noop Score 3409.0 # 6
Atari Games Atari 2600 Berzerk Duel noop Score 1472.6 # 13
Atari Games Atari 2600 Berzerk DDQN (tuned) noop Score 1225.4 # 17
Atari Games Atari 2600 Bowling Duel noop Score 65.5 # 16
Atari Games Atari 2600 Bowling Prior+Duel noop Score 46.7 # 25
Atari Games Atari 2600 Bowling Duel hs Score 65.7 # 15
Atari Games Atari 2600 Bowling DDQN (tuned) noop Score 68.1 # 14
Atari Games Atari 2600 Boxing DDQN (tuned) noop Score 91.6 # 17
Atari Games Atari 2600 Boxing Duel hs Score 77.3 # 20
Atari Games Atari 2600 Boxing Prior+Duel noop Score 98.9 # 8
Atari Games Atari 2600 Boxing Duel noop Score 99.4 # 6
Atari Games Atari 2600 Breakout Prior+Duel noop Score 366.0 # 25
Atari Games Atari 2600 Breakout DDQN (tuned) noop Score 418.5 # 18
Atari Games Atari 2600 Breakout Duel hs Score 411.6 # 19
Atari Games Atari 2600 Breakout Duel noop Score 345.3 # 30
Atari Games Atari 2600 Centipede Duel noop Score 7561.4 # 17
Atari Games Atari 2600 Centipede Prior+Duel noop Score 7687.5 # 15
Atari Games Atari 2600 Centipede Duel hs Score 4881.0 # 22
Atari Games Atari 2600 Centipede DDQN (tuned) noop Score 5409.4 # 21
Atari Games Atari 2600 Chopper Command Duel hs Score 3784.0 # 29
Atari Games Atari 2600 Chopper Command Duel noop Score 11215.0 # 13
Atari Games Atari 2600 Chopper Command DDQN (tuned) noop Score 5809.0 # 22
Atari Games Atari 2600 Chopper Command Prior+Duel noop Score 13185.0 # 11
Atari Games Atari 2600 Crazy Climber Duel noop Score 143570.0 # 11
Atari Games Atari 2600 Crazy Climber Duel hs Score 124566.0 # 20
Atari Games Atari 2600 Crazy Climber DDQN (tuned) noop Score 117282.0 # 24
Atari Games Atari 2600 Crazy Climber Prior+Duel noop Score 162224.0 # 9
Atari Games Atari 2600 Defender Prior+Duel noop Score 41324.5 # 11
Atari Games Atari 2600 Defender Duel noop Score 42214.0 # 10
Atari Games Atari 2600 Defender Prior+Duel hs Score 34415.0 # 12
Atari Games Atari 2600 Demon Attack Prior+Duel noop Score 72878.6 # 15
Atari Games Atari 2600 Demon Attack Duel noop Score 60813.3 # 23
Atari Games Atari 2600 Demon Attack DDQN (tuned) noop Score 58044.2 # 24
Atari Games Atari 2600 Demon Attack Duel hs Score 56322.8 # 25
Atari Games Atari 2600 Double Dunk Prior+Duel noop Score -12.5 # 30
Atari Games Atari 2600 Double Dunk DDQN (tuned) noop Score -5.5 # 23
Atari Games Atari 2600 Double Dunk Duel hs Score -0.8 # 21
Atari Games Atari 2600 Double Dunk Duel noop Score 0.1 # 16
Atari Games Atari 2600 Enduro DDQN (tuned) noop Score 1211.8 # 20
Atari Games Atari 2600 Enduro Prior+Duel noop Score 2306.4 # 6
Atari Games Atari 2600 Enduro Duel noop Score 2258.2 # 7
Atari Games Atari 2600 Enduro Duel hs Score 2077.4 # 12
Atari Games Atari 2600 Fishing Derby Duel hs Score -4.1 # 30
Atari Games Atari 2600 Fishing Derby Duel noop Score 46.4 # 6
Atari Games Atari 2600 Fishing Derby DDQN (tuned) noop Score 15.5 # 22
Atari Games Atari 2600 Fishing Derby Prior+Duel noop Score 41.3 # 10
Atari Games Atari 2600 Freeway Duel hs Score 0.2 # 33
Atari Games Atari 2600 Freeway Duel noop Score 0.0 # 35
Atari Games Atari 2600 Freeway Prior+Duel noop Score 33.0 # 7
Atari Games Atari 2600 Freeway DDQN (tuned) noop Score 33.3 # 5
Atari Games Atari 2600 Frostbite Duel noop Score 4672.8 # 8
Atari Games Atari 2600 Frostbite DDQN (tuned) noop Score 1683.3 # 23
Atari Games Atari 2600 Frostbite Prior+Duel noop Score 7413.0 # 5
Atari Games Atari 2600 Frostbite Duel hs Score 2332.4 # 20
Atari Games Atari 2600 Gopher Duel hs Score 20051.4 # 16
Atari Games Atari 2600 Gopher DDQN (tuned) noop Score 14840.8 # 21
Atari Games Atari 2600 Gopher Prior+Duel noop Score 104368.2 # 7
Atari Games Atari 2600 Gopher Duel noop Score 15718.4 # 19
Atari Games Atari 2600 Gravitar DDQN (tuned) noop Score 412.0 # 27
Atari Games Atari 2600 Gravitar Duel hs Score 297.0 # 34
Atari Games Atari 2600 Gravitar Prior+Duel noop Score 238.0 # 38
Atari Games Atari 2600 Gravitar Duel noop Score 588.0 # 16
Atari Games Atari 2600 HERO Duel noop Score 20818.2 # 21
Atari Games Atari 2600 HERO Prior+Duel noop Score 21036.5 # 18
Atari Games Atari 2600 HERO DDQN (tuned) noop Score 20130.2 # 23
Atari Games Atari 2600 HERO Duel hs Score 15207.9 # 26
Atari Games Atari 2600 Ice Hockey Duel noop Score 0.5 # 10
Atari Games Atari 2600 Ice Hockey Prior+Duel noop Score -0.4 # 14
Atari Games Atari 2600 Ice Hockey DDQN (tuned) noop Score -2.7 # 22
Atari Games Atari 2600 Ice Hockey Duel hs Score -1.3 # 16
Atari Games Atari 2600 James Bond Prior+Duel noop Score 812.0 # 16
Atari Games Atari 2600 James Bond Duel noop Score 1312.5 # 13
Atari Games Atari 2600 James Bond DDQN (tuned) noop Score 1358.0 # 12
Atari Games Atari 2600 James Bond Duel hs Score 835.5 # 15
Atari Games Atari 2600 Kangaroo DDQN (tuned) noop Score 12992.0 # 10
Atari Games Atari 2600 Kangaroo Duel noop Score 14854.0 # 7
Atari Games Atari 2600 Kangaroo Duel hs Score 10334.0 # 17
Atari Games Atari 2600 Kangaroo Prior+Duel noop Score 1792.0 # 24
Atari Games Atari 2600 Krull Duel noop Score 11451.9 # 5
Atari Games Atari 2600 Krull Prior+Duel noop Score 10374.4 # 10
Atari Games Atari 2600 Krull DDQN (tuned) noop Score 7920.5 # 23
Atari Games Atari 2600 Krull Duel hs Score 8051.6 # 22
Atari Games Atari 2600 Kung-Fu Master Duel noop Score 34294.0 # 20
Atari Games Atari 2600 Kung-Fu Master Prior+Duel noop Score 48375.0 # 9
Atari Games Atari 2600 Kung-Fu Master Duel hs Score 24288.0 # 29
Atari Games Atari 2600 Kung-Fu Master