Increasing the Action Gap: New Operators for Reinforcement Learning

15 Dec 2015 • Marc G. Bellemare • Georg Ostrovski • Arthur Guez • Philip S. Thomas • Rémi Munos

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency... (read more)

PDF Abstract
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Atari Games Atari 2600 Alien Advantage Learning Score 4990.91 # 11
Atari Games Atari 2600 Alien Persistent AL Score 5699.81 # 10
Atari Games Atari 2600 Amidar Advantage Learning Score 1557.43 # 13
Atari Games Atari 2600 Amidar Persistent AL Score 1451.65 # 15
Atari Games Atari 2600 Assault Advantage Learning Score 3661.51 # 25
Atari Games Atari 2600 Assault Persistent AL Score 3304.33 # 28
Atari Games Atari 2600 Asterix Persistent AL Score 19564.9 # 20
Atari Games Atari 2600 Asterix Advantage Learning Score 12852.08 # 27
Atari Games Atari 2600 Asteroids Advantage Learning Score 1924.42 # 21
Atari Games Atari 2600 Asteroids Persistent AL Score 1673.52 # 23
Atari Games Atari 2600 Atlantis Persistent AL Score 1465250 # 5
Atari Games Atari 2600 Atlantis Advantage Learning Score 553591.67 # 18
Atari Games Atari 2600 Bank Heist Persistent AL Score 874.99 # 25
Atari Games Atari 2600 Bank Heist Advantage Learning Score 633.63 # 26
Atari Games Atari 2600 Battle Zone Persistent AL Score 34583.07 # 13
Atari Games Atari 2600 Battle Zone Advantage Learning Score 28789.29 # 20
Atari Games Atari 2600 Beam Rider Persistent AL Score 13145.34 # 21
Atari Games Atari 2600 Beam Rider Advantage Learning Score 10054.58 # 24
Atari Games Atari 2600 Berzerk Persistent AL Score 1328.25 # 15
Atari Games Atari 2600 Berzerk Advantage Learning Score 747.26 # 26
Atari Games Atari 2600 Bowling Persistent AL Score 71.59 # 12
Atari Games Atari 2600 Bowling Advantage Learning Score 57.41 # 19
Atari Games Atari 2600 Boxing Persistent AL Score 94.3 # 14
Atari Games Atari 2600 Boxing Advantage Learning Score 93.94 # 15
Atari Games Atari 2600 Breakout Persistent AL Score 431.89 # 16
Atari Games Atari 2600 Breakout Advantage Learning Score 425.32 # 17
Atari Games Atari 2600 Centipede Persistent AL Score 4539.55 # 26
Atari Games Atari 2600 Centipede Advantage Learning Score 4225.18 # 28
Atari Games Atari 2600 Chopper Command Persistent AL Score 5734.93 # 23
Atari Games Atari 2600 Chopper Command Advantage Learning Score 5431.36 # 24
Atari Games Atari 2600 Crazy Climber Persistent AL Score 130002.71 # 17
Atari Games Atari 2600 Crazy Climber Advantage Learning Score 123410.71 # 21
Atari Games Atari 2600 Defender Advantage Learning Score 30643.59 # 14
Atari Games Atari 2600 Defender Persistent AL Score 32038.93 # 13
Atari Games Atari 2600 Demon Attack Persistent AL Score 70908.17 # 17
Atari Games Atari 2600 Demon Attack Advantage Learning Score 27153.48 # 27
Atari Games Atari 2600 Double Dunk Advantage Learning Score -0.15 # 18
Atari Games Atari 2600 Double Dunk Persistent AL Score -2.51 # 22
Atari Games Atari 2600 Elevator Action Persistent AL Score 29100 # 1
Atari Games Atari 2600 Elevator Action Advantage Learning Score 27088.89 # 2
Atari Games Atari 2600 Enduro Advantage Learning Score 1252.7 # 18
Atari Games Atari 2600 Enduro Persistent AL Score 1343.1 # 17
Atari Games Atari 2600 Fishing Derby Persistent AL Score 28.13 # 16
Atari Games Atari 2600 Fishing Derby Advantage Learning Score 21.32 # 19
Atari Games Atari 2600 Freeway Persistent AL Score 32.3 # 10
Atari Games Atari 2600 Freeway Advantage Learning Score 31.72 # 12
Atari Games Atari 2600 Frostbite Advantage Learning Score 2305.82 # 21
Atari Games Atari 2600 Frostbite Persistent AL Score 3248.96 # 17
Atari Games Atari 2600 Gopher Persistent AL Score 10611.81 # 23
Atari Games Atari 2600 Gopher Advantage Learning Score 11912.68 # 22
Atari Games Atari 2600 Gravitar Persistent AL Score 446.92 # 23
Atari Games Atari 2600 Gravitar Advantage Learning Score 417.65 # 26
Atari Games Atari 2600 HERO Persistent AL Score 24175.79 # 15
Atari Games Atari 2600 HERO Advantage Learning Score 24788.86 # 14
Atari Games Atari 2600 Ice Hockey Advantage Learning Score -1.24 # 15
Atari Games Atari 2600 Ice Hockey Persistent AL Score -0.25 # 13
Atari Games Atari 2600 James Bond Advantage Learning Score 848.46 # 14
Atari Games Atari 2600 James Bond Persistent AL Score 772.09 # 17
Atari Games Atari 2600 Kangaroo Advantage Learning Score 10809.16 # 16
Atari Games Atari 2600 Kangaroo Persistent AL Score 11478.46 # 13
Atari Games Atari 2600 Krull Persistent AL Score 8689.81 # 16
Atari Games Atari 2600 Krull Advantage Learning Score 9548.92 # 14
Atari Games Atari 2600 Kung-Fu Master Advantage Learning Score 32182.99 # 22
Atari Games Atari 2600 Kung-Fu Master Persistent AL Score 34650.91 # 17
Atari Games Atari 2600 Montezuma's Revenge Advantage Learning Score 0.42 # 33
Atari Games Atari 2600 Montezuma's Revenge Persistent AL Score 1.72 # 32
Atari Games Atari 2600 Ms. Pacman Persistent AL Score 3917.55 # 15
Atari Games Atari 2600 Ms. Pacman Advantage Learning Score 4065.8 # 13
Atari Games Atari 2600 Name This Game Advantage Learning Score 11025.26 # 19
Atari Games Atari 2600 Name This Game Persistent AL Score 10431.33 # 23
Atari Games Atari 2600 Phoenix Advantage Learning Score 22038.27 # 8
Atari Games Atari 2600 Phoenix Persistent AL Score 14495.56 # 10
Atari Games Atari 2600 Pitfall! Advantage Learning Score 0 # 4
Atari Games Atari 2600 Pong Persistent AL Score 19.76 # 7
Atari Games Atari 2600 Pong Advantage Learning Score 19.66 # 8
Atari Games Atari 2600 Pooyan Advantage Learning Score 4801.27 # 2
Atari Games Atari 2600 Private Eye Advantage Learning Score 5276.16 # 8
Atari Games Atari 2600 Q*Bert Advantage Learning Score 14368.03 # 21
Atari Games Atari 2600 River Raid Advantage Learning Score 10585.12 # 21
Atari Games Atari 2600 Road Runner Advantage Learning Score 52351.23 # 16
Atari Games Atari 2600 Robotank Advantage Learning Score 69.31 # 5
Atari Games Atari 2600 Seaquest Persistent AL Score 13230.74 # 13
Atari Games Atari 2600 Seaquest Advantage Learning Score 8670.5 # 17
Atari Games Atari 2600 Skiing Advantage Learning Score -13264.51 # 10
Atari Games Atari 2600 Solaris Advantage Learning Score 4785.16 # 7
Atari Games Atari 2600 Space Invaders Advantage Learning Score 3460.79 # 17
Atari Games Atari 2600 Space Invaders Persistent AL Score 3277.59 # 18
Atari Games Atari 2600 Star Gunner Advantage Learning Score 61353.59 # 19
Atari Games Atari 2600 Surround Persistent AL Score 0.72 # 8
Atari Games Atari 2600 Tennis Advantage Learning Score 0 # 11
Atari Games Atari 2600 Time Pilot Advantage Learning Score 8969.12 # 16
Atari Games Atari 2600 Tutankham Advantage Learning Score 245.22 # 11
Atari Games Atari 2600 Up and Down Advantage Learning Score 13909.74 # 27
Atari Games Atari 2600 Venture Advantage Learning Score 198.69 # 19
Atari Games Atari 2600 Video Pinball Advantage Learning Score 543504 # 10
Atari Games Atari 2600 Wizard of Wor Advantage Learning Score 9541.14 # 13
Atari Games Atari 2600 Yars Revenge Advantage Learning Score 24240.03 # 10
Atari Games Atari 2600 Zaxxon Advantage Learning Score 9129.61 # 24

Methods used in the Paper


METHOD TYPE
🤖 No Methods Found Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet