We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Atari Games Atari 2600 Alien Gorila Score 813.5 # 44
Atari Games Atari 2600 Amidar Gorila Score 189.2 # 38
Atari Games Atari 2600 Assault Gorila Score 1195.8 # 40
Atari Games Atari 2600 Asterix Gorila Score 3324.7 # 42
Atari Games Atari 2600 Asteroids Gorila Score 933.6 # 40
Atari Games Atari 2600 Atlantis Gorila Score 629166.5 # 24
Atari Games Atari 2600 Bank Heist Gorila Score 399.4 # 37
Atari Games Atari 2600 Battle Zone Gorila Score 19938.0 # 38
Atari Games Atari 2600 Beam Rider Gorila Score 3822.1 # 42
Atari Games Atari 2600 Bowling Gorila Score 54 # 28
Atari Games Atari 2600 Boxing Gorila Score 74.2 # 33
Atari Games Atari 2600 Breakout Gorila Score 313.0 # 43
Atari Games Atari 2600 Centipede Gorila Score 6296.9 # 27
Atari Games Atari 2600 Chopper Command Gorila Score 3191.8 # 40
Atari Games Atari 2600 Crazy Climber Gorila Score 65451.0 # 40
Atari Games Atari 2600 Demon Attack Gorila Score 14880.1 # 36
Atari Games Atari 2600 Double Dunk Gorila Score -11.3 # 26
Atari Games Atari 2600 Enduro Gorila Score 71.0 # 42
Atari Games Atari 2600 Fishing Derby Gorila Score 4.6 # 34
Atari Games Atari 2600 Freeway Gorila Score 10.2 # 50
Atari Games Atari 2600 Frostbite Gorila Score 426.6 # 40
Atari Games Atari 2600 Gopher Gorila Score 4373.0 # 38
Atari Games Atari 2600 Gravitar Gorila Score 538.4 # 30
Atari Games Atari 2600 HERO Gorila Score 8963.4 # 39
Atari Games Atari 2600 Ice Hockey Gorila Score -1.7 # 20
Atari Games Atari 2600 James Bond Gorila Score 444.0 # 37
Atari Games Atari 2600 Kangaroo Gorila Score 1431.0 # 37
Atari Games Atari 2600 Krull Gorila Score 6363.1 # 36
Atari Games Atari 2600 Kung-Fu Master Gorila Score 20620.0 # 40
Atari Games Atari 2600 Montezuma's Revenge Gorila Score 84 # 27
Atari Games Atari 2600 Ms. Pacman Gorila Score 1263.0 # 38
Atari Games Atari 2600 Name This Game Gorila Score 9238.5 # 32
Atari Games Atari 2600 Pong Gorila Score 16.7 # 44
Atari Games Atari 2600 Private Eye Gorila Score 2598.6 # 17
Atari Games Atari 2600 Q*Bert Gorila Score 7089.8 # 42
Atari Games Atari 2600 River Raid Gorila Score 5310.3 # 34
Atari Games Atari 2600 Road Runner Gorila Score 43079.8 # 31
Atari Games Atari 2600 Robotank Gorila Score 61.8 # 23
Atari Games Atari 2600 Seaquest Gorila Score 10145.9 # 22
Atari Games Atari 2600 Space Invaders Gorila Score 1183.3 # 42
Atari Games Atari 2600 Star Gunner Gorila Score 14919.2 # 36
Atari Games Atari 2600 Tennis Gorila Score -0.7 # 29
Atari Games Atari 2600 Time Pilot Gorila Score 8267.8 # 29
Atari Games Atari 2600 Tutankham Gorila Score 118.5 # 34
Atari Games Atari 2600 Up and Down Gorila Score 8747.7 # 38
Atari Games Atari 2600 Venture Gorila Score 523.4 # 21
Atari Games Atari 2600 Video Pinball Gorila Score 112093.4 # 31
Atari Games Atari 2600 Wizard of Wor Gorila Score 10431.0 # 19
Atari Games Atari 2600 Zaxxon Gorila Score 6159.4 # 34

Methods