Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games - the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled - our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Atari Games 100k Atari 100k MuZero Mean Human-Normalized Score 0.562 # 11
Medium Human-Normalized Score 0.227 # 13
Atari Games Atari 2600 Alien MuZero Score 741812.63 # 1
Atari Games Atari 2600 Amidar MuZero Score 28634.39 # 3
Atari Games Atari 2600 Assault MuZero Score 143972.03 # 1
Atari Games Atari 2600 Asterix MuZero Score 998425.00 # 3
Atari Games Atari 2600 Asteroids MuZero Score 678558.64 # 3
Atari Games Atari 2600 Atlantis MuZero Score 1674767.20 # 5
Atari Games Atari 2600 Bank Heist MuZero Score 1278.98 # 13
Atari Games Atari 2600 Battle Zone MuZero Score 848623.00 # 2
Atari Games Atari 2600 Beam Rider MuZero Score 454993.53 # 1
Atari Games Atari 2600 Berzerk MuZero Score 85932.60 # 2
Atari Games Atari 2600 Bowling MuZero Score 260.13 # 1
Atari Games Atari 2600 Boxing MuZero Score 100.00 # 1
Atari Games Atari 2600 Centipede MuZero Score 1159049.27 # 3
Atari Games Atari 2600 Chopper Command MuZero Score 991039.70 # 5
Atari Games Atari 2600 Crazy Climber MuZero Score 458315.40 # 2
Atari Games Atari 2600 Defender MuZero Score 839642.95 # 5
Atari Games Atari 2600 Demon Attack MuZero Score 143964.26 # 5
Atari Games Atari 2600 Double Dunk MuZero Score 23.94 # 5
Atari Games Atari 2600 Enduro MuZero Score 2382.44 # 5
Atari Games Atari 2600 Fishing Derby MuZero Score 91.16 # 1
Atari Games Atari 2600 Freeway MuZero Score 33.03 # 18
Atari Games Atari 2600 Frostbite MuZero Score 631378.53 # 1
Atari Games Atari 2600 Gopher MuZero Score 130345.58 # 3
Atari Games Atari 2600 Gravitar MuZero Score 6682.70 # 6
Atari Games Atari 2600 HERO MuZero Score 49244.11 # 2
Atari Games Atari 2600 Ice Hockey MuZero Score 67.04 # 3
Atari Games Atari 2600 James Bond MuZero Score 41063.25 # 6
Atari Games Atari 2600 Kangaroo MuZero Score 16763.60 # 2
Atari Games Atari 2600 Krull MuZero Score 269358.27 # 2
Atari Games Atari 2600 Kung-Fu Master MuZero Score 204824.00 # 5
Atari Games Atari 2600 Montezuma's Revenge MuZero Score 0.00 # 42
Atari Games Atari 2600 Ms. Pacman MuZero Score 243401.10 # 1
Atari Games Atari 2600 Name This Game MuZero Score 157177.85 # 1
Atari Games Atari 2600 Phoenix MuZero Score 955137.84 # 2
Atari Games Atari 2600 Pitfall! MuZero Score 0.00 # 4
Atari Games Atari 2600 Pong MuZero Score 21.00 # 1
Atari Games Atari 2600 Private Eye MuZero Score 15299.98 # 4
Atari Games Atari 2600 Q*Bert MuZero Score 72276.00 # 9
Atari Games Atari 2600 River Raid MuZero Score 323417.18 # 1
Atari Games Atari 2600 Road Runner MuZero Score 613411.80 # 4
Atari Games Atari 2600 Robotank MuZero Score 131.13 # 1
Atari Games Atari 2600 Seaquest MuZero Score 999976.52 # 5
Atari Games Atari 2600 Skiing MuZero Score -29968.36 # 3
Atari Games Atari 2600 Solaris MuZero Score 56.62 # 23
Atari Games Atari 2600 Space Invaders MuZero Score 74335.30 # 5
Atari Games Atari 2600 Star Gunner MuZero Score 549271.70 # 4
Atari Games Atari 2600 Surround MuZero Score 9.99 # 2
Atari Games Atari 2600 Tennis MuZero Score 0.00 # 20
Atari Games Atari 2600 Time Pilot MuZero Score 476763.90 # 1
Atari Games Atari 2600 Tutankham MuZero Score 491.48 # 2
Atari Games Atari 2600 Up and Down MuZero Score 715545.61 # 4
Atari Games Atari 2600 Venture MuZero Score 0.40 # 48
Atari Games Atari 2600 Video Pinball MuZero Score 981791.88 # 3
Atari Games Atari 2600 Wizard of Wor MuZero Score 197126.00 # 1
Atari Games Atari 2600 Yars Revenge MuZero Score 553311.46 # 6
Atari Games Atari 2600 Zaxxon MuZero Score 725853.90 # 1
Atari Games Atari-57 MuZero Human World Record Breakthrough 19 # 4
Mean Human Normalized Score 4996.20% # 5
Atari Games atari game Muzero Human World Record Breakthrough 19 # 2
Atari Games Atari games MuZero Mean Human Normalized Score 4996.20% # 3

Methods