Google DeepMind’s SIMA is an AI agent that can play 3D video games

Google DeepMind unveiled a unique artificial intelligence (AI) model on Wednesday that can play 3D video games just like a human. The AI ​​model is called a Scalable Instructable Multiworld Agent, or SIMA, and it learns how to interact with different game environments and perform different tasks. The model is currently under research and is still being trained to build expertise in performing more complex movements. Google says that once perfected, the AI ​​model could have a wide range of use cases both online and in the real world.

IN blog post, DeepMind’s SIMA team explained that the AI ​​model is not being developed to be a super-intelligent gamer that can beat every game. Instead, the goal is to teach it how to navigate a 3D game in open world situations and understand how a human would react and interact with it using natural language instructions. Google emphasizes that this is a significantly difficult task and needs to be focused.

“This is an important goal for AI in general, because while large language models have led to powerful systems that can capture knowledge about the world and generate plans, they currently lack the ability to take action on our behalf,” said the team at SIMA.

Tasks performed by AI model SIMA
Photo: Google DeepMind

To create a training environment for the AI ​​model, Google DeepMind partnered with eight game studios and trained SIMA on nine different video games. Some of them include No Man’s Sky by Hello Games, Teardown by Tuxedo Labs, Goat Simulator 3 and Valheim by Coffee Stain Studios and more. The AI ​​model was exposed to new interactive worlds in each game and had to learn to navigate the world, interact with objects, use the menu and much more.

The company also created four research environments, including a new environment built with Unity called the Construction Lab. In this particular environment, the AI ​​model was tasked with building sculptures from building blocks to test object manipulation and intuitive understanding of the physical world.

Google highlighted that the current version of SIMA has been evaluated in 600 basic skills, including navigation such as turning left and driving a car, interacting with objects such as climbing a ladder and making a helmet, and more. The tasks were largely simple and could be completed within 10 seconds.

According to the company, training an AI model on a diverse set of 3D video games, where it follows human instructions and acts accordingly, could have a big impact in the future. The company is now focusing on SIMA training with more complex instructions “that require high-level strategic planning and multiple subtasks to complete, such as finding resources and building a camp.” Google hints that these tasks can translate to the real world, and the company is working on finding ways it can be useful to people.