DeepMind AI learns physics by watching videos that don’t make sense

An algorithm created by artificial intelligence firm DeepMind can distinguish between videos in which objects obey the laws of physics and those in which they don’t

from Matthew Sparks

Training artificial intelligence to understand simple physical concepts, such as that one solid object cannot occupy the same space as another, could lead to more capable software that takes less computing resources to train, DeepMind researchers say.

The UK-based company has previously created an AI that can beat expert chess players, write computer software and solve the problem of protein folding. But these models are narrowly specialized and lack a general understanding of the world. As the DeepMind researchers say in their latest paper, “something fundamental is still missing.”

now, Louis the Pilot at DeepMind and colleagues have created an AI called Physics Learning through Auto-encoding and Tracking Objects (PLATO) that is designed to understand that the physical world is made up of objects that follow basic physical laws.

The researchers trained PLATO to identify objects and their interactions using simulated videos of objects moving as we would expect, such as balls falling to the ground, rolling behind each other and bouncing off each other. They also provided PLATO data showing exactly which pixels in each frame belonged to each object.

To test PLATO’s ability to understand five physical concepts such as permanence (that an object tends not to disappear), solidity and immutability (that an object tends to retain characteristics such as shape and color), the researchers used another series of simulated videos . Some showed objects that obeyed the laws of physics, while others depicted nonsensical actions, such as a ball rolling behind a post that didn’t come out the other side, then reappearing behind another post further along its route.

They tasked PLATO with predicting what would happen next in each video and found that its predictions were reliably wrong for nonsensical videos but generally correct for logical ones, suggesting that the AI ​​has an intuitive knowledge of physics.

Piloto says the results show that an object-centric view of the world can give AI a more general and adaptable set of abilities. “If you consider, for example, all the different scenes an apple can be in,” he says. “You don’t have to learn about an apple on a tree, versus an apple in your kitchen, versus an apple in the trash. When you isolate an apple as its own thing, you’re in a better position to generalize how it behaves in new systems, in new contexts. Ensures learning effectiveness.”

Mark Nixon at the University of Southampton, UK, says the work could lead to new avenues of AI research and may even reveal clues about human vision and development. But he raised concerns about reproducibility, as the paper says “our implementation of PLATO is not externally viable.”

“That means they’re using an architecture that other people probably can’t use,” he says. “In science, it’s good to be reproducible so that other people can get the same results and then take them further.”

Chen Feng from New York University says the findings could help reduce the computational demands of training and working with AI models.

“It’s like teaching a child what a car is by first teaching them what wheels and seats are,” he says. “The benefit of using an object-centric representation, rather than raw visual inputs, enables AI to learn intuitive physical concepts with better data efficiency.”

Journal reference: Nature Human Behavior, DOI: 10.1038/s41562-022-01394-8