Here’s how to sound smart next time someone says "world models"

In the previous installment, we explored why world models are necessary. In this installment, we’ll examine the core architecture of World Models, the 2018 paper considered the origin of world models.

Understanding the World in a Nutshell

No one calculates their speed, the coefficient of friction with the road, or air resistance when riding a bike. Just thinking, “Hehe, let’s go for a bike ride,” is enough.

People don’t calculate the entire world either. They simply quickly grasp the information they need at the moment and act on it. The same is true for the world model. It condenses only the key information from reality into a small framework and uses that to predict what will happen next.

Vision · Memory · Controller

World Models divides this compression and prediction into three parts.

1. Vision — How can I make this scene shorter?
2. Memory — If I move like this, what will happen next?
3. Controller — So, what should we do now?

Vision compresses the scene, and Memory predicts the next state. The Controller acts based on the current state and memory.

Step 1 — Compress the scene

Instead of using the entire screen as is, Vision compresses the scene into a small data set z. Just as you can still make out the general shape of an object even when you take off your glasses, it creates a “summary” of the screen by reducing the details and retaining only the key features.

Step 2 — Predict the next scene

Memory observes the current state and behavior and predicts how the next state will change. This is the stage of learning about changes over time.

It’s the same principle as when only a mosquito’s body and part of its wings are drawn—you can still predict how the next line will continue in various ways.

Step 3 — Now, choose an action

The Controller determines what action to take next based on the current scene as seen by Vision and the sequence of events stored in Memory.

For example, when a car approaches a curve, the controller decides which way to turn the steering wheel and how to adjust the speed.

How far have we come with this structure?

• Google DeepMind is focusing its research on Genie, a project to create a virtual world where users can move around and explore on their own.
• Meta is conducting research using JEPA to predict meaning and movement that go beyond pixels, enabling AI to understand reality and plan its actions.
• NVIDIA is conducting research using Cosmos to enable robots and autonomous driving AI to understand real-world movements and learn in advance through virtual environments.

All three companies are finding their own solutions based on the same framework—compress, predict, and act.