This is a study notes while reading recent World Model literature. [1]

Goal: Understand what a world model is, how it differs from policy learning, and how modern LLM-based world models are used in decision making.

1. Reinforcement Learning Quick Recap

A reinforcement learning (RL) problem is usually formulated as a Markov Decision Process (MDP):

\[(S,A,P,R,\gamma)\]

where

Symbol	Meaning
$S$	State space
$A$	Action space
$P(s’ \mid s,a)$	Transition dynamics
$R(s,a)$	Reward function
$\gamma$	Discount factor

The objective of RL is

\[\pi^* = \arg\max_\pi \mathbb E \left[ \sum_{t=0}^{\infty} \gamma^t r_t \right]\]

That is,

learn a policy that maximizes long-term reward.

2. What is a World Model?

A world model is a model of the environment.

Rather than asking

What action should I take?

it asks

What will happen if I take this action?

Mathematically,

\[P(s'|s,a)\]

or equivalently

\[M(s,a)=s'\]

It predicts the next state given

current state
current action

Example

Current state

Robot at (2,3)
Facing North

Action

Move Forward

World model predicts

Robot at (2,4)

So the world model learns

(state, action)
        ↓
    next state

3. Transition Dynamics

The transition dynamics describe how the environment evolves.

It is the mapping

\[(s,a)\rightarrow s'\]

\[P(s'|s,a)\]

Examples:

robot movement
autonomous driving
game physics
language interaction
dialogue systems

The transition dynamics are essentially the “laws of physics” of the environment.

4. World Model Learning

In Model-Based RL, learning the world model is usually a supervised learning problem.

Dataset:

(s,a,s')

Loss:

Deterministic model

\[\min_\theta \mathbb E \left[ \|s'-M_\theta(s,a)\|^2 \right]\]

Probabilistic model

\[\min_\theta D_{KL} \left( P^* \| P_\theta \right)\]

The goal is simply:

Predict future states accurately.

5. Policy

Policy answers a completely different question.

Instead of

What will happen?

it answers

What should I do?

A policy is

\[\pi(a|s)\]

Input:

State

Output

Action

Example

Obstacle ahead
        ↓
Jump

6. World Model vs Policy

This is probably the most important distinction.

World Model	Policy
Learn the environment	Learn decision making
Predict future	Choose action
$(s,a)→s’$	$s→a$
Supervised learning	RL / Policy optimization

Another way to think about it:

World Model

“What will happen?”

Policy

“What should I do?”

7. Policy Optimization

Policy optimization does not learn the environment. Instead, it learns a policy $\pi$, where $\pi(a

s)$ denotes the probability of taking action $a$ in state $s$.

The goal is to maximize the policy objective:

\[J(\pi) = \mathbb E \left[ \sum_t \gamma^t r_t \right],\]

where $r_t$ is the reward at time step $t$, and $\gamma \in [0,1]$ is the discount factor, which gives lower weight to rewards in the distant future. Thus, $J(\pi)$ represents the expected long-term reward obtained by following policy $\pi$.

Therefore, the optimization objective is

\[\max_\pi J(\pi),\]

or equivalently,

\[\pi^*=\arg\max_{\pi}J(\pi).\]

In words, policy optimization searches for the policy that achieves the highest expected cumulative reward.

8. Model-Free vs Model-Based RL

Model-Free

Policy
    ↓
Real Environment
    ↓
Reward
    ↓
Update Policy

The agent learns entirely from interaction.

Model-Based

Policy
      ↓
World Model
      ↓
Simulate Future
      ↓
Update Policy

Instead of interacting with the real world every time, the agent first learns a simulator.

9. What Happens After Learning a World Model?

Once the world model is learned,

the next question becomes

How do we use it to make decisions?

This is known as

Policy Generation or Planning.

Two classical approaches are

MPC
MCTS

10. Model Predictive Control (MPC)

MPC searches for an optimal action sequence

\[a_t,\ldots,a_{t+\tau}\]

that maximizes future cumulative reward

\[\max \mathbb E \left[ \sum_{t'=t}^{t+\tau} r(s_{t'},a_{t'}) \right]\]

The planning horizon

\[\tau\]

controls how far into the future we look.

Workflow

Current state

↓

Generate many candidate action sequences

↓

Roll out each sequence inside the world model

↓

Compute cumulative reward

↓

Choose the best sequence

↓

Execute only the first action

↓

Observe new state

↓

Repeat

MPC is therefore

Trajectory Optimization

11. Monte Carlo Tree Search (MCTS)

Instead of optimizing an entire trajectory,

MCTS builds a search tree.

State
├── Action A
│     ├── ...
│     └── ...
├── Action B
└── Action C

Each edge stores

visit count
value estimate

Value

The value of an action is

\[Q(s,a) = \mathbb E \left[ \sum \gamma^t r_t \right]\]

It represents

the expected future cumulative reward after taking action $a$ at state $s$.

It is not the immediate reward.

Statistics Stored

Each edge keeps

\[N(s,a)\]

Visit count

and

\[Q(s,a)\]

Average return estimate.

UCT Rule

The next branch is selected by

\[Q(s,a) + c \sqrt{ \frac{\log N(s)} {N(s,a)} }\]

Interpretation:

First term

Exploit the currently good action.

Second term

Explore actions that have not been tried enough.

This balances

exploitation
exploration

12. MPC vs MCTS

Although both imagine the future,

they search differently.

MPC

Optimizes trajectories.

Trajectory A

Trajectory B

Trajectory C

Compare cumulative rewards.

MCTS

Optimizes a search tree.

State
├── Action
│    ├── State
│    └── State
└── Action

Search resources are concentrated on promising branches.

MPC	MCTS
Trajectory optimization	Tree search
Often continuous control	Often discrete decision making
Replan every timestep	Incrementally expand tree

13. LLM-based World Models

Recent work explores using

LLMs
MLLMs

as world models.

The key observation is:

Language itself is a powerful representation of the world.

14. Two Ways to Use LLM World Models

(1) Direct Action Generation

The simplest pipeline is

Observation
      ↓
LLM
      ↓
Action

The LLM

understands the environment
reasons
plans
outputs actions

all by itself.

Advantages

simple
easy to deploy

Disadvantages

Decision quality depends entirely on

the reasoning capability of the LLM.

15. Modular Usage of LLM World Models

Instead of letting the LLM make the final decision,

the LLM becomes one module inside a larger planning system.

Pipeline

Observation
        ↓
LLM
        ↓
World knowledge
Candidate plans
Environment abstraction
        ↓
Planner / MCTS / Verifier / Simulator
        ↓
Final Action

This is the key idea of

Modular Usage.

Why?

Because LLMs can hallucinate.

Instead of trusting them completely,

we combine them with

symbolic planners
search algorithms
simulators
external verifiers

The LLM provides

knowledge and reasoning

The external planner provides

correctness and search.

16. Examples

GPT + PDDL

LLM generates planning rules

↓

Planner searches for valid plans.

LLM + VirtualHome + MCTS

LLM

↓

Provides commonsense

↓

VirtualHome simulates actions

↓

MCTS searches

↓

Execute best plan

Dynalang

Learn a multimodal world model

↓

Imagine future rollouts

↓

Train policy from imagined trajectories

17. Overall Framework

                Reinforcement Learning
                          │
          ┌───────────────┴───────────────┐
          │                               │
   Model-Free RL                  Model-Based RL
          │                               │
   Learn Policy                 Learn World Model
                                          │
                              World Model Learning
                              (s,a)→s'
                                          │
                          ┌───────────────┴───────────────┐
                          │                               │
                     Planning                    Policy Optimization
                  MPC / MCTS                  PPO / Actor-Critic
                          │
                          ▼
                      Generate Policy
                          │
                          ▼
                        Action

18. Key Takeaways

1. World models vs. Policies.

World model learning and policy optimization are different learning problems. A world model predicts

what will happen.

A policy predicts

what should be done.

And they have different learning objects. World model learning

learns the environment.

Policy optimization

learns decision making.

3. Planning connects the two.

Once a world model is learned,

planning algorithms (MPC, MCTS, etc.) use it to generate actions.

4. Modern LLM world models are increasingly modular.

Rather than replacing classical planning,

LLMs are becoming

reasoning modules
knowledge modules
world abstraction modules

that work together with

search
planning
simulation
verification

to improve decision quality.

Summary

A useful mental model for reading world model papers is to always ask three questions:

What is the world model learning?
- State dynamics?
- Images?
- Language?
- Latent representations?
- Multimodal representations?
How is the world model learned?
- Supervised learning?
- Self-supervised learning?
- Generative modeling?
- Contrastive learning?
How is the world model used?
- MPC?
- MCTS?
- Actor-Critic?
- Planning?
- LLM reasoning?
- Multi-agent systems?

Almost every modern world model paper can be understood through these three questions.

References

[1] J. Ding et al., “Understanding World or Predicting Future? A Comprehensive Survey of World Models,” ACM Comput. Surv., vol. 58, no. 3, p. 57:1-57:38, Sep. 2025, doi: 10.1145/3746449.