World Generator
World Solver
def simulate(
pos1,
pos2,
duration,
**params,
):
"Simulate Universe"
return trajectories
Convergence,
Re-implementation tests....
def discovered_law(pos1, pos2, p1, p2, velocity2, duration, **params):
"""Particle 2 is accelerated toward particle 1 by a radial force
per unit along r^ that combines a static source term G*p1,
a radial-velocity term (analogous to an advective/retardation
coupling), and a centripetal-like tangential kinetic-energy
term, all scaled by 1/r and independent of p2."""
...
return trajectories
def fit_parameters():
return {
"G": {"init": 1.0, "bounds": [0.1, 5.0]},
...
}
Running Experiment...
Reward
Solve the task
Hard, but Solvable
(Solver)
(Generator)
"This world consists of ..."
World Definition
Simulation Code
Attempting the task
Proposing a new task
TD-GAMMON
1992
2013
DQN
2016
AlphaGo
AlphaGo
ChatGPT
(RLHF)
2022
Reasoning (RLVR)
2025
Hide And Seek
2019
Agent
Environment
State
Action
Reward
Policy
State
Policy Network
Environment
Reward
Not Differentiable!
The agent generates its own training data by acting
Should I do what's worked before, or try something new?
Tells you how good, not what was right
The consequence of an action may only show up many steps later
1) Pretraining:
fill in the gaps
Learning a prior
2) Postraining:
Reinforcement Learning from Verifiable Rewards
Updating the prior to conform with evidence provided by the reward
Evidence: Reward Model
Posterior: RL-ed model
p(behaviour|correct)
Maximise reward
Stay close to original LM
Variational Inference:
p(behaviour)
Prior: Original Language Model
Mathematics (and coding) are hard to solve but easy to verify
Navier Stokes: No human understood the proof before we knew it was correct
Accuracy on Maths Problems
# RL steps
Response Length
# RL steps
Increase in thinking time
advanced reasoning: reflective + systematic exploration of solutions
https://z.ai/blog/glm-5.2
https://arxiv.org/pdf/2504.13837
https://arxiv.org/pdf/2510.14901
0: Do nothing
1: fire left engine
2: fire main engine
3: fire right engine
Actions
State
(x, y, vx, vy, angle, angular_velocity, left_leg_contact, right_leg_contact)