Reinforcement Learning:
Classical Foundations and the LLM era
Flatiron Institute
Carol(ina) Cuesta-Lazaro

World Generator

World Solver
def simulate(
pos1,
pos2,
duration,
**params,
):
"Simulate Universe"
return trajectories

Convergence,
Re-implementation tests....
def discovered_law(pos1, pos2, p1, p2, velocity2, duration, **params):
"""Particle 2 is accelerated toward particle 1 by a radial force
per unit along r^ that combines a static source term G*p1,
a radial-velocity term (analogous to an advective/retardation
coupling), and a centripetal-like tangential kinetic-energy
term, all scaled by 1/r and independent of p2."""
...
return trajectories
def fit_parameters():
return {
"G": {"init": 1.0, "bounds": [0.1, 5.0]},
...
}

Running Experiment...
Reward
Solve the task
Hard, but Solvable
(Solver)
(Generator)
"This world consists of ..."
World Definition
Simulation Code
Learning to do Physics with Asymmetric Self Play

Attempting the task
Proposing a new task


TD-GAMMON
1992

2013
DQN
2016
AlphaGo
AlphaGo
ChatGPT
(RLHF)
2022
Reasoning (RLVR)
2025
Hide And Seek
2019

Agent


Environment
State


Action

Reward
Policy
The Learning Problem:
State

Policy Network
Environment
Reward
Not Differentiable!
RL vs Supervised Learning
- In RL the data distribution is changing
- Reward signal is evaluative, not instructive
- Very stochastic outcomes
- Trade off between exploration and exploitation
The agent generates its own training data by acting
Should I do what's worked before, or try something new?
Tells you how good, not what was right
- Delayed Feedback
The consequence of an action may only show up many steps later

Reinforcement Learning from Human Feedback (RLHF)

1) Pretraining:
fill in the gaps
Learning a prior
2) Postraining:
Reinforcement Learning from Verifiable Rewards
Updating the prior to conform with evidence provided by the reward
Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026
Post-training for LMs

Evidence: Reward Model

Posterior: RL-ed model
p(behaviour|correct)
Maximise reward
Stay close to original LM
Variational Inference:
p(behaviour)
Prior: Original Language Model

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026
Mathematics (and coding) are hard to solve but easy to verify
Navier Stokes: No human understood the proof before we knew it was correct
DeepSeek R1: RL with Verifiable Rewards

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026

DeepSeek R1: RL with Verifiable Rewards
Accuracy on Maths Problems
# RL steps
Response Length
# RL steps
Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026
Increase in thinking time
advanced reasoning: reflective + systematic exploration of solutions


https://z.ai/blog/glm-5.2
GLM 5.2

https://arxiv.org/pdf/2504.13837
Are models learning something new via RL?
https://arxiv.org/pdf/2510.14901

Are high likelihood responses more correct?




Resources
Tutorial: Lunar Lander

0: Do nothing
1: fire left engine
2: fire main engine
3: fire right engine
Actions
State
(x, y, vx, vy, angle, angular_velocity, left_leg_contact, right_leg_contact)RL-Lectures-CCA-2026
By carol cuesta
RL-Lectures-CCA-2026
- 102