Reinforcement Learning:

Classical Foundations and the LLM era

Flatiron Institute

 

Carol(ina) Cuesta-Lazaro

World Generator

World Solver


def simulate(
	pos1, 
    pos2, 
    duration, 
    **params,
  ):
	"Simulate Universe"

    return trajectories

Convergence,

Re-implementation tests....


def discovered_law(pos1, pos2, p1, p2, velocity2, duration, **params):
    """Particle 2 is accelerated toward particle 1 by a radial force 
    per unit along r^ that combines a static source term G*p1,
    a radial-velocity term (analogous to an advective/retardation
    coupling), and a centripetal-like tangential kinetic-energy 
    term, all scaled by 1/r and independent of p2."""
    ...
    return trajectories

def fit_parameters():
    return {
        "G":     {"init": 1.0, "bounds": [0.1, 5.0]},
        ...
    }

Running Experiment...

Reward

Solve the task

Hard, but Solvable

(Solver)

(Generator)

"This world consists of ..."

World Definition

Simulation Code

Learning to do Physics with Asymmetric Self Play

Attempting the task

Proposing a new task

TD-GAMMON

1992

2013

DQN

2016

AlphaGo

AlphaGo

ChatGPT

(RLHF)

2022

Reasoning (RLVR)

2025

Hide And Seek

2019

Agent

Environment

State

Action

s_t
s_{t+1}
a_t \sim \Pi_\theta(s_t)

Reward

R_t

Policy

The Learning Problem:

\frac{\partial J}{\partial \theta} = \frac{\partial J}{\partial s_T} \cdot \frac{\partial s_T}{\partial a_{T-1}} \cdot \frac{\partial a_{T-1}}{\partial s_{T-1}} \cdots \frac{\partial a_0}{\partial \theta}

State

s_{t-1}

Policy Network

Environment

Reward

s_{t}

Not Differentiable!

a_t \sim \Pi_\theta(s_{t-1})
J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ R(\tau) \right]

RL vs Supervised Learning

  • In RL the data distribution is changing
  • Reward signal is evaluative, not instructive 
  • Very stochastic outcomes
  • Trade off between exploration and exploitation

The agent generates its own training data by acting

Should I do what's worked before, or try something new?

Tells you how good, not what was right

  • Delayed Feedback

The consequence of an action may only show up many steps later

Reinforcement Learning from Human Feedback (RLHF)

1) Pretraining:

fill in the gaps

p(\mathrm{behaviour})

Learning a prior

p(\mathrm{behaviour}|\mathrm{success})

2) Postraining:

Reinforcement Learning from Verifiable Rewards

Updating the prior to conform with evidence provided by the reward

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026

Post-training for LMs

\mathbb{E}_{x \sim \Pi_\theta}\left[ R(x) \right] - \beta \mathrm{KL}(\Pi_\theta, \Pi_0)

Evidence: Reward Model

\exp \left( r(x) \right)

Posterior: RL-ed model

\Pi^\star (x) \propto \Pi_0(x) \exp \left( r(x) \right)

p(behaviour|correct)

Maximise reward

Stay close to original LM

\mathrm{KL}(\Pi_\theta, \Pi^\star)

Variational Inference:

\Pi_0 (x)

p(behaviour)

Prior: Original Language Model

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026

Mathematics (and coding) are hard to solve but easy to verify

Navier Stokes: No human understood the proof before we knew it was correct

DeepSeek R1: RL with Verifiable Rewards

x, y > 1, \qquad \log_x\left(y^x\right) = \log_y\left(x^{4y}\right) = 10. \qquad \text{Find } xy.
xy = 25

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026

DeepSeek R1: RL with Verifiable Rewards

Accuracy on Maths Problems

# RL steps

Response Length

# RL steps

Carolina Cuesta-Lazaro Flatiron/NYU @ CCA 2026

Increase in thinking time

advanced reasoning: reflective + systematic exploration of solutions

https://z.ai/blog/glm-5.2

GLM 5.2

https://arxiv.org/pdf/2504.13837

Are models learning something new via RL?

https://arxiv.org/pdf/2510.14901

Are high likelihood responses more correct?

Resources

Tutorial: Lunar Lander

0: Do nothing
1: fire left engine
2: fire main engine
3: fire right engine

Actions

State

(x, y, vx, vy, angle, angular_velocity, left_leg_contact, right_leg_contact)

RL-Lectures-CCA-2026

By carol cuesta

RL-Lectures-CCA-2026

  • 102