Scientific Foundation Models

Carol(ina) Cuesta-Lazaro

Representations, Capabilities and the Search for New Physics

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Am I a Foundation Model?

Regression of Physical Parameters

Retrieval of rare objects

Outlier/Anomaly detection

Follow up priorities

Predict missing modalities

Super Resolution

Physics

Systematics

Learning What's Real

DiscoveryPhysics Benchmark

Disentangling Physics and Instruments in Foundation Models

Can LLMs discover new physics?

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Physics

Systematics

[arXiv:2503.15312]

Euclid Quick Data Release (Q1)

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Pablo Mercader

Daniel Muthukrishna

Jeroen Audenaert

Legacy Survey

HSC

DESI

SDSS

Same Object / Different Instrument

Different Object / Same Instrument

Can we disentangle Systematics from Physics?

[arXiv:2604.09787]

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Object 1

Object 2

Object 1

z_\mathrm{instrument}

Orientation + Scale

Number

z_\mathrm{instrument},

z_\mathrm{object}

)

Instrument 1

Instrument 2

Instrument Encoder

z_\mathrm{object}

Object Encoder

Instrument Pair

Object Pair

Instrument Pair

Object Pair

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Original

Donor

z^O_\mathrm{physics}

z^O_\mathrm{ins}

z^D_\mathrm{physics}

z^D_\mathrm{ins}

p \left( z^O_\mathrm{physics}, z^O_\mathrm{ins} \right)

p \left( z^D_\mathrm{physics}, z^O_\mathrm{ins} \right)

p \left( z^O_\mathrm{physics}, z^D_\mathrm{ins} \right)

z^D_\mathrm{physics}

z^O_\mathrm{physics}

z^O_\mathrm{ins}

z^D_\mathrm{ins}

z^D_\mathrm{physics}

Reconstruction

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Finding Outliers

(no disentanglement)

arXiv2510.17960

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Similarity Search

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

A Data-driven Noise Model

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

2. What is the role of simulations in foundation models? From outlier detection to anomaly detection

#TODOs

1. How do we include information that is private to the instrument

3. Going to space! JWST x Euclid

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

BEFORE

Artificial General Intelligence?

AFTER

DiscoverPhysics: Benchmarking LLMs for
Out-of-the-Box Scientific Thinking

Matt Wiemann

Lindsay Smith

Hypothesis

Simulate World

Invisible particles

Extra dimensions

Multi Species ...

Simulate World

Invisible particles

Extra dimensions

Multi Species ...

Propose Experiment

Simulate

Text: Conceptual Understanding

Trajectories (.csv)

Science Agent (LLM)

Science Agent (LLM)

Outputs

Python Code: Trajectory MSE

["DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking" Wiemann, Smith et al (including CCL)]

Matt Wiemann

Lindsay Smith


def discovered_law(pos1, pos2, p1, p2, velocity2, duration, **params):
    """Particle 2 is accelerated toward particle 1 by a radial force 
    per unit along r^ that combines a static source term G*p1,
    a radial-velocity term (analogous to an advective/retardation
    coupling), and a centripetal-like tangential kinetic-energy 
    term, all scaled by 1/r and independent of p2."""
    ...
    return trajectories

def fit_parameters():
    return {
        "G":     {"init": 1.0, "bounds": [0.1, 5.0]},
        ...
    }

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Predictive Code

Rubric Based LLM judge

Mean Square Error

Agent Explanation

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

How Predictive the Theory is

Conceptual Understanding

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Adversarial Experiments to test Generalization

You are an expert physicist tasked with discovering scientific laws in a simulated universe. Your goal is to propose experiments, analyse the data they return, and ultimately deduce the underlying scientific law. Note that the laws of physics in this universe may differ from those in our own. You can perform experiments to gather data but must follow the protocol strictly.

If F = G/r²: G = 80. If F = G/r: G = 8. Linear in 1/r!

m=1: a = 0.664 · m=2: a = 0.614 · m=4: a = 0.664 · m=8: a = 0.714 … That's strange."

Consistent pattern: small +y 'extra' force (~0.05) regardless of position … maybe the anchor itself moves.

a_y_extra ≈ +0.05 everywhere on probes. Spot on! So anchor too just experiences g_y = 0.05 … g = (0, 0.05) is a uniform external acceleration on every particle.

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Confirmed: a ∝ 1/r.

observed −5.18. WAY too small. Something is very wrong with my understanding!

artifact of integration accuracy, NOT the true law. Probably because of close encounters causing chaos.

I'll just submit my law. The functional form is solid.

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

a × r seems roughly constant … not quite 1/r.

Let me try a × r²: … Not consistent.

So at large r, a ~ 1/r. At small r, a ~ 1/r².

Actually note the file path in the error message: 'extra_dimensions.csv'! This is a hint! The world might have an extra compactified dimension…

[Model starts fitting Yukawa potential, Bessel functions... At some point fitting tool errors.]

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Conceptual Understanding

Trajectory Prediction

Random Seeds

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

World Generator

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

World Solver


def simulate(
	pos1, 
    pos2, 
    duration, 
    **params,
  ):
	"Simulate Universe"

    return trajectories

Convergence,

Re-implementation tests....


def discovered_law(pos1, pos2, p1, p2, velocity2, duration, **params):
    """Particle 2 is accelerated toward particle 1 by a radial force 
    per unit along r^ that combines a static source term G*p1,
    a radial-velocity term (analogous to an advective/retardation
    coupling), and a centripetal-like tangential kinetic-energy 
    term, all scaled by 1/r and independent of p2."""
    ...
    return trajectories

def fit_parameters():
    return {
        "G":     {"init": 1.0, "bounds": [0.1, 5.0]},
        ...
    }

Running Experiment...

Reward

Predictiveness

Conceptual Understanding

(MSE)

(Evaluation Score)

"This world consists of ..."

World Definition

Simulation Code

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Easy

Medium

Hard

The challenge: Long roll outs / Attribution

["An LLM-driven framework for cosmological
model-building and exploration" Mudur, Cuesta-Lazaro, Toomey ]

Can LLMs help us explore the space of hypothesis?

Propose a model for Dark Energy

Implement it in a Cosmology simulation code: CLASS

Test fit to DESI Observations

Iterate to improve fit

Quintessence, DE/DM interactions....

Must pass a set of general tests for "reasonable" models

Ideally, compare evidence to LCDM.

For now, Bayesian Information Criteria (BIC)

Nayantara Mudur (Harvard)

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

\Lambda \mathrm{CDM}

["DESI 2024 VI: Cosmological Constraints from the Measurements of Baryon Acoustic Oscillations" arXiv:2404.03002]

Dark Energy is constant over time

w(z) = \frac{p_\mathrm{DE}}{\rho_\mathrm{DE}} = w_0 + \frac{z}{1+z}w_a

2 - 4 \sigma

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Kai Lehmann

p(x|\mathcal{M}) = \int p(x|\theta,\mathcal{M}) p(\theta|\mathcal{M}) d\theta

Bayesian Model Comparison

Ocham's razor: Penalize overparametrized theories

How well does a theory fit the data?

Conditional Time Score matching (CTSM)

Flow Models

Bayesian Evidence

Density Ratio Estimation (DRE)

Cross Entropy Loss

Flow

Model A

Model B

Model C

p(x|\mathcal{M_A}) / p(x|\mathcal{M_B})

p(x|\mathcal{M_A}) = p^\mathrm{Flow}(x) \frac{p(x|\mathcal{M_A})}{p^\mathrm{Flow}(x)}

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

Accuracy

Dimensionality

Training set size

Observation

Question

Hypothesis

Testable Predictions

Gather data

Alter, Expand, Reject Hypothesis

Develop General Theories

[Figure adapted from ArchonMagnus]

The Scientific Method in > 2025

Carolina Cuesta-Lazaro Flatiron/NYU @ KAAI 2026

p(\mathcal{M}|x)