Universal scale-free representations in human visual cortex

Brice Ménard

Raj Magesh Gauthaman

Michael F. Bonner

2, 1

Outline

Motivation	Two competing theories of human vision
Framework	Neural population geometry
Tool	Cross-decomposition
Results	Universal power-law covariance spectra
Implications	What I think a "theory of vision" would look like

Vision 101

light enters your eyes

signals travel to visual cortex

many stages of processing (V1 to V4, ...)

perceptual experience

How do we understand the visual system?

Hubel & Wiesel (1968)

monkey V1

single neurons respond to oriented bars of light

How do we understand the visual system?

Kanwisher, McDermott & Yovel (1997)

human fusiform face area (FFA)

Information can also be encoded in neural population activity.

How can we study population codes?

Credit: NordicNeuroLab

fMRI

functional magnetic resonance imaging

resolution

~1 mm

~3 s

coarse measure of neural activity

image seen by participant

stimulus

fMRI BOLD signal

neural response

blood oxygen level dependent

abstract representation

vector of voxel responses

volumetric pixel

voxel 1

voxel 4

voxel N

...

voxel 1

voxel 2

voxel 3

voxel 4

voxel N

...

Representational space

voxel 1

voxel 2

data spans both available dimensions

voxel 1

voxel 2

data lives in a 1-D subspace

"latent dimension"

How many voxels are really being used?

voxel 1

voxel 2

voxel 3

voxel 4

voxel N

...

A key statistic: dimensionality

Representational space

Is the visual code low-dimensional?

" [...] the topographies in VT cortex that support a wide range of stimulus distinctions [...] can be modeled with 35 basis functions"

Haxby et al., 2011, 2020

Computational goal: compressing dimensionality?

~87 dimensions of object representations in monkey IT

"A progressive shrinkage in the intrinsic dimensionality of object response manifolds at higher cortical levels might simplify the task of discriminating different objects or object categories."

Lehky et al., 2014, 2016

But high-dimensional coding has benefits!

detecting high-dimensional codes requires large datasets!

high representational capacity, expressive

makes learning new tasks easier

Fusi et al. (2016)

Elmoznino & Bonner (2024)

Does human vision use a

low- or high-dimensional code?

The Natural Scenes fMRI dataset

Allen et al. (2022)

8 subjects

7 T fMRI

10,000 images (repeated 3×)

from the COCO dataset

"Have you seen this image before?"

continuous recognition

Natural Scenes Dataset

10,000 images, seen thrice

"Have you seen this image before?"

Ideal dataset to characterize dimensionality

Allen et al. (2022)

very large-scale
complex naturalistic stimuli

How can we estimate dimensionality?

same geometry,

new perspective!

ambient
dimensions

voxel 1

voxel 2

rotation

latent
dimensions

Key statistic: the covariance spectrum

latent dimensions sorted by variance

variance along the dimension

low-dimensional code

high-dimensional code

variance along each dimension

stimulus-related signal

trial-specific noise

Should we just apply PCA?

Cross-decomposition: a better PCA

generalizes

... across stimulus repetitions

... to novel images

X_\text{train}

Y_\text{train}

trial 1

trial 2

neurons (or) voxels

stimuli

\text{cov}\left(X_\text{train}, Y_\text{train}\right) = U \Sigma V^\top

Learn latent dimensions

Step 1

Step 2

Evaluate reliable variance

\hat{\Sigma} = \text{cov}\left(X_\text{test} U, Y_\text{test} V\right)

X_\text{test}

Y_\text{test}

If there is no stimulus-related signal, expected value = 0

PCA vs cross-decomposition

logarithmic binning + 8-fold cross-validation

Our covariance spectra

latent dimensions sorted by variance on the training set

reliable variance on the test set

\text{covariance} \propto \left(\text{rank}\right)^\alpha

no small "core subset"
thousands of dimensions!
limited by dataset size

\text{covariance} \propto \left(\text{rank}\right)^\alpha

A power-law covariance spectrum

more data

more dimensions?

Zipf's law: the classic power law

\text{word frequency} \propto \left(\text{rank}\right)^{-1}

Using only the 200 most frequent words...

i love sci-fi and am willing to put up with a lot. Sci-fi movies/tv are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good tv sci-fi as babylon 5 is to star trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, cg that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a'sci-fi'setting. (I'm sure there are those of you out there who think babylon 5 is good sci-fi tv. It's not. It's clichéd and uninspiring.) while us viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star trek). [...]

i love and to up with a lot. Are, and. I to like this, I really did, but it is to good as is to (the).,,, that doesn't the, and characters be with a''. (I'm there are those of you out there who think is good. It's not. It's and.) while us like and character, is a that does not take [...]