1
2, 1
1
light enters your eyes
1
signals travel to visual cortex
2
cortical processing
3
perceptual experience
4
vector of voxel responses
voxel 1
voxel 4
voxel N
...
stimulus
neural response
- What information about the stimulus is represented here?
- Can we build computational models that capture these responses?
Credit: NordicNeuroLab
fMRI
functional magnetic resonance imaging
resolution
~1 mm
~3 s
coarse measure of neural activity
voxel 1
voxel 2
voxel 3
voxel 4
voxel N
...
What is the geometry of neural representations?
A key statistic: dimensionality
voxel 1
voxel 2
data spans both "ambient" dimensions
voxel 1
voxel 2
data lives in a 1-D subspace
How many dimensions are really being used?
"latent dimension"
Why do we care about dimensionality?
How much information about the stimulus can be encoded?
- animate vs inanimate
- stubby/boxy vs thin/elongated
- ...
How stable is the representation to irrelevant perturbations?
- invariant recognition
- smoothness of the code
robustness
expressivity
low-D
high-D
Is the visual code low-dimensional?
" [...] the topographies in VT cortex that support a wide range of stimulus distinctions [...] can be modeled with 35 basis functions"
Is the visual code low-dimensional?
"Thus, our fMRI data are sufficient to recover semantic spaces for individual subjects that consist of 6 to 8 dimensions."
Computational goal: compressing dimensionality?
~87 dimensions of object representations in monkey IT
"A progressive shrinkage in the intrinsic dimensionality of object response manifolds at higher cortical levels might simplify the task of discriminating different objects or object categories."
But high-dimensional coding has benefits!
Detecting high-dimensional structure requires large-scale, high-quality fMRI datasets...
What is the dimensionality of visual cortex representations?
How are the latent dimensions spatially organized?
Outline
8 subjects
7 T fMRI
"Have you seen this image before?"
continuous recognition
Ideal dataset to characterize dimensionality
- very large-scale
- complex naturalistic stimuli
How can we estimate dimensionality?
same geometry,
new perspective!
ambient
dimensions
voxel 1
voxel 2
rotation
latent
dimensions
Key statistic: the covariance spectrum
latent dimensions sorted by variance
variance along the dimension
- logarithmic scales
- multiple OOMs
low-dimensional code
high-dimensional code
no variance along other dimensions
variance along many dimensions
"core subset" of relevant dimensions
variance along each dimension
Should we just apply PCA?
stimulus-related signal
trial-specific noise
Cross-decomposition: a better PCA
generalizes
... across stimulus repetitions
... to novel images
X_\text{train}
Y_\text{train}
trial 1
trial 2
neurons (or) voxels
stimuli
\text{cov}\left(X_\text{train}, Y_\text{train}\right) = U \Sigma V^\top
Learn latent dimensions
Step 1
Step 2
Evaluate reliable variance
\hat{\Sigma} = \text{cov}\left(X_\text{test} U, Y_\text{test} V\right)
X_\text{test}
Y_\text{test}
PCA vs cross-decomposition
logarithmic binning + 8-fold cross-validation
If there is no stimulus-related signal...
... but cross-decomposition
has an expected value of 0.
PCA
cross-decomposition
Our cross-validated covariance spectra
\text{covariance} \propto \left(\text{rank}\right)^\alpha
binned logarithmically to increase SNR at high ranks
reliable variance on the test set
latent dimensions sorted by variance on the training set
\text{word frequency} \propto \left(\text{rank}\right)^{-1}
i love sci-fi and am willing to put up with a lot. Sci-fi movies/tv are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good tv sci-fi as babylon 5 is to star trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, cg that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a'sci-fi'setting. (I'm sure there are those of you out there who think babylon 5 is good sci-fi tv. It's not. It's clichéd and uninspiring.) while us viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star trek). [...]
i love and to up with a lot. Are, and. I to like this, I really did, but it is to good as is to (the).,,, that doesn't the, and characters be with a''. (I'm there are those of you out there who think is good. It's not. It's and.) while us like and character, is a that does not take [...]
- no small "core subset"
- thousands of dimensions!
- limited by dataset size
\text{covariance} \propto \left(\text{rank}\right)^\alpha
A power-law covariance spectrum
more data
more dimensions?
What about smaller visual cortex regions?
artifact of analyzing a large ROI?
high-dimensional everywhere!
or
- Stimulus-related visual information is spread over thousands of latent dimensions, even in fMRI data
- Cortical representations are high-dimensional even after several stages of visual processing — even more than I've shown you here!
high-dimensional
- power-law covariance
- unbounded
Is this high-dimensional structure shared across people?
vs
Cross-decomposition still works!
generalizes
... across stimulus repetitions
... to novel images
... across participants
X_\text{train}
Y_\text{train}
subject 1
subject 2
neurons (or) voxels
stimuli
\text{cov}\left(X_\text{train}, Y_\text{train}\right) = U \Sigma V^\top
Learn latent dimensions
Step 1
Step 2
Evaluate reliable variance
\hat{\Sigma} = \text{cov}\left(X_\text{test} U, Y_\text{test} V\right)
X_\text{test}
Y_\text{test}
Functional – not anatomical – alignment is required
\frac{\text{cov}(S_1, S_2)}{\sqrt{\text{cov}(S_1, S_1) \text{cov}(S_2, S_2)}}
Cross-individual covariance spectra
reliable shared variance on the test set
latent dimensions sorted by variance on the training set
Shared high-dimensional structure across individuals
Shared high-dimensional structure across individuals
Are these spectra just an artifact of the method used?
PCA
cross-decomposition
or
or
high-dimensional
- power-law covariance
- unbounded
striking universality across individuals
Standard methods are insensitive to high-dimensional structure
Cognitive neuroscience needs new approaches!
- Impossible to interpret every single latent dimension :(
- Let's embrace the high-dimensional nature of neural representations instead!
- What are the generative mechanisms that induce this latent structure?
What is the dimensionality of visual cortex representations?
How are the latent dimensions spatially organized?
Outline
A brief aside: findings from systems neuroscience
6 mice
calcium imaging
~10,000 neurons
~2,800 natural images
Not due to the 1/f of natural images
Power-law index of -1: a critical point
Similar power laws in mouse V1 representations
similar neural population statistics
- in different species
- across imaging methods
- at very different resolutions
also shared across individuals!
What is the dimensionality of visual cortex representations?
How are the latent dimensions spatially organized?
Outline
\text{cov}\left(X_\text{train}, Y_\text{train}\right) = U \Sigma V^\top
Learn latent dimensions
Step 1
each is a linear combination of voxels
U_1, U_2, U_3, \dots
V_1, V_2, V_3, \dots
voxel 1
voxel 4
voxel N
...
U_{k,1}
U_{k,4}
U_{k,N}
low-rank
high-rank
1
10
100
1,000
large spatial scale
small spatial scale
4
8
20
1,000
1
1,000
30 mm
80 μm
Measuring characteristic spatial scales
Reminiscent of a Fourier basis ...
1-D Fourier basis
2-D Fourier basis
V1 activations are ~translationally invariant
When do latent dimensions look like a Fourier basis?
Answer: when data are translationally invariant
- Most variance in the data is on large spatial scales, in both humans and mice
- Low-resolution imaging should get us most of the way to explaining the variance in the data ...
Does coarse-graining preserve covariance spectra?
high-resolution data
simulated low-resolution data
spatial binning
If our tools are not sensitive to high-dimensional structure, we might as well collect low-resolution data.
Spatial scale: a potential organizing principle?
Spatial scale: a potential organizing principle?
number of neurons in human V1
spacing between neurons
a code that leverages all latent dimensions
Credit: NordicNeuroLab
Spatial-scale invariance explains similar findings from ...
calcium imaging
fMRI
shared representations across individuals
striking universality across species
Takeaway: universal scale-free representations!
Theoretical constraints
The power-law exponent has an upper bound of -1 to maximize expressivity while being robust
Physical scaffolding
Scale-free structures in the connectivity patterns of neurons in cortex
Generic learning mechanism
Allows cortex to scale arbitrarily in size while maintaining the same representational format
Why do we see a universal scale-free spectrum?
Some current follow-ups I'm working on
Are these high-dimensional neural representations shared with
... mental representations derived from a behavioral task?
... internal representations of artificial neural networks?
What is the representational content of these latent dimensions?
Chihye (Kelsey) Han
Are humans behaviorally sensitive to low-variance visual information present in the tail of the spectrum?
Current related work from our lab
Some future project ideas
In category-selective regions, are preferred stimuli represented more high-dimensionally? What about patients with deficits?
Are there equivariant object representations in cortex?
In category-selective regions, are preferred stimuli represented more high-dimensionally? What about patients with deficits?
FFA
LOC
Are there equivariant object representations in cortex?
Are there systematic general transformations in the cortical representations of related stimuli?
Thank you!
The Bonner Lab!
The Isik Lab!
scale-free-visual-cortex
By raj-magesh
scale-free-visual-cortex
A slide deck to present the work described in the preprint "Universal scale-free representations in human visual cortex" (https://arxiv.org/abs/2409.06843v1)
