The High-Dimensional Structure of

Visual Cortex Representations

A Dissertation Proposal

Outline

  • Motivation
  • Projects
    1. Universal scale-free representations in human visual cortex
    2. Spatial-scale invariant properties of mammalian visual cortex
    3. Characterizing the representational content of different latent subspaces
  • Timeline

Motivation

  • Population codes in cortex are not fully understood
  • Standard methods include
    • decoding: "Can the neural responses do task ___ ?"
    • encoding: "Can my model predict neural responses?"

 

What are the statistical properties of the cortical representations themselves?

 

Specifically, let's look at dimensionality.

A low-dimensional theory of visual cortex

Goal

 

Compress high-dimensional images onto a low-dimensional manifold that supports behavior while being robust to stimulus variation

Haxby (2011), movie-viewing fMRI

Huth (2012), movie-viewing fMRI, semantic space

Lehky (2014), objects, monkey electrophysiology

A high-dimensional theory of visual cortex

Benefits

 

Expressive enough to capture the complexity of the real world; supports performing a variety of tasks

Stringer (2019), mouse visual cortex, ImageNet

mouse cortex also scales to ~10^6 dimensions, Manley et al. (2024)

Posani (2024), mouse cortex during behavior

How can we resolve these contradictions?

Use new large-scale, high-quality fMRI datasets!

Project 1

 

Universal scale-free representations in human visual cortex

 

[manuscript under review]

The Natural Scenes dataset

Cross-decomposition ~ cvPCA + hyperalignment

Universal scale-free covariance spectra

Anatomical alignment is insufficient

2024-05-18T07:29:15.773633 image/svg+xml Matplotlib v3.8.4, https://matplotlib.org/ 10 −7 10 −5 10 −3 10 −1 V1-4 V1 V2 V3 V4 category-selective faces bodies places words large regions general frontal 10 0 10 1 10 2 10 3 10 4 10 −7 10 −5 10 −3 10 −1 ventral stream early midventral ventral 10 0 10 1 10 2 10 3 10 4 parietal stream early midparietal parietal 10 0 10 1 10 2 10 3 10 4 lateral stream early midlateral lateral rank covariance

Systematic variation aross some ROIs

RSA is insensitive to high-rank dimensions

Summary

  • Scale-free covariance spectra
  • Detectable in fMRI data (!) with spectral binning
  • Present everywhere in visual cortex
  • High-dimensional structure shared between individuals
  • Only detectable with spectral methods

Project 2

 

Spatial scale-invariant properties of mammalian visual cortex

 

[manuscript in preparation]

Similar covariance spectra in humans and mice*

Latent dimensions appear spatially structured

Covariance functions are spatially stationary

Covariance spectra are spatial scale-invariant

Some implications

  • Universality across species: mouse visual cortex is perhaps not as much of an exception anymore
  • Universality across scales: explains why similar phenomena are observed in measurements at different resolutions

How does this universal power spectrum arise?

Some possible explanations

  • Theoretical constraints: the power-law exponent has an upper bound of -1 to maximize expressivity while being robust
  • Physical scaffolding: scale-free structures in the connectivity patterns of neurons in cortex
  • Generic learning mechanism: allows cortex to scale arbitrarily in size while maintaining the same representational format

Project 3

 

Characterizing the representational content of different latent subspaces

 

[planned; preliminary results]

What information is available at different ranks?

Fine-grained semantic annotations available

Hypothesis 1

coarse category information?

fine-grained distinctions?

Hypothesis 2

  • CNNs
    • AlexNet
    • VGG-16
    • ResNet-32
  • Vision transformers
  • Scattering network

 

  • Untrained baselines

low-variance dimensions?

high-variance dimensions?

Testing these hypothesis: Proposed methods

  1. Decoding category-level information at different levels of granularity from the neural representations
    • linear classifiers (SVMs, logistic regression)
  2. Encoding models to predict the activation in different latent subspaces of neural representations
    • pretrained artificial neural networks
    • linear regression + regularization

Is SNR too low in high-rank subspaces?

Apparently not.

 

As a simple proof-of-concept, a nearest centroid classifier can perform pairwise instance-level classification using information in all latent subspaces.

ranks 1-10

10-100

Preliminary results

Timeline

Made with Slides.com