The Cortical Representation of Body Part Concepts

Stephen Mazurchuk

10/28/21

A plug

Language Imaging Laboratory

Central Question:

What are the neural substrates that support concept representation?

Why?

In general, there are many parts of the cortex that might be lesioned, and we want to be able to know what functions are supported by different regions

Why should something like word meaning be localizable in the cortex?

A Case

Case

66 y/o man with severe Broca's aphasia
MRI shows haemorrhagic infarcts in ithe left angular gyrus, posterior end of the left STG, and lateral part of the left frontal lobe
On exam, alert and cooperative
Comprehension of spoken speech was judged to be good
Showed a consistent and striking disability in pointing to 'body parts'

Case

66 y/o man with severe Broca's aphasia
MRI shows haemorrhagic infarcts in ithe left angular gyrus, posterior end of the left STG, and lateral part of the left frontal lobe
On exam, alert and cooperative
Comprehension of spoken speech was judged to be good
Showed a consistent and striking disability in pointing to 'body parts'

Testing

Point to a corresponding picture among 10 cards belonging to the same category from either auditory or visual presentation of a name

Case

Suzuki, K., Yamadori, A. & Fuji, T. Category-specific comprehension deficit restricted to body parts. Neurocase 3, 193–200 (1997)

Category Specific Semantic Deficits

A brief history
Experiential models
- RSA
Decoding Analysis
Encoding Analysis
ROI Specificity
Clinical Correlates
Closing Thoughts

Talk Outline

Earliest Reference

"These three cases then suggest that animate objects may be recognized and revisualized by the left occipital lobe while the same functions for inanimate objects proceed through functional activity of the right lobe"

* 2nd revised edition, 1940

Many proposed category distinctions
- Artifacts, Animals, Tools, Plants/Food, Numerosity, Vehicles, Events, Human Traits
- Sometimes called "natural categories"
Living/Non-living (Animals and Artifacts) is most widely studied dissociation

Key Point

Few fMRI studies have focused on determining the lexical-semantic neural correlates of body-part knowledge

Body part

Conceptual

Representations

(sometimes termed body-image)

Explain how focal Lesions to the cortex can result in deficits to a particular category of concepts

Goal:

Explain how focal Lesions to the cortex can result in deficits to a particular category of concepts

Goal:

Some areas of the cortex preferentially process certain sensory, motor, affective, and other experiential phenomena

Concepts are composed of sensory, motor, affective, and other experiential content

Conclusion:

1)

2)

"experiential features"

The spatial distribution and disruption of category knowledge is explained by "experiential" accounts of concept representation

Propositions:

(hypothesis)

both primary sensory cortices, but also adjacent association cortices

"Natural categories" of concepts arise from how concepts "cluster" on sensory, motor, and affective dimensions

3)

Explain how focal Lesions to the cortex can result in deficits to a particular category of concepts

Goal:

Some areas of the cortex preferentially process certain sensory, motor, affective, and other experiential phenomena

Concepts are composed of sensory, motor, affective, and other experiential content

Conclusion:

1)

2)

"experiential features"

The spatial distribution and disruption of category knowledge is explained by "experiential" accounts of concept representation

Propositions:

(hypothesis)

both primary sensory cortices, but also adjacent association cortices

"Natural categories" of concepts arise from how concepts "cluster" on sensory, motor, and affective dimensions

3)

Evidence that concepts are experiential

Part I

Neural Representation of Concepts

Model Representation of Concepts

\rho

Evidence that concepts are experiential

Part I

Neural Representation of Concepts

Model Representation of Concepts

\rho

Concept Representation as Experiential Attributes

CREA

Participants are asked to rate on a likert scale how important they think a given attribute is to a given concept

Evidence that concepts are experiential

Part I

Neural Representation of Concepts

Model Representation of Concepts

\rho

\checkmark

How can fMRI tell us about concept representation?

The tool

\hat{f}(t) = \beta_1*s(t) + ... \ \varepsilon

f(t)

s(t)

\hat{f}(t) = \beta_1*s(t) + ... \ \varepsilon

Acquisition Parameters

Images acquired on 3T GE Premier
T1 MPRAGE and T2 CUBE acquisition (0.8 x 0.8 x 0.8 mm\(^3\)) on each day
T2*-weighted echoplanar images were obtained using a simultaneous multi-slice sequence
- SMS factor = 4
- TR = 1500 ms, TE = 23 ms, flip angle = 50\(^\circ\)
- 104 x 104 x 72, voxel size = 2 x 2 x 2 mm\(^3\)
- Pairs of SE with opposing PE directions were acquired before run 1, between runs 3 and 4, and after run 6

Study Design

Words presented visually in a fast event-related design with variable inter-stimulus intervals
All 300 stimuli presented 6 times in psuedo-random order over three imaging sessions on separate days
- Over 6 hours of scans!

Pre-Processing

All data processed in a containarized version of fmriprep
- BIDS format
- ANTS used for brain extraction and normalization
- fsrecon all to reconstruct the surface. Then BOLD time series was projected to the surface
- CSF and WM time series extracted
- Feild map estimated with 3dQwarp

1st level Analysis

Seperate regressor for each word and 3dREMLfit was used to estimate t-maps for each word
- 12 motion regressors and CSF and WM
- Response time z-scored and regressed out
Regression was performed on the surface time-series data

Surface normalization

Pial Surface Segmentation

Abdomen

Alligator

Ankle

Apathy

Apple

Armpit

Single Word fMRI Activation Patterns

Evidence that concepts are experiential

Part I

Neural Representation of Concepts

Model Representation of Concepts

\rho

\checkmark

?

Representational Similarity Analysis

Suppose we have two models (or representations of the data). How do we compare them?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Voxel 1	Voxel 2	Voxel 3	Voxel 4	Voxel 5
Car	.23	.58	.49	.78	.86
Airplane	.98	.28	.34	.18	.52
Chicken	.62	.82	.91	.36	.17

Model 2 (Neural Data)

Model 1 (experiential)

Representational Similarity Analysis

How we compare models?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane	Chicken
Car
Airplane
Chicken

RSA

How we compare models?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane	Chicken
Car	1
Airplane
Chicken

RSA

How we compare models?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane
Car	1	.62
Airplane
Chicken

RSA

How we compare models?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane	Chicken
Car	1	.62	.1
Airplane
Chicken

RSA

How we compare models?

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane	Chicken
Car	1	.62	.1
Airplane		1	.12
Chicken

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Taste	Shape	Color
Car	0	4.2	4.9
Airplane	0	5.1	4.3
Chicken	5.5	2.2	2.2

	Car	Airplane	Chicken
Car	1	.62	.1
Airplane		1	.12
Chicken			1

	Voxel 1	Voxel 2	Voxel 3	Voxel 4	Voxel 5
Car	.23	.58	.49	.78	.86
Airplane	.98	.28	.34	.18	.52
Chicken	.62	.82	.91	.36	.17

	Voxel 1	Voxel 2	Voxel 3	Voxel 4	Voxel 5
Car	.23	.58	.49	.78	.86
Airplane	.98	.28	.34	.18	.52
Chicken	.62	.82	.91	.36	.17

	Car	Airplane	Chicken
Car	1	.42	.06
Airplane		1	.31
Chicken			1

\rho

Called an RDM

Model 1

Word voxel activation patterns

Model 2

Word attribute ratings (CREA)

Advice

Banana

Celebration

(actual model has 65 features)

Note:

Some regions of the cortex have been consistently measured to be involved with language
We will limit ourselves to only looking at activation patterns from vertices within a liberal estimation of the generally accepted semantic areas

Yeo, T. B. et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. J Neurophysiol 106, 1125–1165 (2011).

Result

\rho

51,040 data points in each correlation

But, is this a good result?

Depends if subjects correlate with each other
Depends on if there are any better models

But, is this a good result?

Depends if subjects correlate with each other
Depends on if there are any better models

The (lower) Noise Ceiling

The best that a model could be is given by:

The average correlation of one subject's similarity matrix with the average similarity matrix of all other subjects

Intuition:

The best possible model to the data is just the average similarity matrix of the subjects
If subjects don't look like the average of all other subjects, than there is no shared variance!

\mu = .19

\sigma = .07

\sigma = .04

\mu = .10

Lower Noise Ceiling

CREA Correlation

In general, our model explains about 1/2 of all explainable variance

How do other models compare?

Previously Discussed

Short answer is the experiential model outperforms other models with this metric

\checkmark

Question:

Yes No

Do experiential features correlate with neural activation patterns?

\checkmark

Question:

Yes No

* With a particular focus on the category of body-parts

Do experiential features correlate with neural activation patterns?

Do "natural categories"* emerge from their loadings on different experiential features?

Mean Ratings For Individual Categories

Clustering Analysis of Concepts based on experiential features

Do "natural categories"* emerge from their loadings on different experiential features?

\checkmark

Question:

Yes No

Are there any parts of the cortex where "natural categories"* emerge from the fMRI activation patterns?

* With a particular focus on the category of body-parts

Do experiential features correlate with neural activation patterns?

\checkmark

Part II

Decoding Analysis

Decoding Model

For a patch of cortex, train a linear Support Vector Machine (SVM) to classify a pattern as being a body part or not

Decoding

"If decoding succeeds on the test set, then the region must contain some information about the decoded variable"

Non-bodyparts

\(\psi_2\)

Body parts

\(\psi_1\)

Body Part Searchlight SVM

toggle

FDR p=.005

Do "natural categories"* emerge from their loadings on different experiential features?

\checkmark

Question:

Yes No

Are there any parts of the cortex where "natural categories"* emerge from the fMRI activation patterns?

* With a particular focus on the category of body-parts

\checkmark

Can experiential features predict the fMRI activation patterns in the above region?

Do experiential features correlate with neural activation patterns?

\checkmark

Part III

Encoding Analysis

Explaining the Result

** Important **

The encoding model predicts the distance between activation patterns (RDM) for body parts (test set) without having any body-part words in the training set

Encoding

(Predict activation patterns for body-parts using observed activation patterns for other words)

Decoding

Body parts

Animals

\(\psi_1\)

\(\psi_2\)

Model

Words

Features

Explaining the Result

Model

Words

Features

\text{Word}_i = \beta_1 \cdot R_i + ... + \beta_{65}\cdot R_i

Encoding

For a given voxel

For word \(i\)

	Car	Airplane	Chicken
Car	1	.62	.1
Airplane		1	.12
Chicken			1

	Car	Airplane	Chicken
Car	1	.42	.06
Airplane		1	.31
Chicken			1

\rho

Predicted Correlational Structure

Observed Correlational Structure

predicted voxel response

feature \(1\) rating for word \(i\)

BUT ...

Our model (regressors) is Co-linear
We have a lot of regressors

Shrinkage!

Ridge Regression

regular least squares

\sum_{i=1}^n (y_i - \sum_{j=1}^p x_{ij}\beta_j)^2

\text{Loss =}

+ \lambda \sum_{j=1}^p \beta_j^2

penalization

Need to choose the strength of the regularization parameter (\(\lambda \))

Shrinkage

An important concept in shrinkage is the "effective" degrees of freedom associated with a set of parameters.

df(\lambda) = \displaystyle\sum\limits_{j=1}^p \frac{d_j^2}{d_j^2 + \lambda}

Plan: Do PCA to estimate intrinsic dimensionality. Then estimate shrinkage parameter

Result

CREA encoding model

Body part ROI

Encoding Model ROI

FWEP p=.005

Can Other Models account for the activation patterns?

A very common model used in the literature is called Word2Vec

I bought chicken for dinner

bought chicken dinner

bought pork dinner

bought steak dinner

remove stop words

context words

target word

Basic premise is that words that appear in similar context's have similar meaning
Each word is a vector. The dot product of two vectors should reflect how similar the contexts are for those words

*No area survived multiple comparisons correction

W2V Encoding Performance

Do "natural categories"* emerge from their loadings on different experiential features?

\checkmark

Question:

Yes No

Are there any parts of the cortex where "natural categories"* emerge from the fMRI activation patterns?

* With a particular focus on the category of body-parts

Can experiential features predict the fMRI activation patterns in the above region?

\checkmark

While the above region is sensitive to the body-part distinction, is this region specific to body-parts?

Do experiential features correlate with neural activation patterns?

\checkmark

Part IV

ROI Specificity

Is the classification accuracy the same for all categories in the body-part ROI?

Q)

Animal

Artifact

Body part

Plant\Food

As compared to Animals and Artifacts, body parts appear to have a somewhat specific representation in this ROI

Do "natural categories"* emerge from their loadings on different experiential features?

\checkmark

Question:

Yes No

Are there any parts of the cortex where "natural categories"* emerge from the fMRI activation patterns?

* With a particular focus on the category of body-parts

Can experiential features predict the fMRI activation patterns in the above region?

\checkmark

While the above region is sensitive to the body-part distinction, is this region specific to body-parts?

\checkmark

Is there clinical/previous evidence to corroborate these results?

Do experiential features correlate with neural activation patterns?

\checkmark

Clinical Correlates

Part VI

Bodypart assessment

Developed two tests to assess the lexical-semantic representation of body
- point to body part most similar in function to target body part
- point to body part most closely associate with a pictures item (clothing / tool)
Administered to 70 stroke patients
3 had selective body-image impairment

Schwoebel, J. & Coslett, H. B. Evidence for Multiple, Distinct Representations of the Human Body. J Cognitive Neurosci 17, 543–553 (2005)

Finding

All three subjects with body image lesions had suffered temporal lesions; as shown in Figure 2, the lesion involved portions of Brodmann's area 37 as well as underlying white matter in 2 subjects

BA 37

Peak BP Area

``

Deeper Neuropsychological Investigation

Battery of 12 tests assessing lexical and conceptual aspects of body part knowledge to 104 brain-damaged patients
Main finding was remarkably intact semantic understanding
10 had impaired production, only 1 patient with impaired composite comprehension index

In the 9 patients with body part anomia, oral naming of concrete entities was evaluated, and this revealed that 4 patients had disproportionately worse naming of body parts relative to other types of concrete entities

Kemmerer, D. & Tranel, D. Searching for the elusive neural substrates of body part terms: A neuropsychological study. Cognitive Neuropsych 25, 601–629 (2008)

``

Caveat

In the composite comprehension index, only one patient had an impaired score, and that happened to be the one with the EBA lesion
None had impaired understanding of the "meanings" of body part terms
- Most errors were paraphasias or omissions

lexical verses conceptual deficits

Of 12 tests, 4 required production of terms, and 7 required comprehension of terms

Goldstein, E. B. & Brockmole, J. Sensation and Perception. (Cengage Learning, n.d.)

Previous fMRI Result

16 subjects, 4 semantic categories
Task was within group similarity comparison
Found body and clothing activated BA 37

Do "natural categories"* emerge from their loadings on different experiential features?

\checkmark

Question:

Yes No

Are there any parts of the cortex where "natural categories"* emerge from the fMRI activation patterns?

* With a particular focus on the category of body-parts

Can experiential features predict the fMRI activation patterns in the above region?

\checkmark

While the above region is sensitive to the body-part distinction, is this region specific to body-parts?

\checkmark

Is there clinical/previous evidence to corroborate these results?

\checkmark

Do experiential features correlate with neural activation patterns?

\checkmark

Closing Thoughts

Indication that left pMTG might play an important role in lexical-semantic representations of body parts
Evidence that experiential accounts explain fMRI activation patterns

Future Directions

Determine the experiential features most important for explaining the neural data
Repeat similar analysis for other categories
Examine the less significant ROI's for the body-part category

Takeaway:

Decent evidence that the activation patterns within the brains "conceptual system" are somewhat explained by the sensory-motor (experiential) features being processed

MSTP:

Joseph Barbieri, PhD

Calvin Williams, MD, PhD

Nita Salzman, MD, PhD

Gil White, MD

Sid Rao, MD, PhD

Language Lab:

Jia-Qing Tong

Jeffrey Binder, MD

Leonardo Fernandino, PhD

Lisa Conant, PhD

Songhee Kim, PhD

Alex Helfand, PhD

Ann Moll

Kim Peplinksi

Joe Heffernan, MS

Jed Mathis

Samantha Drane

Belle Banke

Thank You!

Questions?

EXTRA SLIDES

Shrinkage

W2V Shrinkage

df(\lambda) = \displaystyle\sum\limits_{j=1}^p \frac{d_j^2}{d_j^2 + \lambda}

Extra:

Similar question to doing RSA overall.
The multivariate second moment of the activity profiles fully defines the representational geometry and with it all the information that can nbe linearly or nonlinearly decoded
- If the the noise is assumed gaussian, than the second moment also determines SNR
It doesn't depend on getting single neurons right, but rather, capturing the population code

Why do RSA with the encoding model?

Word List

Body-Parts
instep	spine
eyebrow	muscle
pancreas	waist
leg	cheek
trachea	skeleton
abdomen	liver
ligament	cartilage
clavicle	knuckle
kidney	navel
testicle	thumb
forearm	eyelid
nipple	elbow
stomach	diaphragm
finger	shoulder
beard	tooth
skull	wrist
nostril	pelvis
torso	intestines
heel	armpit
toenail	belly
earlobe	bladder
retina	ankle
thigh	mustache
fingernail	nose
lip	forehead

Encoding Model

For each voxel, the activation is predicted by the mean rating for the word
We then compare the relationships between the predicted activation patterns to the observed activation patterns.

Case

64 y/o man, gradual onset of a right hemiparesis
Alert cooperative man, fully oriented for place and person, but mildly disoriented for time.
Agraphia, but no reading deficit.

Called Autotopagnosia

Could not point to named body parts (on himself or manikin)
No errors in naming body part pointed to by the examiner
The patient was amazed by this discrepancy.

Was able to describe function of body parts
Conversational speech was fluent and appropriate

Renzi, E. D. & Scotti, G. Autotopagnosia: Fiction or Reality?: Report of a Case. Arch Neurol-chicago 23, 221–227 (1970)

Where is there shared variance in activation patterns?

A "semantic model" would be unable to explain:
- Noise inherent to measurement
- "Unique" representational content

Body part

Conceptual

Representations

sometimes termed body-image

Tasks were:

Point to body part most similar in function to target body part
Body part most associated with clothing or tool