Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

We measure social perception

of human faces

in a vision-language model.

A photo of a

person

A photo of a

person

 A photo of a
person

Measuring social perception via

Cosine Similarity

FairFace
Karkkainen et al. (2021)

UTKFace
Zhifei et al. (2017)

FairFace
Karkkainen et al. (2021)

UTKFace
Zhifei et al. (2017)

CausalFace
Liang et al. (2023)

age

CausalFace

CausalFace

female

male

age

CausalFace

female

male

age

Asian

Black

White

Legally protected

Legally protected

Non-protected

smiling

lighting

pose

 A photo of a
person

 A photo of a

person

Stereotype Content Model

Fiske et al. (2007)

Agency Belief Communion Model

Koch et al. (2016)

Warmth

Competence

unfriendly
friendly

Agency

Belief

Communion

+

–

C

P

–

+

surgeon

parent

 A photo of a

 

person

friendly

Prompt templates

  • A photo of a <attribute> person.
  • A <attribute> person.
  • This is a <attribute> person.
  • Cropped face photo of a <attribute> person.

We deploy an experimental dataset.

1.

We deploy theories of social perception.

2.

We investigate the embedding space directly.

3.

Protected attributes

female

male

age

Asian

Black

White

Non-protected attributes

smiling

lighting

pose

How do

protected and

non-protected

attributes affect social perception?

smiling

Bootstrapping differences

Bootstrapping differences

smiling

—

Bootstrapping differences

smiling

—

—

protected and non-protected attributes

—

Wilcoxon Rank-Sum test, independent samples,
\(p<0.001\)

—

—

ns

—

ns

  • Non-protected attributes cause as much variation as protected ones.
  • Considering a wide spectrum of protected and non-protected variables is necessary to understand and measure biases comprehensively.

Appendix

Word Embedding Association Test (WEAT)

Caliskan et al. (2017)

pooled sd

asian                                     black

photo of a warm person

photo of a warm person

asian                                     black

—

WEAT

Kruskal-Wallis  \(\chi^2\) = 1.6,
p-value = 0.4

protected and non-protected attributes

–

+

Bootstrapping Variations 

  • We randomly choose two distinct values, \(x_1,x_2 \sim X\), for the chosen dimension (e.g., white and black).
  • For each pair of values, we select the respective image embeddings, \(i_1(x=x_1), i_2(x=x_2)\) that are equal in all other dimensions (in this example: gender, age, smiling, lighting, and pose).
  • We then compute the difference in cosine similarities between each image embedding and a text embedding \(t\), defined as \(\Delta(t, i_1, i_2) = \lvert \cos(i_1, t) - \cos(i_2, t) \rvert\).
  • This process is repeated 1,000 times, generating a bootstrap distribution of \( \Delta \) values.
  • This distribution describes the impact of the specific dimension on the cosine similarity of image embeddings and text embedding.
     

Heatmap of Pearson correlation coefficients of positive and negative valence dimensions of the ABC model.

Smiling

a photo of a person
a photo of a person