Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a CLIP vision-language model

by @kevinschawinski

Bloomberg, March 8th, 2024

Humans spontaneously make social judgments from the photographs of faces.

(Oosterhof 2008, Sutherland 2018, Todorov 2017)

We measure social perception

• CLIP is a state-of-the-art vision-language model that connects images and text.
• It is used for tasks like image recognition, captioning,
• and powering applications such as DALL·E.

A photo of a

person

 A photo of a
person

Measuring Social Perception via

Cosine Similarity

age

CausalFace

CausalFace

female

age

CausalFace

female

age

Asian

Black

White

Legally Protected

 A photo of a
person

 A photo of a

person

 A photo of a

person

friendly

FairFace

UTKFace

CausalFace

How does CausalFace compare to wild-collected datasets?

Commonly used bias-metrics

Markedness 

• Markedness (Wolfe and Caliskan, 2022)
• WEAT (Caliskan et al., 2017)
• Skew@k, NDKL (Geyik et al., 2019)
• Mean cosine similarities

✓

✓

✓

✓

Commonly used bias-metrics

Legally "Protected" Attributes

female

age

Asian

Black

White

Non-protected Attributes

smiling

lighting

pose

How do

protected and

non-protected

attributes affect social perception?

smiling

Bootstrapping Differences

Bootstrapping Differences

smiling

Bootstrapping Differences

smiling

How does

age-related

social perception compare across datasets?

Warmth

Competence

Communion

–

+

Agency

–

+

Agency

Agency

Agency

–

+

Agency

+

Positive Agency

youngest

oldest

• The observation that Black women are a special category in the social perception of age is consistent with human subject research.
• 'Strong Black Woman ideal' is reinforced with age (Baker 2015).

age

smiling

Smiling

Smiling

Warmth

most frowning

most smiling

most frowning

most smiling

Limitations

• Attribute Manipulation Effectiveness
  • ​Effects of lighting, facial expressions etc. might differ across demographic groups. .
• Residual Confounds?
  • Some color confounds might still be present despite controls for background, clothing, and hair color.
• Dataset vs. Model Bias
  • ​Three datasets. But only one CLIP model. No comparison to human ratings.

The impact of protected and non-protected characteristics is comparable in size.

Social Perception of age show six clustered race-gender groups in CausalFace.

Strongly diverging age effects for Black Women.

Strong impact of smiling of Black Women on positive social perception.

carinah@ethz.ch

slides.com/carinah

https://github.com/carinahausladen

Appendix

Smiling

Smiling

Caliskan et al. (2017)

asian                                     black

—

WEAT

protected and non-protected attributes

–

+

Theoretical Models

Statistical discrimination (Arrow, 1998)

Unfair treatment of ethnic minorities can result from rational actions executed by profit-maximizing actors who are confronted with the uncertainties accompanying selection decisions.

Taste-based discrimination (Becker, 2010)

Discriminatory behavior is the result of people’s unfavorable attitudes toward ethnic minorities.

Prompt templates

A photo of a <attribute> person.
A <attribute> person.
This is a <attribute> person.
Cropped face photo of a <attribute> person.

Bootstrapping Variations 

• We randomly choose two distinct values, \(x_1,x_2 \sim X\), for the chosen dimension (e.g., white and black).
• For each pair of values, we select the respective image embeddings, \(i_1(x=x_1), i_2(x=x_2)\) that are equal in all other dimensions (in this example: gender, age, smiling, lighting, and pose).
• We then compute the difference in cosine similarities between each image embedding and a text embedding \(t\), defined as \(\Delta(t, i_1, i_2) = \lvert \cos(i_1, t) - \cos(i_2, t) \rvert\).
• This process is repeated 1,000 times, generating a bootstrap distribution of \( \Delta \) values.
• This distribution describes the impact of the specific dimension on the cosine similarity of image embeddings and text embedding.
   

Heatmap of Pearson correlation coefficients of positive and negative valence dimensions of the ABC model.

How does Facial Expression impact Social Perception?

Smiling

Smiling

a photo of a person

Smiling

a photo of a person

Smiling

a photo of a person

a photo of a person

Smiling

a photo of a person

Smiling

a photo of a
liberal
person

Smiling

Belief (progressive)

Gender

Females

Males

Race

Asian 

Black

White

How does CausalFace compare to wild-collected datasets w.r.t. gender and race?

Commonly used bias-metrics

Markedness 

• Markedness (Wolfe and Caliskan, 2022)
• adapted WEAT (Caliskan et al., 2017)
• Skew@k, NDKL (Geyik et al., 2019)

✓

✓

✓

Commonly used bias-metrics