AI, Society, and Human Behavior

Research Methods in Context

Carina I Hausladen

Topics

Four cutting-edge topics at the frontier of computation social science:

1. Measuring Bias in AI
2. Social Choice for LLM Alignment
3. Clustering Multidimensional Time Series — Modeling Human Behavior
4. Modeling Social Dilemmas through Reinforcement Learning

• Research Skills

  • Design your own research question

  • Replicate, extend, or reinterpret topics we discuss

• Applied Methods

  • Analyze real data using computational tools

  • Code in teams to explore your question

  • Build a GitHub repository for open, replicable research

• Communication & Impact

  • Write a short research-style paper

  • Present your insights to others

  • Discussion & active participation

Skills

Activities & Assessment

• 10:00 – 11:30:
  • Carina introducing topics and methods
• 11:30 – 12:30:
  • Lunch (together)
• 12:30 – 14:00:
  • 30 minutes discussants
  • 30 min explaining of core concepts to each other
  • 30 min coding

1. Reading Response

1. Reading Response

• Your response should answer the following:
  1. What is the core idea or contribution?
  2. What questions would you like to ask in class?
  3. What parts of the paper are interesting to you and why?
  4. How would you replicate or extend the paper?
      
• These responses are not graded.
• Responses are contributed via Overleaf.

• Serve as a discussant for one paper (only once!)

• Probably in pairs of two

• Deliver a brief (~7–10 min) presentation, focusing on:

  • Summarize the core idea of the paper

  • Does it introduce an interesting dataset we could utilize?

  • Is there an analysis worth replicating? How could this work be extended*?

    • *who did recently cite this paper?

  • Encourage discussion with your classmates

• Graded (20%)

• Deadline: Thursdays, 10 PM

2. Discussant Role

3. Group Project

• Group Project, delivered as
  • presentation (30%)
  • paper (50%)
• The paper should have around 8 pages and 4,000-8,000 words, and should be structured like a paper.
  • You should include a 'contributions' section outlining what group member did what.
• You should link a Github repo with the code you developed.

Code Clinic

• Checking analysis choices; assessing whether additional statistical tests are needed.
• Do the figures make the point?
• Does your GitHub repository support replication?

Writing Clinic

• Good writing
  • specifically focusing on abstract, figure captions, title
• Good presentations: what makes a talk effective

Presentation and Paper

• Writing is thinking
  • Ideally, the core of your paper is in a good shape before the presentation
  • When do you want to hand in your final paper? 
• Your presentation should also include a short introduction to your GitHub repository

1. Ethics of AI

January 9

Plan for today

First Session

• 40'
  • 15' Introduction
  • 25' Defining Bias
    
    
    —5' break—
     
• 45'
  • Bias Metrics  (JN Tutorials)
    • 15' WEAT 
    • 15' Probability Based
    • 15' Generated Text

Second Session

• 40'
  • Bailey 2022
  • Bai 2025
    
    
    —5' break—
     
• 40' 
  • Khan 2025
  • Hausladen 2025

A Career Track

📚 Academia

• Bias & fairness is a core research area

• Survey papers regularly reach thousands of citations
  (e.g. Mehrabi et al. 2019 >8,000 citations)

• Dedicated top-tier venue: ACM Conference on Fairness, Accountability, and Transparency (FAccT)

• Strong presence at NeurIPS, ICML, ICLR, ACL, EMNLP

• Interdisciplinary work = high visibility + funding relevance

🏭 Industry

• Major companies run dedicated fairness teams

  • Apple, Google, Meta, Microsoft, IBM, ...

• Common job titles:

  • Responsible AI Scientist

  • Fairness / Bias Engineer

  • Algorithmic Auditor

  • Trustworthy ML Researcher

• Regulation (EU AI Act, audits, compliance) → growing demand

Are Emily and Greg
More Employable than
Lakisha and Jamal?

 Bertrand & Mullainathan (2003)

Leon Yin et al. (2024)

https://www.linkedin.com/pulse/unveiling-hidden-bias-lessons-from-derek-mobley-v-workday-bareket-v7off/

(2024)

https://business.linkedin.com/talent-solutions/hiring-assistant?ss=1

Why do you care about fairness and bias? 

Defining Bias for LLMs

1. Fairness Definitions
2. Social Biases
3. Where Bias Enters the LLM Lifecycle
4. Biases in NLP Tasks
5. Fairness Desiderata

* much of following slide content is based on  "Bias and Fairness in Large Language Models: A Survey"

1. Fairness Definitions

2. Social Biases

3. Where Bias Enters the LLM Lifecycle

4. Biases in NLP Tasks

5. Fairness Desiderata

5' break

Bias Metrics

1. Embedding Based
2. Probability Based
3. Generated Text

1. Embedding Based Metrics

man

work
salary

man

home
family

women

work
salary

women

home
family

2. Probability Based Metrics I

Log Probability Bias Score
(LPBS)

$$LPBS = \log\left(\frac{P(\text{she}\mid context)}{P(\text{she}\mid prior)}\right) - \log\left(\frac{P(\text{he}\mid context)}{P(\text{he}\mid prior)}\right)$$

2. Probability Based Metrics II

1. mask one word at a time
2. calculate e.g. P('she' | context)
3. calculate log(P)
4. sum all log probabilities

3. Generated Text Based Metrics

It's your turn!

1_metrics_weat.ipynb

 

github.com/carinahausladen/konstanz-ai-behavior-2026

Papers

• cosine similarity between static word embeddings (fasttext / glove)

  • embedding-based

• WEAT

  • embedding-based

• LLM Word Association Test (LLM-WAT)

  • generated text-based → distribution

• LLM Relative Decision Test (LLM-RDT)

  • generated text-based → distribution

• WEAT

  • embedding-based

• coreference confidence

  • probability-based

• coreference confidence disparity

  • probability-based

• accuracy

  • probability-based

January 16

Plan for today

First Session

• 15' Simpson's Paradox
• 40' Datasets
  • 20' Introduction
  • 20' JN
    
    
    —5' break—
     
• 30'
  • Social Perception of Faces

Second Session

• 40'
  • Conizer 2024
  • Qui 2024
    
    
    —5' break—
     
• 20' 
  • Kirk 2024
• 20'
  • research ideas for the PRISM dataset

Simpson's Paradox in AI Fairness

In the 1970s, UC Berkeley was sued for alleged gender bias in graduate admissions.

• Aggregate view: Men were admitted at 44%, women at 35%. Looked biased.
• Department-level view: Within most departments, women had higher admission rates than men.

What is Simpson’s Paradox?

Simpson’s Paradox happens when a trend seen in aggregated data reverses or disappears when the data is broken into groups.

Datasets for Bias Evaluation

Counterfactual Inputs

• Masked Tokens (WinoBias)
• Unmasked Sentences (CrowS-Pairs)

Prompts

• Sentence Completions (BOLD)
• Question-Answering (BBQ)

WinoBias

Zhao et al. (2018)

CrowS-Pairs

Bias in Open-Ended Language Generation Dataset (BOLD)

Dhamala et al. (2021)

Bias in Open-Ended Language Generation Dataset (BOLD)

Bias Benchmark for QA (BBQ)

Parrish et al. (2022)

Bias Benchmark for QA (BBQ)

It's your turn!

2_metrics_maskedtoken.ipynb

3_metrics_pll.ipynb

4_metrics_generatedtext.ipynb

5_datasets

github.com/carinahausladen/konstanz-ai-behavior-2026

Papers

Ideas for the PRISM dataset

How does the choice of aggregation rule reshape who benefits from alignment?

• Test multiple aggregation rules:
  Utilitarian (mean), Thiele-style proportional scoring, Rawlsian (floor-maximizing), inequality-adjusted welfare, etc.

• For each rule, select the top-$K$ models.

• Compute user welfare under access to the selected models (e.g., random-choice lower bound; best-choice upper bound).

• Compare welfare across socio-demographic groups:
  Gender, ethnicity, age

• Report outcomes, e.g.

  • Mean welfare

  • Bottom-decile welfare (10th percentile / bottom 10%)

  • Welfare gaps between groups (e.g., max–min group mean; or pairwise differences)

Appendix

Dutch Childcare Benefits Scandal

• What happened

  • ~26,000–35,000 families wrongly accused of childcare-benefit fraud

  • Parents forced to repay tens of thousands of euros

    • Many families fell into severe poverty;

    • children were removed from some families as a downstream consequence

• Where the bias came from

  • Fraud risk-scoring system used nationality/dual nationality as risk indicators

  • Zero-tolerance rule:

    • any suspected irregularity ⇒ 100% benefit clawback

    • Minor administrative errors treated as intentional fraud

  • Caseworkers did not independently evaluate cases.
    They treated the system’s risk flags as ground truth, not as advice.

3. Where Bias Enters the LLM Lifecycle

2. Social Biases

Topics

• No advanced math or ML required

  • Focus on intuition, discussion, and conceptual understanding.

• Choose what interests you

  • You can catch up on background knowledge as needed.

  • Work in groups to support and complement each other’s skills.

• Recommended:

  • Interest in machine learning, social science, or AI ethics

  • Basic probability and statistics

  • Introductory Python programming

Prerequisites

1. Measuring Bias in AI

• Where Bias in AI Appears

  • Hiring

  • Predictive policing

  • Ad targeting

• Sources of Bias

  • Human bias & feedback loops

  • Sample imbalance / unreliable data

  • Model & deployment effects

• ​Fairness Criteria

• Bias and Embeddings

  • ​Word embeddings encode stereotypes

  • Embedding geometry

• Causality

  • Simpson’s Paradox

  • Causal inference

• ​Case Study

Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona

Social perception of faces in a vision-language model

• Preference elicitation
  • Ordinal vs cardinal preferences
  • Methods of elicitation
• From individual to collective choice
  • Fairness and proportionality principles
  • Key properties: monotonicity ...
• Committee elections

• Participatory budgeting (PB)
  • PB as generalization of committee elections
  • Aggregation methods for PB: proportional and cost-aware
• Human-centered LLMs
  • Learning from human preferences (RLHF)
  • Pluralistic alignment

2. Social Choice and
LLM Alignment

Guest Lecture

3. Clustering Multidimensional Time Series

• Behavioral data as multidimensional time series
• Distance Metrics
  • Local
    • e.g. Euclidean Distance
  • Global
    • Dynamic Time Warping (DTW)

• Clustering Methods
  • Hierarchical clustering:
  • PAM (Partitioning Around Medoids)
  • DBSCAN/HDBSCAN: density-based
• Evaluation & Validation
  • Internal indices 
  • External validation 

4. Modeling Social Dilemmas

• Social Dilemma Games
  • Prisoner’s Dilemma, Stag Hunt, Public Goods Game.
  • Emergent dynamics.
• Reinforcement Learning
  • Agents learn from rewards and punishments over time.
• Markov Decision Processes
  • Sequential decision-making under uncertainty.

• Q-Learning
  • learning state–action values through trial and error
  • latest literature on social dilemmas
• Inverse Reinforcement Learning
  • Infer the hidden reward function.
  • Useful in social science: recover fairness concerns, reciprocity, etc.

Identifying Latent Intentions
via
Inverse Reinforcement Learning
in
Repeated Public Good Games

Carina I Hausladen, Marcel H Schubert, Christoph Engel

MAX PLANCK INSTITUTE
FOR RESEARCH ON COLLECTIVE GOODS