Fairness in data-driven decision

Jeremias Sulam

Fairness

Supervised Learning

What is discrimination?

Protected categories:
race, sex, religion, disability status, national origin, age, ...

Domain Specific:
Credit (Equal Credit Opportunity Act)
Education (Civil Rights Act of 1964, Education Amendments of 72)

Employment (Civil Rights Act of 1964)

Housing (Fair Housing Act)

Disparate treatment
Purposeful consideration of a protected attributed with the intention of discrimination

Disparate Impact
Harm to different groups via potentially indirect mechanisms

[Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact. California Law Review.]

Supervised Learning

f(x) : \mathcal X \to \{0,1\}

(binary classification)

e.g. student admissions, credit/loans assignment, etc

  • We need design          : offer or reject a job for an individual
f(x)
f(x) = 1 \text{ if } x > \tau ~~~ \text{ (60k, say)}
  •     : estimate of number of hours an individual will work over the next 5 years

This rule disadvantages individuals who are more likely to take parental leave (women), and this should be morally irrelevant to asses a candidate.

x

The ProPublica and Northpointe debate

  • COMPAS Risk Score
  • Black defendants have a higher false positive rate
  • Black and white defendants have equal recidivism given the same score (precision)
\text{PPV} = \frac{\text{TP}}{\text{TP}+\text{FP}}
\text{FPR} = \frac{\text{FP}}{\text{TN}+\text{FP}}
TN FP
FN TP

Reoffended

Did not reoffend

"high risk"

"low risk"

Formal Fairness in ML

(Y,S,X) = (\text{label, sensitive attribute, features}),
S \in \{A,B\}
\hat{Y} = f(X,S) \approx Y

predictor:

re-paid loan?

natural origin, ...

income, profession, ... 

Formal Fairness in ML

(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y

Demographic Parity

S \in \{A,B\}
\mathbb P [ \hat{Y} = 1 | S = A] = \mathbb P [ \hat{Y} = 1 | S=B]

prediction should not be correlated with the protected attribute

imagine \(\hat Y = \text{high risk of breast cancer}, S = \text{biological sex} \)

Equal Opportunity

\mathbb P [ \hat{Y} = 1 | S=A, Y = 1 ] = \mathbb P [ \hat{Y} = 1 | S=B, Y = 1 ]

TPR should be equal for both groups

Formal Fairness in ML

(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y
S \in \{A,B\}

Formal Fairness in ML

Equalized Odds

\mathbb P [ \hat{Y} = 1 | S=A, Y = y_i ] = \mathbb P [ \hat{Y} = 1 | S=B, Y = y_i ],

TPR and FPR should be equal for both groups

y_i = 1, 0
(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y
S \in \{A,B\}

Formal Fairness in ML

Formal Fairness in ML

S= 0
S = 1

TPR

FPR

0

1

0.5

0.5

1

0

0.25

0.25

0.75

0.75

\text{FPR}_{A=0} = \mathbb P [\hat Y = 1 | Y = 0, {\color{red}S=0}]
\text{TPR}_{A=1} = \mathbb P [\hat Y = 1 | Y = 0, {\color{blue}S=1}]

Formal Fairness in ML

A = 0
A = 1

TPR

FPR

0

1

0.5

0.5

1

0

0.25

0.25

0.75

0.75

\text{FPR}_{A=0} = \mathbb P [\hat Y = 1 | Y = 0, {\color{red}S=0}]
\text{TPR}_{A=1} = \mathbb P [\hat Y = 1 | Y = 0, {\color{blue}S=1}]

Pneumonia

Clear

95% accurate

Estimating and Controlling for Fairness without sensitive attributes

Does your model achieve a \(\Delta_{\text{TPR}}\) of at most (say) 6% ?

Maximum TPR
discrepancy

True TPR
discrepancy

Estimating and Controlling for Fairness without sensitive attributes

Theorem: max TPR will never be greater than this

Questions

1. In predicting recidivism, should black and white subjects have same FPR or the same PPV?

Who's right: defendants or judges?

Questions

2. In controlling for fairness when we don't have the sensitive attribute, we must predict it.

Should we?

Questions

3. In controlling for fairness, we must use the sensitive attribute to modify the predictions.

Many current legislation says
otherwise.

Who's right?

Questions

2. In controlling for fairness, we must use the sensitive attribute to modify the predictions.

Many current legislation says
otherwise.

Who's right?

[https://jamessmurphy.com/2026/01/07/the-2025-post-sffa-enrollment-tracker/]

Jeremias Sulam   

jsulam.github.io