Fairness in data-driven decision
Jeremias Sulam
Fairness
Supervised Learning
What is discrimination?
Protected categories:
race, sex, religion, disability status, national origin, age, ...
Domain Specific:
Credit (Equal Credit Opportunity Act)
Education (Civil Rights Act of 1964, Education Amendments of 72)
Employment (Civil Rights Act of 1964)
Housing (Fair Housing Act)
Disparate treatment
Purposeful consideration of a protected attributed with the intention of discrimination
Disparate Impact
Harm to different groups via potentially indirect mechanisms
[Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact. California Law Review.]
Supervised Learning
f(x) : \mathcal X \to \{0,1\}
(binary classification)
e.g. student admissions, credit/loans assignment, etc
- We need design : offer or reject a job for an individual
f(x)
f(x) = 1 \text{ if } x > \tau ~~~ \text{ (60k, say)}
- : estimate of number of hours an individual will work over the next 5 years
This rule disadvantages individuals who are more likely to take parental leave (women), and this should be morally irrelevant to asses a candidate.
x
The ProPublica and Northpointe debate
- COMPAS Risk Score
- Black defendants have a higher false positive rate
- Black and white defendants have equal recidivism given the same score (precision)
\text{PPV} = \frac{\text{TP}}{\text{TP}+\text{FP}}
\text{FPR} = \frac{\text{FP}}{\text{TN}+\text{FP}}
| TN | FP |
| FN | TP |
Reoffended
Did not reoffend
"high risk"
"low risk"
Formal Fairness in ML
(Y,S,X) = (\text{label, sensitive attribute, features}),
S \in \{A,B\}
\hat{Y} = f(X,S) \approx Y
predictor:
re-paid loan?
natural origin, ...
income, profession, ...
Formal Fairness in ML
(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y
Demographic Parity
S \in \{A,B\}
\mathbb P [ \hat{Y} = 1 | S = A] = \mathbb P [ \hat{Y} = 1 | S=B]
prediction should not be correlated with the protected attribute
imagine \(\hat Y = \text{high risk of breast cancer}, S = \text{biological sex} \)
Equal Opportunity
\mathbb P [ \hat{Y} = 1 | S=A, Y = 1 ] = \mathbb P [ \hat{Y} = 1 | S=B, Y = 1 ]
TPR should be equal for both groups
Formal Fairness in ML
(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y
S \in \{A,B\}
Formal Fairness in ML
Equalized Odds
\mathbb P [ \hat{Y} = 1 | S=A, Y = y_i ] = \mathbb P [ \hat{Y} = 1 | S=B, Y = y_i ],
TPR and FPR should be equal for both groups
y_i = 1, 0
(Y,S,X) = (\text{label, sensitive attribute, features}),
\hat{Y} = f(X,S) \approx Y
S \in \{A,B\}
Formal Fairness in ML
Formal Fairness in ML
S= 0
S = 1
TPR
FPR
0
1
0.5
0.5
1
0
0.25
0.25
0.75
0.75
\text{FPR}_{A=0} = \mathbb P [\hat Y = 1 | Y = 0, {\color{red}S=0}]
\text{TPR}_{A=1} = \mathbb P [\hat Y = 1 | Y = 0, {\color{blue}S=1}]
Formal Fairness in ML
A = 0
A = 1
TPR
FPR
0
1
0.5
0.5
1
0
0.25
0.25
0.75
0.75
\text{FPR}_{A=0} = \mathbb P [\hat Y = 1 | Y = 0, {\color{red}S=0}]
\text{TPR}_{A=1} = \mathbb P [\hat Y = 1 | Y = 0, {\color{blue}S=1}]
Pneumonia
Clear
95% accurate
Estimating and Controlling for Fairness without sensitive attributes
Does your model achieve a \(\Delta_{\text{TPR}}\) of at most (say) 6% ?
Maximum TPR
discrepancy
True TPR
discrepancy
Estimating and Controlling for Fairness without sensitive attributes
Theorem: max TPR will never be greater than this
Questions
1. In predicting recidivism, should black and white subjects have same FPR or the same PPV?
Who's right: defendants or judges?
Questions
2. In controlling for fairness when we don't have the sensitive attribute, we must predict it.
Should we?
Questions
3. In controlling for fairness, we must use the sensitive attribute to modify the predictions.
Many current legislation says
otherwise.
Who's right?
Questions
2. In controlling for fairness, we must use the sensitive attribute to modify the predictions.
Many current legislation says
otherwise.
Who's right?
[https://jamessmurphy.com/2026/01/07/the-2025-post-sffa-enrollment-tracker/]
Jeremias Sulam
jsulam.github.io
Bias in ML
By Jeremias Sulam
Bias in ML
Bias in data-driven decision BME
