Bias: prejudice, usually in a way considered to be unfair

(mathematical) systematic distortion of a statistical result

Prejudice: preconceived opinion

(legal) harm or injury resulting from some action or judgment

Algorithm: a set of rules to be followed, especially by a computer

Promises of algorithmic screening

• explicit criteria
• equally applied
• auditable

"I usually look for candidates who will be a good fit"

Promises of algorithmic data-driven screening

• explicit criteria "superhuman" accuracy
• equally applied
• auditable

Machine learning extracts statistical patterns from employment records to automatically generate accurate and optimal classifications

Federal laws make it illegal to discriminate on the basis of  race, color, religion, sex, national origin, pregnancy, disability, age, and genetic information.

Important quantities:

features $X$ and label $Y$

(e.g. resume and employment outcome)

classifier $c(X) = \hat Y$

(e.g. interview invitation)

Accuracy
$\mathbb P( \hat Y = Y)$ = ________

Positive rate
$\mathbb P( \hat Y = 1)$ = ________

False positive rate
$\mathbb P( \hat Y = 1\mid Y = 0)$ = ________

False negative rate

$\mathbb P( \hat Y = 0\mid Y = 1)$ = ________

Positive predictive value
$\mathbb P( Y = 1\mid\hat Y = 1)$ = ________

Negative predictive value
$\mathbb P( Y = 0\mid\hat Y = 0)$ = ________

$c(X)$

$3/4$

$9/20$

$1/5$

$3/10$

$7/9$

$8/11$

individual with protected attribute $A$

(e.g. race or gender)

Independence: decision does not depend on $A$

$\hat Y \perp A$

e.g. applicants are accepted at equal rates across gender

Separation: given outcome, decision does not depend on $A$

$\hat Y \perp A~\mid~Y$

e.g. qualified applicants are accepted at equal rates across gender

Sufficiency: given decision, outcome does not depend on $A$

$Y \perp A~\mid~\hat Y$

e.g. accepted applicants are qualified at equal rates across gender

“In comparison with whites, a slightly lower percentage of blacks were ‘Labeled Higher Risk, But Didn’t Re-Offend.’ [...] A slightly higher percentage of blacks were ‘Labeled Lower Risk, Yet Did Re-Offend.”’

$\mathbb P(\hat Y = 1\mid Y=0, A=\text{Black})> \mathbb P(\hat Y = 1\mid Y=0, A=\text{White})$

$\mathbb P(\hat Y = 0\mid Y=1, A=\text{Black})< \mathbb P(\hat Y = 0\mid Y=1, A=\text{White})$

$\mathbb P(Y = 0\mid \hat Y=1, A=\text{Black})\approx  \mathbb P( Y = 0\mid \hat Y=1, A=\text{White})$

$\mathbb P(Y = 1\mid \hat Y=0, A=\text{Black})\approx  \mathbb P( Y = 1\mid \hat Y=0, A=\text{White})$

Attempt #1: Remove protected attribute $A$ from features

Interactive post-processing example

Pre-existing Bias: exists independently, usually prior to the creation of the system, with roots in social institutions, practices, attitudes

Technical Bias: arises from technical constraints and considerations; limitations of formalisms and quantification of the qualitative

 Emergent Bias: arises in context of use  as a result of changing societal knowledge, population, or cultural values

Classic taxonomy by Friedman & Nissenbaum (1996)

training data

$(X_i, Y_i)$

measurement

action

Existing inequalities can manifest as pre-existing bias

Further technical bias results from formulating learning task and training the model: optimization bias

e.g. optimizing for average accuracy will prioritize majority groups

rom-com 80% of the time

horror 20% of the time

optimize for probable click

recommend rom-com 100% of the time

Small Is Beautiful: Economics as if People Mattered, E.F. Schumacher:

1. Full predictability exists only in the absence of human freedom
2. Relative predictability exists with regard to the behavior pattern of very large numbers of people doing 'normal' things
3. Relatively full predictability exists with regard to human actions controlled by a plan which eliminates freedom
4. Individual decisions by individuals are in principle unpredictable

Emergent bias results from real-world dynamics, including those induced by the decisions themselves

e.g. Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."

PredPol analyzed by Lum & Isaac (2016)

recorded drug arrests

police deployment

estimated actual drug use

image cropping

facial recognition

information retrieval

generative models