CMSC 304

Social and Ethical Issues in Information Technology

Fairness + Algorithmic Bias

Quiz Review

• authors describe a "leap of faith" when applying patterns found in historical data to predict individual outcomes, assuming that "unknown outcomes...in the life trajectory of an individual follow the patterns they have found."
• machine learning systems trained on historical data are likely to reflect "historical prejudices against certain social groups, prevailing cultural stereotypes, and existing demographic inequalities," leading the system to "replicate these very same dynamics." This means that even if the system is built with good intentions, it can still reinforce and perpetuate existing inequalities due to biases embedded in the data.

Announcements

1. Our activity on Wednesday will be online, pls bring your laptops and a charger!
2. If you haven't already, please check your email and respond to the survey

Recap: Data, Information, Knowledge, Wisdom

• New information is always slotted into current knowledge, your brain doesn’t erase every time
• Knowledge exists before data and information-- our worldview shapes them

• a worldview = implicit assumptions, perspectives, or biases that shape how data is collected, interpreted, and presented
  • Just like an individual's worldview affects how they perceive and interpret the world
• a dataset's worldview can include:
  • selecting which data to collect. This is guided by assumptions and priorities
  • deciding how to collect it. Surveys, instruments, sensor placement, cookies, etc
  • what NOT to collect. Populations of interest, generalizations, etc.

Recap: Data Worldviews

• There's no such a thing as "raw data" or "objective data"
  • "Data need to be imagined as data to exist and function as such, and the imagination of data entails an interpretative base"
    • a person selects which data is relevant
    • a person cleans that data
    • a person has to give the data context and meaning
• It's not possible to create systems that are completely independent of human thought processes
  • this makes sense, because what are we trying to do with AI in the first place? Replicate human capabilities!
• Data not only produces knowledge, it is also the product of human knowledge

When we ignore data's worldview, we get "bias"

• The term "bias" is overloaded

When we ignore data's worldview, we get "bias"

• The term "bias" is overloaded
  • in statistics it means "a systematic tendency for the true value and estimated value for a population differ"
    • common example = selection bias: sample is not chosen randomly, so estimates are inaccurate
      • we want to measure median income of a county, so we post a survey QR code at the bank and ask people to fill it out
  • in cognitive science it means a "brain shortcut" we use to make decisions. Sometimes those decisions are irrational
    • optimism bias: believe things will work out
    • status quo bias: do something because it requires less cognitive effort (hungry judges)

When we ignore data's worldview, we get "bias"

• The term "bias" is overloaded
  • in statistics it means "a systematic tendency for the true value and estimated value for a population differ"
  • in cognitive science it means a "brain shortcut" we use to make decisions. Sometimes those decisions are irrational
  • colloquially, people use the term to insinuate someone has inclination for or against some people, ideas, or things, leading to injustice
  • in AI ethics, bias means using past data to make future judgements or predictions in ways that may disproportionally affect different groups

Algorithmic Decision-making

• What is an algorithm?
  • A set of rules and procedures that leads to a decision
  • Could be machine learning, or simply control flow logic (rule-based if-then-else)
• What can algorithms do for us?
  • They improve accuracy and efficiency over human decision-making
  • Algorithms can make decisions faster and more consistently than humans
  • We can just create the model to be optimal, since it's just numbers (right?)
• How can algorithms harm us?
  • "An algorithm is only as good as ..."
    • its training data
    • its designer
  • Widespread rapid adoption can encode bias, threaten fairness, erase privacy, transparency, and due process

Burton, Emanuelle; Goldsmith, Judy; Mattei, Nicholas; Siler, Cory; Swiatek, Sara-Jo. Computing and Technology Ethics: Engaging through Science Fiction (pp. 117-118). MIT Press.

The promise of "algorithms"

• Many modern systems rely on some combination of data, algorithms, AI/ML to make decisions and manage complex processes
  • sometimes you'll see this called buzzwords "data-driven" or "AI-powered"
• Automated decision-making systems are typically meant to improve and enhance human decision-making
  • medical diagnosis
  • bail and sentencing
  • finance and lending
  • product recommendation
• The implication is that human decision-making is flawed, and automated systems can remove human bias, inefficiency, limitation for information processing, speed, etc.

Fairness in Algorithmic Decision-making

• Algorithms require formalization of what to compute
  • you need to actually write an equation of your goal i.e. an objective function​
    • What is the cost of making each decision?

• You also need to write an equation for evaluating the results for accuracy and fairness
• If we want to use algorithms to make decisions, it forces us to be precise about what we think "fairness" is and how we define it
• Let us now look at how we can define fairness

https://web.stanford.edu/class/cs182/

Fairness in Machine Learning

• Machine learning algorithms will inevitably make mistakes
• “Fairness” asks: how are those mistakes distributed across individuals and groups?
  • e.g. do certain sub-populations suffer a disproportionately high rate of errors? 

https://www.fdic.gov/analysis/cfr/consumer/2018/documents/aaron-roth.pdf

Solon Barocas and Moritz Hardt, "Fairness in Machine Learning", NeurIPS 2017

Criteria for Algorithmic Accountability

1. Does it work? (Efficiency to achieve public safety)
2. Is it fair? (Fairness)
3. Can people understand how it works? (Transparency)
4. Can people appeal its judgment? (Due Process)
5. Does it use information that an individual might reasonably expect to remain private? (Privacy)

https://web.stanford.edu/class/cs182/

Human Decision-making

Example: Ford Pinto

https://www.classiccarstodayonline.com/classic-car-print-advertisements/ford-1974-ford-pinto-ad-a/

https://www.tortmuseum.org/ford-pinto/#images-1

Human Decision-making

Example: Ford Pinto

• Since $137 M > $49.5 M, they said "meh"

• It did not go well

  • 500-900 people burned to death.

  • Ford found not guilty of homicide. Did the recall afterwards. Incurred legal fees.

Human Decision-making

Example: Ford Pinto

Expected payout for potential accidents:

• Estimated incidents: 180 burn deaths, 180 serious burn injuries, and 2,100 vehicles lost in fires annually.
• Value assigned per fatality: $200,000
• Value per injury: $67,000
• Value per vehicle lost: $700
• Using these figures, Ford calculated the total cost of settlements for burn deaths, injuries, and vehicle losses to be approximately $49.5 million.

Cost to fix the fuel tank design:

• $11 part cost per vehicle
• With 12.5 million vehicles and 1.5 million light trucks potentially affected, the total cost to fix all vehicles was calculated at approximately $137 million.

https://www.tortmuseum.org/ford-pinto/

Human Involvement in Algorithmic Decision-making

The "Loop"

Human Involvement in Algorithmic Decision-making

The "Loop"

Predictive Policing

• Several large city police departments are using "analytics" to try and predict where crimes will occur
  • send more police to those areas to get ahead of the crime
• If there are more police in an area looking for something to do, more crime will be found
  • more scrutiny by police in that area
  • leads to "overpolicing"

Weblining

• Allusion to "redlining," a practice of offering services on a selective basis (e.g. home loans or insurance)
  • redlining: "withholding of home-loan funds or insurance from neighborhoods considered poor economic risks."
  • outlawed by the Fair Housing Act of 1968, we can still see its impacts today
• These "analytics" typically use people's home address (or some other data as a proxy     ) as a basis for service availability or price
  • tends to result in services being more expensive, or unavailable to neighborhoods that are predominantly poor or have a majority of ethnic minorities
• This lets businesses discriminate against certain groups, while staying within the bounds of law that forbid making such decisions on the basis of race

Targeted Advertising

• Retailers collect customer purchasing information to predict shoppers' choices and habits, and to advertise according to these
• Advertising servers can also target users for reasons other than shopping
  • The Cambridge Analytica scandal (2016-2018)
    • Unauthorized harvesting of personal data from millions of Facebook users
    • CA used this data to create psychological profiles of voters
      • used to target individuals with personalized political advertisements and misinformation campaigns
  • Also in 2016 it was shown that Google was displaying ads for high-paying jobs to men more frequently than women
    • based on historical characteristics
    • Prediction task: who would click on the ad and be successfully hired?

Sentencing + Bail Software

• Judges in parole cases use "analytics" to determine the likelihood that an individual will re-offend
  • this is called recidivism
• Black defendants were 2x as likely to be flagged incorrectly as high-risk, whereas white defendants were 2x as likely to be incorrectly flagged as low-risk
• People tend to be un-concerned with this until it comes for them...

Hiring and Admissions

• Almost universally, algorithms completely control where job ads appear on hiring websites, social media, etc.
• The placement of these ads can reflect historic bias on perceived race, gender, economic, or educational status
  • "See? They don't click on our ads, women don't want these high-stress jobs, they want to stay at home and chill with bebe"
• Then, after using ads to "filter" which candidates apply, they also use algorithms to sort and rank resumes and application, reinforcing that bias
  • "We ranked Brad #1, since he looks like the other guys at the top, he must be headed there too!"
• This kind of thing happens for college admissions also

The problem with measures

• "race multiplier" based on the long-standing but now discredited theory that Black individuals had higher muscle mass, which supposedly increased creatinine levels.
• This assumption fails to consider individual variations within racial groups and led to overestimating kidney function in many Black patients
  • made them appear healthier, delaying the diagnosis of kidney disease, making them ineligible for the transplant waiting list
• from a study in the late 1990s, it was widely adopted into clinical practice across the U.S. and internationally. Not addressed until 2022

• Bloodwork seems like it should be objective, but there is bias there as well
• Glomerular filtration rate (GFR) is used to quantify severity of kidney disease
• For Black patients, the formula multiplied the result by a factor of 1.21, while for non-Black patients, it did not apply this multiplier:

Big Data + Bias at its worst

• In all of these examples, "big data" can reproduce existing patterns of bias and discrimination, even when such discrimination is entirely unintentional!
• Many times there is "reinforcement learning" or feedback loops that amplify this and keep people in their place

Bias Amplification

Big Data + Bias at its best

• Three main positive roles for algorithms and big data for promoting positive social change
  • Synecdoche (this is a wild word I only learned recently. It means "to use something as an example that stands in for a larger whole")
  • Diagnostics
  • Rebuttal

Big Data + Bias at its best

• Three main positive roles for algorithms and big data for promoting positive social change
  • Synecdoche (this is a wild word I only learned recently. It means "to use something as an example that stands in for a larger whole")
    • As each new example of bias is unearthed and discussed, it helps the issue gain new attention, which comes with resources and calls for change
    • Some great positive outcomes in large, automated decision-making systems to identify people suffering from drug and depression issues, and matching them with treatment options, cutting rates of suicide and overdoses
    • Bias can be used for good: In finance there have been massive decreases in fraud and credit card theft through the use of ML systems that are biased by users' individual behavioral data, able to detect anomalies
  • Diagnostics
  • Rebuttal

Big Data + Bias at its best

• Three main positive roles for algorithms and big data for promoting positive social change
  • Synecdoche (this is a wild word I only learned recently. It means "to use something as an example that stands in for a larger whole")
  • Diagnostics
    • Since systems can be tested for bias, we can use them to perform diagnosis, audit, or "debug" decisions that we have made in the past, and to highlight fairness issues in existing sociotechnical systems
  • Rebuttal

Big Data + Bias at its best

• Three main positive roles for algorithms and big data for promoting positive social change
  • Synecdoche (this is a wild word I only learned recently. It means "to use something as an example that stands in for a larger whole")
  • Diagnostics
  • Rebuttal
    • We can use large amounts of data to argue against the use of that data in a particular way
      • for example, researchers were able to convince Immigration and Customs Enforcement Agency (ICE) to NOT create a system
      • ICE wanted a system that predicts whether an immigrant will be a "positive contributing member of society"
      • Researchers were able to show that no system could perform that analysis with the proposed data

So as a developer, is it your fault?

• Cowgill et al. 2020 conducted a study to answer the question:
  • Are biases introduced by biased data or biased developers??
    • Since CS professionals are not representative of the general population that will be impacted by their system (higher socio-economic status, more education, etc.), do they therefore write biased algorithms?
    • Since data has a worldview, and historical data encodes any bias that was present at that time, is it the data that causes the bias?
  • In their study they examined over 400 AI professionals and analyzed their characteristics and code and found that there is no evidence that minority or low-implicit-bias workers generate better, less biased code.
  • They found that overwhelmingly, better data leads to less biased algorithms

Activity: Choose one w/ your table

Feedback Loops

Social media: A user interacts with content supporting a particular political view → the system recommends similar content → user continues engaging with similar content, deepening their viewpoint → system reinforces this trend by recommending even more similar content, creating an echo chamber.

Health analytics: A person in a low-income area with limited healthcare access has a poor health outcome → the model predicts high risk for future health issues, prioritizing emergency treatment over preventive care → person receives only limited treatment → health does not improve → cycle repeats.

Activity #1

2 main notions of fairness

• Individual fairness
• Group fairness

https://chrispiech.github.io/probabilityForComputerScientists/en/examples/fairness/

Example: ads that target software engineers

Remember how we can evaluate classifiers

• TPR (true positive rate): something that is good for the individual
  • SWE classifier, likely to graduate, likely to get a loan
  • also called Recall: "Of all the actual positive instances, how many did the model correctly identify as positive?" It's important in situations where it is critical to identify as many positive instances as possible (maximum coverage), even if it means allowing some false positives.
• FPR (false positive rate): something that is harmful for the individual
  • Likely to recommit a crime, likely to go bankrupt, screening for terrorists
  • "Of all the actual negative instances, how many did the model incorrectly classify as positive?" It reflects the model's tendency to produce false alarms.
• Precision: the ratio of true positive predictions to the total number of positive predictions made by the model (both true positives and false positives).
  • Precision answers the question, "Of all the instances that the model predicted as positive, how many were actually positive?" It reflects the accuracy of the positive predictions made by the model.

precision = coverage

recall = accuracy of the things you want to detect

Precision Recall Primer

Evaluating for bias: Precision

• We want to make sure our classifier is good to go, so we check the rates individually for different groups
  • we do this, because we took CMSC 304 and learned that there can be socio-economic, gender, and racial disparities when serving targeted ads

• looking good, the Precision for low- income is approximately the same as for high income
• This means that across the board, there's a ~75% probability that a person that is a SWE will be identified as a SWE, from their user data
  • now they can enjoy that ad 

Evaluating for bias: Recall

• hmm, the Recall for low income is much lower than it is for high income
• smaller recall = for lower income people who are actually software engineers, more of them are being classified as "not software engineers" and not getting served the ad
  • is this a good or bad thing?
    • maybe good if you hate ads
    • maybe bad if those ads are for high paying jobs

• We want to make sure our classifier is good to go, so we check the rates individually for different groups
  • we do this, because we took CMSC 304 and learned that there can be socio-economic, gender, and racial disparities when serving targeted ads

Precision and Recall as Probabilities

• a more concise way to write them
• the vertical line means "given", or "out of how many?"

low income

high income

low income

high income

Why might this be?

• Recall might be lower for a group if:
  • those examples are underrepresented in the training data
    • The model may not learn to recognize patterns or features associated with that group effectively.
    • As a result, the model might miss positive instances (e.g., correctly identifying members of that group), leading to lower recall
  • if the training data has different features between groups. e.g. maybe SWE's with higher income have more time to surf the web, so there is more rich data
  • there may be proxies for income status in the dataset, e.g. zip code, last name, bank cookies
• How can we fix it?

How to fix it?

• How can we fix it?
  • and how will we know it's fixed?
  • option 1: aim for statistical parity
    • ​equalize [positive thing] among the groups

• recall is the percentage of correct positives

• Y hat = predicted class
• Y = true class

How to fix it?

• How can we fix it?
  • and how will we know it's fixed?
  • option 2: aim for equal odds
    • ​equalize [correct positive thing] among the groups (recall): SWE's that are correctly identified as SWE
    • also equalize [incorrect positive thing] (false positive rate): people that are incorrectly identified as SWE

• recall is the percentage of correct positives

• Y hat = predicted class
• Y = true class

As with everything, Pros and Cons

Strict Statistical Parity / Group Fairness

Pros

• easy to understand and explain
• provides equal access to opportunities
• ensures no group is underrepresented

Cons

• may overlook individual merit, potentially leading to reverse discrimination
• may reduce overall model accuracy, particularly if base rates differ across groups
• does not address differences in error rates (e.g., false positives/negatives) across groups

As with everything, Pros and Cons

​Equal Odds

Pros

• ensures equal TPR and FPR across groups
• balances opportunities with minimizing harm
• reduces the risk of unfair treatment due to errors

Cons

• harder to understand and explain, requiring technical expertise
• can conflict with other fairness goals like overall accuracy or statistical parity.
• difficult to achieve in practice, especially with differing base rates

A caution about metrics

• Metrics can help us determine whether we are on the right track
  • but we should not have an over-reliance on metrics
  • they often don't tell the whole story
  • they can't cancel deeper issues in the data and in society, or where data doesn't exist
• Metrics are very utilitarian, and it can be important to:
  • not reduce individuals to the groups they belong to
  • recognize possible harms with the "optimization mindset"
• Some notions of fairness are mutually exclusive, so you cannot find one definition of fairness to rule them all
  • for example, equalized odds and demographic parity (Barocas et al., 2017)