Alignment as Social Choice

Welfare Effects of Four Major AI Developers’ Alignment Frameworks in Large-Scale Human Preference Data

Carina I Hausladen

  • RLHF must aggregate heterogeneous feedback into one model
  • This is precisely the problem social choice theory has studied for decades

Is there large scale human alignment-preference data?

Strategic Placement

Can we read alignment docs through a social choice lens?

Can we make the welfare consequences more explicit / tangible?

Literature

Literature

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

.

.

.

.

.

.

Initial Language Model

 

 

 

 

 

.

.

.

.

.

.

Reward Preference Model

 

 

 

 

 

.

.

.

.

.

.

Tuned Language Model

 

 

 

 

 

Reinforcement Learning Update

Reinforcement Learning from Human Feedback

Bradley-Terry model

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

Literature

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

  • conclude the welfare-correct object is a distribution over several winners
    (as opposed to single-winner aggregation)
  • develop aggregators/training rules for heterogeneous populations
  • the magnitude of the flattening their methods correct for has not been quantified on human alignment-preference data.
    • Kim et al. use movie rating data
    • Gölz et al. is axiomatic

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

  • analyze ethics guidelines & principles
    published by AI actors
    • Jobin et al.: governments / NGOs / academia
    • Hagendorff: AI labs
  • in aggregate / by sector
    • <> company-by-company welfare profile
  • code principles
    • <> explicit social welfare functionals
  • corpora predate the pluralistic alignment era

Methods

  • Three conversation types: unguided, values-guided, controversy-guided
  • 21 different models
  • Up to 4 models respond per prompt
  • 1,396 unique evaluators

Kirk et al. (2024)

The PRISM Alignment dataset

Ask, request, or talk to the model about anything. It is up to you! 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Perfect

100

Terrible

0

Bradley-Terry

  • Each user rates only a subset of models
  • Turn the ratings into a network of pairwise comparisons
    • model i beats j when rated higher
  • Fit a latent strength \( \beta_i \) per model
  • The odds one model beats another depend only on the gap:
    \( \text{odds}(i \succ j) = e^{\beta_i - \beta_j} \)
  • RLHF objective: the reward model is trained by maximizing this BT likelihood on human preference pairs

Results

Size-vs-coverage curve

x

x

x

user 100

x

x

x

mean-centered within user

27.5%

63.3%

81.8%

  1. cardinal prefs → per-user BT scores
  2. top-3 truncation →k-approval ballot (drop order)
  3. seriation (user-model-matrix reordering)
  4. coverage under the Chamberlin–Courant objective

Size-vs-coverage curve

A single-model solution is insufficient

  • Kim et al.: proportionality and putting all mass on one winner are mutually exclusive.
  • their preference learning framework aligns aggregate opinions proportionally with the true population distribution
  • their policy is a probability distribution over the alternatives
    • deployment-layer translation: a menu of models

A single-model solution is insufficient

  • distortion measures how much welfare is lost when alignment fits a single BT reward via RLHF
    • collapses users into one 'mythical user'
  • this paper: empirical lens based on real preference data

Results

Performance across substantive domains

Ask, request, or talk to the model about anything. It is up to you! 

.

.

.

.

.

.

Terrible

Perfect

100

0

Performance varies across substantive domains

  • assume "a single latent notion of LLM quality"
  • acknowledge that LLMs serve "diverse … tasks, each with their own goals"
  • defer task-heterogeneity to future work
  • Chatbot Arena ranks models by Elo
    • capability columns (coding, vision, math)
    • every column is still a single ranking; none of them is a committee
  • Theory (Gölz et al., Kim et al.): the welfare-correct object is a committee, not a winner
  • coverage–committee-size tradeoff
    • ​how many models, covering how many users, on which topics

Implications for AI Leaderboards

Do company alignment policy documents reflect these insights? 

Results

Preliminary

Alignment Documents

  • How would moral philosophy / welfare economics / social choice read those documents?
     
  • Can we find a set of prevalent normative lenses ...
     
  • ... that also can be operationalized in the PRISM data?
  • not exhaustive
  • broad normative traditions that recur across moral philosophy, welfare economics, and social choice theory

Explicit language about maximizing aggregate welfare?

Avoiding catastrophic harms, protecting vulnerable groups?

Reducing unequal treatment across (demographic) groups?

Rights-Based: Are certain harms treated as non-negotiable prohibitions?

Four Normative Lenses

Four Normative Lenses

applied to the PRISM dataset

Coded concepts are inherently contested, so a single definitive coding is not achievable
(Mittelstadt 2019)

Alignment documents

  • 4 AI labs

  • 8 documents (2 per lab)

    • ~139k words, ~295 pages overall

    • 2022–2026  

    • type

      • 5 arXiv papers

      • 3 RSP, Model Spec, RAI Standard / policy/specs

"developing ASL-3 Deployment Safeguards to mitigate catastrophic risks"

"prioritize this sort of safety
even above ethics"

A catastrophe threshold gates
the ranking

Safety is lexically prior to
helpfulness

"being more careful in
consumer contexts . . . due to ... vulnerable
people"

Priority to the worse-off; a
concave transform (gains to
low-utility users count more)

"similar quality of service for identified demographic groups, including marginalized groups"

Welfare is evaluated per
demographic group, not
pooled over individuals.

"target minimum performance
level for all groups"

score the
model by its weakest group

"target performance
difference between groups"

admissible only if the spread
across groups is small enough

When two root-level principles conflict, the model should default to inaction.

Safety floor as feasibility constraint

"synthesizes peoples' values in a way that achieves broad consensus amongst many groups."

Among feasible models,
aggregate by averaging
preferences (a utilitarian
objective).

Mixture of Judges: We define the feasible region … that satisfies all constraints as Σ = Σ1 ∩ … ∩ ΣM

the floor is an intersection
of mandatory judges

maximize the likelihood of generating outputs that adhere to all constraints and achieve high reward

utilitarian maxim among survivors

no single winner dominates on the three welfare criteria

A principled welfare rule can choose a rarely-competitive model:

  • Chat-Bison reaches a topic's top-3 only 3/22
  • only on low-stakes topics (Cooking, Reality TV,  Weather)

Discussion

 

  • quantify tradeoff between committee-size and coverage
    • multiple models to cover preferences of a substantive part of the population 
  • model preferences vary across substantive domains
  • AI developers normative fairness choices
    • no single winner dominates
    • pick a model performing poorly on substantive domains

Summary

Limitations

 

  • four lenses determine which aspects get highlighted
  • SWF-translation is a subjective decision
  • two documents per lab is not exhaustive
  • adding a second human-preference dataset 

carina.hausladen@uni-konstanz.de

slides.com/carinah

github.com/carinahausladen

Appendix

Bradley-Terry

Performance varies across substantive domains

  • Command does not cover top three in several Political/Civic domains
  • Importantly, the alignent documents reviewed do not specify concrete substantive domains for which performance should be optimized or guaranteed
  • A more explicitly democratic approach—such as PRISM, where users reveal what matters to them by selecting discussion topics—could provide an empirical basis for identifying domains that warrant additional fine-tuning.

The corpus

  • 4 AI labs

  • 8 documents (2 per lab)

    • ~139k words, ~295 pages overall

    • 2022–2026  

    • type

      • 5 arXiv papers

      • 3 RSP, Model Spec, RAI Standard / policy/specs

Anthropic

  • Constitutional AI: Harmlessness from AI Feedback | 2022 (research)
  • Responsible Scaling Policy (v3.3) | 2026 (public)

OpenAI

  • Training LMs to Follow Instructions w/ Human Feedback| 2022 (research)
  • OpenAI Model Spec | 2025 (public)

Meta

  • Llama Guard: LLM-based Input-Output Safeguard | 2023 (research)
  • The Perfect Blend: Redefining RLHF with Mixture of Judges | 2024 (research)

 Microsoft

  • Controllable Safety Alignment | 2024 (research)
  • Microsoft Responsible AI Standard | 2022 (public)
  • "developing ASL-3 Deployment Safeguards to mitigate catastrophic risks"
  • "prioritize this sort of safety even above ethics"
  • "being more careful in consumer contexts … due to the potential for vulnerable people"

Threshold is lexically prior to helpfulness

Concave prioritarian weighting

  • "When two root-level principles conflict, the model should default to inaction."
  • "synthesizes peoples' values in a way that achieves broad consensus amongst many groups."

Safety floor as feasibility constraint

  • "similar quality of service for identified demographic groups, including marginalized groups"
  • "target minimum performance level for all groups"
  • "target maximum (absolute or relative) performance difference between groups"

the worst-off demographic group

Outlook

  • We have shown the structure of the heterogeneity and the cost of a single reward.
    • The data and the coverage measure can serve as a common testbed: existing aggregators (Kim 2025; Gölz 2025) can be fit on this data and compared on our diagnostic plot
      • each method shown by how much of the population it actually serves.

 

Outlook–MoE

Mixture of Experts in Large Language Models,
Zhang et al. (2025)

  • router picks a number of experts
    • how many experts
    • and route on what
    • --> set to minimize loss
  • Our qualitative findings point to:
    • how many distinct regimes a heterogeneous population actually has
    • that the right expert is topic-conditional — which our per-topic results show directly.
  • RLHF must aggregate heterogeneous feedback into one model
    •   Human preferences are collected, then aggregated to fine-tune a single model
    •   Feedback is inconsistent, irrational, and heterogeneous across people
    •   This is precisely the problem social choice theory has studied for decades
  • No aggregation is neutral
    •   Any rule for combining preferences embeds value judgments (Arrow)
    •   RLHF silently fixes one rule — Bradley–Terry — and calls it technical
    •   "RLHF is a voting rule in disguise": the choice is normative, not engineering
  • So every aligned model already implements some SWF
    •   Whether or not developers say so, a welfare function is being applied
  •  The implicit rule becomes readable in company documents
    • Frontier developers describe alignment in CS / engineering terms
    • But the same texts state whose preferences count, how they combine, which constraints are lexicographically prior
    • we read this through the vocabulary of welfare economics
  • Different implied SWFs → different winners on real data
    • We extract each company's implied SWF and apply it to PRISM
    • Aggregation happens at the model-selection layer, not the RLHF loss
    • Does rule choice change the outcome? → yes: three distinct winning models

Scope

  •   - We study public commitments — two flagship alignment/safety documents per company — not internal practice or actual RLHF pipelines
  •   - We map texts to stylized SWF archetypes — representative welfare-function classes, not precise or audited specifications
  •   - The mapping is interpretive; correspondence is not one-to-one, and alternative formalizations are plausible
  •   - The claim is illustrative: stated commitments can be translated into distinct aggregation rules, and the rule choice is consequential on real human data
  •   - This is step one of a broader agenda — finer-grained coding, more companies, more datasets, and eventually linking stated commitments to observed model behavior --> I am looking for interested collaborator :) 

Democratic Alignment and RLHF

  • As AI becomes a ubiquitous phenomenon in society, it is crucial to specifically design these technologies to support collective intelligence.
  • This requires aligning AI systems with principles of plurality and diversity.
    • Democratic alignment
  • One widely used method for aligning AI systems with human values is RLHF
    • human preferences are collected and then aggregated to fine-tune models; the goal is to guide them more socially aligned behavior.
  • RLHF and Constitutional AI face exactly the problem social choice theory has studied for decades — aggregating inconsistent, irrational, and heterogeneous human feedback into a single collective decision.
  • RLHF is a voting rule in disguise

Does rule choice change the outcome when aggregating alignment preferences?

Which rules should we pick?

Lazar, Seth. "Governing the algorithmic city." Philosophy & Public Affairs 53.2 (2025): 102-168.

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/papa.12279

Welfare trade-offs across alignment criteria

  • These companies' philosophies, as operationalized in my code, all point to command as the best choice.
  • Not because command has the highest median specifically, but because it scores best under each company’s own rule.

Pluralistic Alignment and
Preference Aggregation

  • Once human preferences are collected, a crucial question arises: How should these preferences be aggregated?
    • Should some groups have more influence than others?
    • If so, which ones and based on what criteria?
    • These concerns fall under the broader framework of pluralistic alignment.
    • In fact, the choice of whom to ask and how to aggregate their responses is interconnected: different aggregation strategies may inherently privilege certain groups over others, shaping the outcomes of AI alignment in subtle yet powerful ways.
  • Frontier LLM developers describe their alignment approaches in a variety of ways.
  • These descriptions are typically framed in CS / engineering terms.
  • However, they can also be read as informal statements about whose preferences matter, how they are aggregated, and which constraints are lexicographically prior to others.  
  • Social choice theory offers a rich vocabulary for talking about such questions.  

Company Alignment Philosophies

  • pick SWFs that are grounded in company alignment documents 
  • use them to aggregate large scale human alignment preference data
  • see how outcomes change

Alignment documents --> SWFs

  • We fix five normative lenses standard in social choice
    • utilitarian, prioritarian/catastrophe-averse, egalitarian, rights/constraints, democratic
  • Two independent reads per company.
    • ⚙️  Method read → the form. What the alignment mechanism does fixes the aggregation form (mean / lexicographic / config-family).
    • 🔍 Lens coding → the content. Blind dual-coding of the docs against the five lenses fixes which value.
  • Validation. Pull the verbatim evidence quotes, re-code them blind (2 passes by Claude Opus 4.8 + adjudication), report reliability → κ = 0.95

Aligment Intuition

  • Make sure nothing goes catastrophically wrong,
    then protect the worst-off.
    ​
  • Make the average user
    happy,
    within safety guardrails.
    ​
  • Learn what humans want on average. Optimize that.
     
  • Don't just look at the average,
    look at the
    worst-off group.
     
  • Different contexts get different rules. Be robust across all of them.

SWF Mapping

  • Minimize share below harm threshold, then maximize \( \sum_i f(u_i) \), \( f' > 0 \), \( f'' < 0 \).

  • Constrained utilitarianism with safety floor,
    maximize \( \frac{1}{|I_m|} \sum_{i \in I_m} u_i(m) \).

  • Plain utilitarianism:
    maximize \( \frac{1}{|I_m|} \sum_{i \in I_m} u_i(m) \).

  • Group maximin: \( W(m) = \min_g U_g(m) \) over
    demographic groups \( g \).

  • Config-robust: over configurations \( \theta \in \Theta \), use \( \min_{\theta \in \Theta} W(m \mid \theta) \) or its average.

Terrible

Perfect

100

0

SWF Mapping

  • Minimize share below harm threshold, then maximize \( \sum_i f(u_i) \), \( f' > 0 \), \( f'' < 0 \).
     

  • Constrained utilitarianism with safety floor,
    maximize \( \frac{1}{|I_m|} \sum_{i \in I_m} u_i(m) \).

  • Plain utilitarianism:
    maximize \( \frac{1}{|I_m|} \sum_{i \in I_m} u_i(m) \).
     

  • Group maximin: \( W(m) = \min_g U_g(m) \) over
    demographic groups \( g \).
     

  • Config-robust: over configurations \( \theta \in \Theta \), use \( \min_{\theta \in \Theta} W(m \mid \theta) \) or its average.

Government Engagement

Professed values "buy" standing/credibility to shape regulation lobbying.