Alignment as Social Choice

Welfare Effects of Four Major AI Developers’ Alignment Frameworks in Large-Scale Human Preference Data

Carina I Hausladen

• RLHF must aggregate heterogeneous feedback into one model
• This is precisely the problem social choice theory has studied for decades

Strategic Placement

Literature

Literature

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

Reinforcement Learning from Human Feedback

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

Literature

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

• conclude the welfare-correct object is a distribution over several winners
  (as opposed to single-winner aggregation)
• develop aggregators/training rules for heterogeneous populations
• the magnitude of the flattening their methods correct for has not been quantified on human alignment-preference data.
  • Kim et al. use movie rating data
  • Gölz et al. is axiomatic

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

Social choice as a lens for RLHF

Dai & Fleisig (2024)

Conitzer et al. (2024)

Halpern et al. (2026)

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

Axiomatic & algorithmic work

Kim et al. (2025)

Goelz et al. (2025)

AI ethics as public communication

Jobin et al. (2019)

Mittelstadt (2019)

Hagendorff (2020)

• analyze ethics guidelines & principles
  published by AI actors
  • Jobin et al.: governments / NGOs / academia
  • Hagendorff: AI labs
• in aggregate / by sector
  • <> company-by-company welfare profile
• code principles
  • <> explicit social welfare functionals
• corpora predate the pluralistic alignment era

Methods

• Three conversation types: unguided, values-guided, controversy-guided
• 21 different models
• Up to 4 models respond per prompt
• 1,396 unique evaluators

Kirk et al. (2024)

The PRISM Alignment dataset

Bradley-Terry

• Each user rates only a subset of models
• Turn the ratings into a network of pairwise comparisons
  • model i beats j when rated higher
• Fit a latent strength \( \beta_i \) per model
• The odds one model beats another depend only on the gap:
  \( \text{odds}(i \succ j) = e^{\beta_i - \beta_j} \)
• RLHF objective: the reward model is trained by maximizing this BT likelihood on human preference pairs

Results

Size-vs-coverage curve

1. cardinal prefs → per-user BT scores
2. top-3 truncation →k-approval ballot (drop order)
3. seriation (user-model-matrix reordering)
4. coverage under the Chamberlin–Courant objective

• Kim et al.: proportionality and putting all mass on one winner are mutually exclusive.
• their preference learning framework aligns aggregate opinions proportionally with the true population distribution
• their policy is a probability distribution over the alternatives
  • deployment-layer translation: a menu of models

A single-model solution is insufficient

• distortion measures how much welfare is lost when alignment fits a single BT reward via RLHF
  • collapses users into one 'mythical user'
• this paper: empirical lens based on real preference data

Results

Performance across substantive domains

• assume "a single latent notion of LLM quality"
• acknowledge that LLMs serve "diverse … tasks, each with their own goals"
• defer task-heterogeneity to future work

• Chatbot Arena ranks models by Elo
  • capability columns (coding, vision, math)
  • every column is still a single ranking; none of them is a committee
• Theory (Gölz et al., Kim et al.): the welfare-correct object is a committee, not a winner
• coverage–committee-size tradeoff
  • ​how many models, covering how many users, on which topics

Implications for AI Leaderboards

Do company alignment policy documents reflect these insights? 

Results

Preliminary

Alignment Documents

• How would moral philosophy / welfare economics / social choice read those documents?
   
• Can we find a set of prevalent normative lenses ...
   
• ... that also can be operationalized in the PRISM data?

• not exhaustive
• broad normative traditions that recur across moral philosophy, welfare economics, and social choice theory

Four Normative Lenses

Four Normative Lenses

applied to the PRISM dataset

Alignment documents

• 4 AI labs

• 8 documents (2 per lab)

  • ~139k words, ~295 pages overall

  • 2022–2026  

  • type

    • 5 arXiv papers

    • 3 RSP, Model Spec, RAI Standard / policy/specs

A principled welfare rule can choose a rarely-competitive model:

• Chat-Bison reaches a topic's top-3 only 3/22
• only on low-stakes topics (Cooking, Reality TV,  Weather)

Discussion

• quantify tradeoff between committee-size and coverage
  • multiple models to cover preferences of a substantive part of the population 
• model preferences vary across substantive domains
• AI developers normative fairness choices
  • no single winner dominates
  • pick a model performing poorly on substantive domains

Summary

Limitations

• four lenses determine which aspects get highlighted
• SWF-translation is a subjective decision
• two documents per lab is not exhaustive
• adding a second human-preference dataset 
  • Ghafouri et al. (2026): PRISM preferences are largely non-attitudes/artifacts (only ~2% "genuine")
  • Community Alignment Dataset (2026)

Appendix

Bradley-Terry

Performance varies across substantive domains

• Command does not cover top three in several Political/Civic domains
• Importantly, the alignent documents reviewed do not specify concrete substantive domains for which performance should be optimized or guaranteed
• A more explicitly democratic approach—such as PRISM, where users reveal what matters to them by selecting discussion topics—could provide an empirical basis for identifying domains that warrant additional fine-tuning.

The corpus

• 4 AI labs

• 8 documents (2 per lab)

  • ~139k words, ~295 pages overall

  • 2022–2026  

  • type

    • 5 arXiv papers

    • 3 RSP, Model Spec, RAI Standard / policy/specs

Anthropic

• Constitutional AI: Harmlessness from AI Feedback | 2022 (research)
• Responsible Scaling Policy (v3.3) | 2026 (public)

OpenAI

• Training LMs to Follow Instructions w/ Human Feedback| 2022 (research)
• OpenAI Model Spec | 2025 (public)

Meta

• Llama Guard: LLM-based Input-Output Safeguard | 2023 (research)
• The Perfect Blend: Redefining RLHF with Mixture of Judges | 2024 (research)

 Microsoft

• Controllable Safety Alignment | 2024 (research)
• Microsoft Responsible AI Standard | 2022 (public)

• "developing ASL-3 Deployment Safeguards to mitigate catastrophic risks"
• "prioritize this sort of safety even above ethics"
• "being more careful in consumer contexts … due to the potential for vulnerable people"

Threshold is lexically prior to helpfulness

Concave prioritarian weighting

• "When two root-level principles conflict, the model should default to inaction."
• "synthesizes peoples' values in a way that achieves broad consensus amongst many groups."

Safety floor as feasibility constraint

• "similar quality of service for identified demographic groups, including marginalized groups"
• "target minimum performance level for all groups"
• "target maximum (absolute or relative) performance difference between groups"

the worst-off demographic group

Outlook

• We have shown the structure of the heterogeneity and the cost of a single reward.
  • The data and the coverage measure can serve as a common testbed: existing aggregators (Kim 2025; Gölz 2025) can be fit on this data and compared on our diagnostic plot
    • each method shown by how much of the population it actually serves.

Outlook–MoE

Mixture of Experts in Large Language Models,
Zhang et al. (2025)

• router picks a number of experts
  • how many experts
  • and route on what
  • --> set to minimize loss
• Our qualitative findings point to:
  • how many distinct regimes a heterogeneous population actually has
  • that the right expert is topic-conditional — which our per-topic results show directly.

• RLHF must aggregate heterogeneous feedback into one model
  •   Human preferences are collected, then aggregated to fine-tune a single model
  •   Feedback is inconsistent, irrational, and heterogeneous across people
  •   This is precisely the problem social choice theory has studied for decades
• No aggregation is neutral
  •   Any rule for combining preferences embeds value judgments (Arrow)
  •   RLHF silently fixes one rule — Bradley–Terry — and calls it technical
  •   "RLHF is a voting rule in disguise": the choice is normative, not engineering
• So every aligned model already implements some SWF
  •   Whether or not developers say so, a welfare function is being applied
•  The implicit rule becomes readable in company documents
  • Frontier developers describe alignment in CS / engineering terms
  • But the same texts state whose preferences count, how they combine, which constraints are lexicographically prior
  • we read this through the vocabulary of welfare economics
• Different implied SWFs → different winners on real data
  • We extract each company's implied SWF and apply it to PRISM
  • Aggregation happens at the model-selection layer, not the RLHF loss
  • Does rule choice change the outcome? → yes: three distinct winning models

Scope

•   - We study public commitments — two flagship alignment/safety documents per company — not internal practice or actual RLHF pipelines
•   - We map texts to stylized SWF archetypes — representative welfare-function classes, not precise or audited specifications
•   - The mapping is interpretive; correspondence is not one-to-one, and alternative formalizations are plausible
•   - The claim is illustrative: stated commitments can be translated into distinct aggregation rules, and the rule choice is consequential on real human data
•   - This is step one of a broader agenda — finer-grained coding, more companies, more datasets, and eventually linking stated commitments to observed model behavior --> I am looking for interested collaborator :) 

Democratic Alignment and RLHF

• As AI becomes a ubiquitous phenomenon in society, it is crucial to specifically design these technologies to support collective intelligence.
• This requires aligning AI systems with principles of plurality and diversity.
  • Democratic alignment
• One widely used method for aligning AI systems with human values is RLHF
  • human preferences are collected and then aggregated to fine-tune models; the goal is to guide them more socially aligned behavior.

• RLHF and Constitutional AI face exactly the problem social choice theory has studied for decades — aggregating inconsistent, irrational, and heterogeneous human feedback into a single collective decision.
• RLHF is a voting rule in disguise

Does rule choice change the outcome when aggregating alignment preferences?

Which rules should we pick?

Lazar, Seth. "Governing the algorithmic city." Philosophy & Public Affairs 53.2 (2025): 102-168.

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/papa.12279

Welfare trade-offs across alignment criteria

Pluralistic Alignment and
Preference Aggregation

• Once human preferences are collected, a crucial question arises: How should these preferences be aggregated?
  • Should some groups have more influence than others?
  • If so, which ones and based on what criteria?
  • These concerns fall under the broader framework of pluralistic alignment.
  • In fact, the choice of whom to ask and how to aggregate their responses is interconnected: different aggregation strategies may inherently privilege certain groups over others, shaping the outcomes of AI alignment in subtle yet powerful ways.

• Frontier LLM developers describe their alignment approaches in a variety of ways.
• These descriptions are typically framed in CS / engineering terms.
• However, they can also be read as informal statements about whose preferences matter, how they are aggregated, and which constraints are lexicographically prior to others.  
• Social choice theory offers a rich vocabulary for talking about such questions.  

Company Alignment Philosophies

• pick SWFs that are grounded in company alignment documents 
• use them to aggregate large scale human alignment preference data
• see how outcomes change

Alignment documents --> SWFs

• We fix five normative lenses standard in social choice
  • utilitarian, prioritarian/catastrophe-averse, egalitarian, rights/constraints, democratic
• Two independent reads per company.
  • ⚙️  Method read → the form. What the alignment mechanism does fixes the aggregation form (mean / lexicographic / config-family).
  • 🔍 Lens coding → the content. Blind dual-coding of the docs against the five lenses fixes which value.
• Validation. Pull the verbatim evidence quotes, re-code them blind (2 passes by Claude Opus 4.8 + adjudication), report reliability → κ = 0.95

Aligment Intuition

100

0

Government Engagement

Professed values "buy" standing/credibility to shape regulation lobbying.