• Data from 10 published studies
• 2,938 participants,
  50,390 observations
• Standard linear public good games
• No behavioral interventions
• Identical information treatment

• Data from 10 published studies
• 2,938 participants,
  50,390 observations
• Standard linear public good games
• No behavioral interventions
• Identical information treatment

Dataset

Step 1

Clustering

→ uncover patterns

Analysing Game Play Data

Data Driven

• Dimensionality reduction

• Identifies empirical regularities

• Moves from assumed types to discovered response pattern

• Our approach:

  1. Unsupervised modeling to uncover structure

  2. Then apply theory to interpret the clusters.

• Bottom line:
  → Clustering is not a substitute for theory — it is a way to structure behavioral complexity before theorizing.

• Mullainathan & Spiess (2017)

• • "For empiricists, these theory- and data-driven modes of analysis have always coexisted. [...] Machine learning provides a powerful tool to hear, more clearly than ever, what the data have to say. These approaches need not be in conflict."​

1. Choose the number of clusters k
2. Calculate distances between series
3. Apply a clustering algorithm to group similar series

Clustering consists of three main steps

1. Choose the number of clusters k
   
   
   
   
   
   
   
   
   
    
2. Calculate distances between series
3. Apply a clustering algorithm to group similar series

• form of model fit for unsupervised learning
• internal criteria for whether the clustering makes sense

Clustering consists of three main steps

1. Choose the number of clusters k
2. Calculate distances between series
   • DTW distance (global)
   • Euclidean distance (local)
3. Apply a clustering algorithm to group similar series

• Minimum wage increase in NJ (April 1992) hit all firms at the same calendar time.
• The policy hit all fastfood companies in NJ at the same time.

Minimum Wages and Employment
Cart & Krueger (2000)

• Round 2 ≠ same experience
• Local time-points are misleading
• Better: compare global shape

Local Distance Metric

Local Distance Metric

• Finds the best match between similar patterns

• Does not force strict x-alignment.

Dynamic Time Warping (DTW)

Local Similarity Measure

Global Similarity Measure

Local Similarity Measure

Global Similarity Measure

• Results depend fundamentally on how similarity is defined

• The way we measure differences between time series is not neutral — it encodes our theoretical assumptions about what counts as meaningful variation.

• The criteria for similarity embed prior ideas about learning, stability, or change.

Clustering consists of three main steps

1. Choose the number of clusters k
2. Calculate distances between series
   • DTW distance (global)
   • Euclidean distance (local)
3. Apply a clustering algorithm to group similar series

Clustering consists of three main steps

1. Choose the number of clusters k
2. Calculate distances between series
3. Apply a clustering algorithm to group similar series
   • k-means clustering
   • Gaussian Mixture Models (GMM)
   • Agglomerative clustering

Agglom

GMM

k-means

k-means

Agglom

GMM

k-means

k-means

Contrasting Perspectives

• A local— drop from high to low contributions at different times,
• within a global trend of sustained high then low contributions.

• Results depend fundamentally on how similarity is defined.

• A canonical finding in behavioral economics is the downward trend in PGGs.

• We focus on patterns, not specific timepoints.

• → Not “people switch in round 5,”

• → But: a tipping point emerges after which contributions remain low.

• This holds despite idiosyncratic group differences and varying game lengths.

• Most literature relies on ad-hoc interpretation of heat in own vs. others' contribution plots. 
• However, we have seen that local alignment has shortcomings.

• evolutionary game-theoretic learning
• best-response learning
• reinforcement learning

Fatima (2024)

Learning in social dilemmas

Reinforcement Learning (RL)

Artificial Intelligence, Algorithmic Pricing, and Collusion
(Calvano et al. 2020)

Reward Function

Inverse RL recovers reward functions from data.

• Lead to breakthroughs in robotics, autonomous driving, and modeling animal behavior.
• Past models used smoothly time-varying reward functions
  (Alyahyay 2023).
• New approaches model behavior with discrete reward functions
• ​              Hierarchical Inverse Q-Learning (HIQL) (Zhu 2024)

Hierarchical Inverse Q-Learning

• Markov Decision Process: $P(s' |s,a)$
• Behavior of a Q-learner:
  • maintains a Q-table
  • exploitation vs. exploration
  • Q-value update

Hierarchical Inverse Q-Learning

action

state

• Agents learn a value function (Q) that maps state-action pairs to expected future rewards.

• Agents learn to act optimally over time without solving the full Bellman equation upfront.

• Very intuitive for modeling boundedly rational players in repeated games.

Hierarchical Inverse Q-Learning

$Q(s,a) = (1- \alpha) Q(s,a) + \alpha \left( r + \gamma \max Q(s', a') - Q(s,a) \right)$

re-ward​

 

action

state

Hierarchical Inverse Q-Learning

action

state

• Instead of learning Q-values from rewards, we observe actions and states and try to infer what reward function the agent must be optimizing.

• What objective function (preferences, incentives) would rationalize the observed behavior as approximately optimal?

• IRL is a structural estimation method dressed in AI terms.

Hierarchical Inverse Q-Learning

Hierarchical Inverse
Q-Learning

How many reward functions should we estimate?

• • The intention to free-ride is not fixed.
  • Some variation exists, but the competing intention never exceeds the adoption threshold .

• The cluster actively experiments with new strategies.
• Deliberate switching between strategies.

• The intention to free-ride may be less rigid than previously assumed.
• Longer horizons increase intention volatility.
• Longer cooperation horizons may create more opportunities for behavioral shifts.

Freeriders

• Longer time horizons generally promote cooperation.
• A small group of participants remains cooperative regardless of game duration.
  • For some, cooperation is a stable trait rather than a response to game length.

• Dataset with ~ 50'000 observations from PGG
   
• Use a global distance metric to partition the data
  • Threshold Switching
    
     
• Estimate latent intentions vis HIQL
  • ​Unifying theory to explain all behavioral clusters
    (including the Other cluster)
  • Intention of consistent cooperation is independent of game length
  • Intention to freeride is less stable over long time horizons

carinah@ethz.ch

slides.com/carinah

github.com/carinahausladen

S

• Intentions become less smooth with increasing game length.
• RL perspective: expected
• Longer time horizons = more opportunities to explore

RL Heuristics

Nax (2016)

• Individuals display higher inertia when others ↑ as opposed to other↓

• Adjustments exhibit greater variability following other↓ as opposed to others ↑

Data-Driven Approaches

Theory-driven classifications may not accurately capture the heterogeneity observed.

• Bayesian Model
  (Houser 2004)
• C-Lasso / Penalized Regression
  (Su 2016, Bordt 2019)
• Clustering
  (Bolle 2021)

Analysing Game Play Data

Theory-Based Approaches

Strategy-method types often align with actual gameplay data
(e.g. Muller 2008).

• Finite Mixture Model (Bardsley 2006)

1. Bayesian Model (Houser 2004)
   1. Assume functional form: Decision dntdepends on past choices, payoffs, and future value function.
   2. Parameter estimation: Bayesian mixture model estimates how much each factor influences decisions.
   3. Parameter clsutering: Gibbs sampling assigns individuals to behavioral types based on posterior probabilities.
2. C-Lasso (Su 2016, Bordt 2019)​
   1. Assume functional form: Decision depends on past choices, payoffs, and group behavior.
   2. Simultaneous classification and estimation.
      1. C-Lasso penalizes differences between individuals, clustering them into distinct decision-making types.

Analysing Game Play Data

Strong assumptions on decision-making :

• predefined functional form (e.g., polynomial)
• assumes which features matter

1. Clustering (Bolle 2021, Fallucchi (2018))​
   1. ​does not assume a functional form.
   2. pattern recognition in the game data (own decision, decisions of group player
   3. hierarchical clustering with Manhattan Distance

Analysing Game Play Data

Detect behavioral patterns without assumptions.

Future Research Ideas

Use RL to derive a policy that maximizes agents' intrinsic motivation for sustainable behavior.

Simulate an economy to study emergent dynamics.

Existing simulations lack grounding in behavioral types (e.g., research on cooperation).

Incorporating behavioral types could improve simulations.

Tailored policies—potentially even type-specific—may lead to better outcomes.

• Intentions are inferred latent variables that combine observed actions and states using the Bellman equation.
• The Bellman equation assumption: Current actions are selected to maximize expected future discounted rewards.
• The intention itself is not directly observed but is instead estimated as a posterior probability based on the observed sequence of actions and states.
• These probabilities are computed in a way that maximizes the likelihood of observing the given actions and states, assuming the hierarchical structure.

Hierarchical Inverse Q-Learning

action

state

• Markov Decision Process: $P(s' |s,a)$
• Behavior of a Q-learner:
  • maintains a Q-table
  • exploitation vs. exploration
  • Q-value update

$Q(s,a) = (1- \alpha) Q(s,a) + \alpha \left( r + \gamma \max Q(s', a') - Q(s,a) \right)$

Expected best possible outcome from the next state

Compare to now

 

re-ward​

 

Data-Driven Approaches

• Fallucchi (2019):
  "Various" group –
  Unpredictable
  contributions: 16.5%
• Fallucchi (2021):
  "Others" type –
  Neither reciprocation nor strategy: 22% – 32%

Loosely / Un - explained Behavior

Theory-Based Approaches

• Houser (2004):
  Confused –
  High randomness and decision errors: 24%
• Bardsley (2006):
  Tremble terms –
  Random deviations (decline with experience): 18.5%.

Theory-Based Approaches

• Houser (2004):
  Confused –
  High randomness and decision errors: 24%
• Bardsley (2006):
  Tremble terms –
  Random deviations decline with experience: 18.5%.

Why do we find these "other" clusters?
And can we find a better descriptor than "confused"?

• Based on a theoretical model that initially was not built for game data.
• Ignores e.g. learning.

Data-Driven Approaches

• Fallucchi (2019):
  "Various" group –
  Unpredictable
  contributions: 16.5%
• Fallucchi (2021):
  "Others" type –
  Neither reciprocation nor strategy: 22% – 32%

• The partitioning is tailored to the data. 
• The interpretation, however is not.
• It relies on the strategy-data type of thinking.

Action

State

• Markov Decision Process: $P(s' |s,a)$
• Goal: infer unknown reward function $r (s, a)$
• Behavioral policy of a Q-learner:
  • Future rewards shape current decisions $\gamma$.
  • Humans implicitly learn and update $Q(s,a)$ based on experience.
  • $Q(s,a) \;=\; r(s,a) \;+\; \gamma \sum_{s' \epsilon S} P(s'|s,a) \max_{a' \epsilon A} Q(s',a')$

•  

• Explanations  

  • Social preferences  

  • Confusion 

Unexplained behavior due to strict theoretical assumptions and limited techniques for interpretation.
 

Reliance on perfect point-wise alignment

Methods Analysing Game Play Data

Two comparative observations

• Traditionally, this behavior would have been dismissed as noise.
• This HIQL framework offers a unifying theoretical interpretation.
• Stable trends in other clusters indicate the dominance of one intention.
• Cluster 2 reflects frequent intention switching.
• Participants in Cluster 2 struggle between competing intentions.

Behavior Beyond Theory

• Finite Mixture Model
  
  
   
• Bayesian Model
  
  
   
• C-Lasso
  
  
   
• Clustering
   

Theory Driven

Data Driven