Costly belief Elicitation

Brandon Williams

Alistair Wilson

University of Pittsburgh

ESA Columbus, October 2024

  • Experimental economists often give incentives to eliciting beliefs. Why?
  • We hope providing incentives leads to collecting better, more accurate beliefs:
    • Understanding what is asked requires effort
    • Overcome personal motives to distort
    • Doing burdensome calculations
  • Therefore, if belief elicitation is an effortful exercise, how do we best increase the precision of the expressed belief?

 

Motivation

  • We want to understand what incentives produce honest, deliberative beliefs

 

Motivation

0

100

20

80

  • We want to understand what incentives produce honest, deliberative beliefs

 

Motivation

0

100

20

80

  • We want to understand what incentives produce honest, deliberative beliefs
    • We need to understand the psychological costs within a testing environment
    • The scale and structure of the marginal costs becomes important
  • Drawing a distinction between revealing a true belief and the effort required to generate a deliberative belief

 

Motivation

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept
    • Understand how hard this task is to guess
    • Vary the reward structure
  • To do: Measure the psychological costs for actual elicitations
    • Objective Bayesian Posteriors
    • Subjective Beliefs

 

Roadmap

  • Some examples of recent papers in belief elicitation:
    • Testing incentive compatibility:
      • Danz, Vesterlund, and Wilson, 2022
      • Healy and Kagel, 2023
    • "Close enough" payments:
      • Enke, Graeber, Oprea, and Young, 2024
      • Ba, Bohren, and Imas, 2024
      • Settele, 2022
    • QSR or BSR:
      • Hoffman and Burks, 2020
      • Radzevick and Moore, 2010
      • Harrison et al., 2022
    • Others (exact or quartile):
      • Huffman, Raymond, and Shvets, 2022
      • Bullock, Gerber, Hill, and Huber, 2015
      • Prior, Sood, and Khanna, 2015
      • Peterson and Iyengar, 2020

Literature

  • Create a task that mirrors forming a probabilistic belief that requires effort

 

Task

  • Create a task that mirrors forming a probabilistic belief that requires effort

Task

What is the proportion of blue tokens to total tokens in this urn?

81 blue

63 non-blue

56.25% true amount

=

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept
    • Understand how hard this problem is to guess
    • Vary the reward structure

Calibration: Exact

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)
    • Understand how hard this problem is to guess
    • Vary the reward structure

Calibration: Exact

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)
    • Understand how hard this problem is to guess (initial guess treatment)
    • Vary the reward structure

Calibration: Exact

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)
    • Understand how hard this problem is to guess (initial guess treatment)
    • Vary the reward structure (incentives treatment)
      • BSR with only qualitative information
      • BSR with quantitative information
      • A "close enough" incentive

Calibration: Exact

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)

Calibration: Exact

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept

Calibration: Exact

How to get you to exert effort when formulating your belief?

We start by paying $0.50 if you exactly count:

  1. Number of blue tokens
  2. Number of total tokens

Measure accuracy and time taken as a proxy for effort

Vary the difficulty over 5 tasks

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept

Calibration: Exact

Vary the difficulty over 5 tasks

Small with gaps

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept

Calibration: Exact

Vary the difficulty over 5 tasks

Larger with no gaps

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept

Calibration: Exact

Vary the difficulty over 5 tasks

Larger with gaps

  •  
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept

Calibration: Exact

Vary the difficulty over 5 tasks

Largest with no gaps

  •  
    • Ten rounds with an easy task or the hard task base pay plus $X (Oprea, 2020)

Calibration: WTP

LHS:

Constant

Difficulty

RHS:

Varying

Difficulty

Always

Pays $.50

If Correct

$X 

If Correct

Choose

$X

Calibration: Results (n=250)

Model Fitted Values

Calibration: Results (n=250)

Number of Tokens

Indifference Payment

Model Fitted Values

Calibration: Results (n=250)

Number of Tokens

Indifference Payment

Number of Tokens

Time in Seconds

Model Fitted Values

Initial Guesses

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)
    • Understand how hard this problem is to guess (initial guess treatment)
    • Vary the reward structure (incentives treatment)
      • BSR with only qualitative information
      • BSR with quantitative information
      • A "close enough" incentive

Initial Guesses

  •  
    • Understand how hard this problem is to guess (initial guess treatment)

Initial Guesses

  •  
    • Understand how hard this problem is to guess (initial guess treatment)
    • Participants have 15 or 45 seconds to form and enter a guess on the proportion
    • High powered rewards:
      • $2.50 if within 1%
      • $1.00 if within 5%
      • $0.50 if within 10%
    • 10 rounds with varying proportions (pay three decisions)

Initial Guesses: Results (N=200)

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

4.8%

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

95.1%

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

15 Seconds

Initial Guesses: Results (N=200)

Within 10%

Within 5%

Within 1%

Exact

45 Seconds

Incentives

  • Create a task that mirrors forming a probabilistic belief that requires effort
  • Use experiments on Prolific to understand the relationship between cost, effort, and precision
    • Vary the cost for precision and calibrate on how long it takes to complete and willingness to accept (calibration treatment)
    • Understand how hard this problem is to guess (initial guess treatment)
    • Vary the reward structure (incentives treatment)
      • BSR with only qualitative information
      • BSR with quantitative information
      • A "close enough" incentive

Incentives

  •  
    • Vary the reward structure
      • BSR with only qualitative information
      • BSR with quantitative information
      • A "close enough" incentive

Incentives

  •  
    • Vary the reward structure
      • BSR with only qualitative information
        • Text description of payoff structure (Vespa & Wilson, 2018)
      • BSR with quantitative information
        • Full information on the quantitative incentives (Danz et al., 2022)
      • A "close enough" incentive
        • $1.50 if within 1%; $0.50 if within 5%
        • Current use in several papers (e.g. Ba et al., 2024)

BSR Qual.

BSR Quant.

Incentives

How does the expected reward vary by incentive?

Incentives

Incentives

Incentives

  •  
    • Vary the reward structure
      • BSR with only qualitative information
      • BSR with quantitative information
      • A "close enough" incentive
    • 10 rounds, pay 3
    • No time limits
    • N=100 in each incentive treatment

Results

Incentives: Exactly correct

★ ★ ★

Incentives: Within 1%

★ ★ ★

Incentives: Within 5%

★ ★ ★

Incentives: accuracy

Incentives: accuracy

Incentives: accuracy

Incentives: accuracy

By Difficulty

Incentives: time taken (effort)

Incentives: time taken (effort)

★ ★ ★

-22%

Incentives: time taken (effort)

★ ★ ★

-22%

★ ★ ★

+21%

Incentives: Research costs

  • "Close enough" outperformed BSR on both accuracy and time spent
  • Also cheaper in payments to participants (~50%)
  • With a fixed budget, how much more effort could be induced?

Conclusions and future work

  • Have an effortful task that responds to incentives
    • Works well on Prolific
    • Induces objective simple priors
    • Effort decreases variance of the belief
  • Exact incentive works best for inducing effort (and cheaper)
    • Followed by BSR qualitative
    • And BSR quantitative, though output similar to qual.
  • To Do:
    • Examine and measure the effective costs for common elicitations in experiments
    • Examine effects of uncertainty on incentives

Thank you!

Alistair Wilson

Brandon Williams

alistair@pitt.edu

brandon.williams@pitt.edu

Incentives

  •  
    • Vary the reward structure
      • BSR with only qualitative information

Back

Incentives

  •  
    • Vary the reward structure
      • BSR with quantitative information

Back

Incentives

  •  
    • Vary the reward structure
      • BSR with only qualitative information

Back

Incentives

  •  
    • Vary the reward structure
      • BSR with quantitative information

Back

Incentives: accuracy

Back