• For each type there is a different uniquely maximizing choice within the mechanism
  • This yields fully separating behavior
• Given this one-to-one correspondence, the analyst can interpret the relevant choice as the type

• Elicitation mechanisms are at the sharp end of mechanism design, using the above correspondence for measurement
  • ​Often as an input into inferential regressions
  • Subjective beliefs are a clear example, and are often central for inference

• Need to examine the extent to which the theoretical assumptions holds
• Show that a prominent mechanism with relatively weak theoretic conditions for IC is not behaviorally incentive compatible
  • Demonstrate the substantive effects of this failure over: Objective probabilities, Objective posteriors, Subjective beliefs 
• Use this to motivate two methodological checks to assess Behavioral Incentive Compatibility

Main Idea:

Paper examines gender & competition:

• Do women compete less than men?
• Use a clever experimental design to examine tournament entry
• Measure beliefs with a very simple elicitation.

Evolution of Belief Elicitation

• Niederle & Vesterlund used a very simple (and coarse) elicitation device:
  • Modal rank in tournament (paying $1 if correct)
• But the literature has increasingly focused on elicitations of arbitrarily precise probabilities
  • Initial IC elicitations assumed risk-neutral EU maximizers: Quadratic Scoring Rule (QSR)
  • But risk aversion pushes beliefs to center
• One response to this was to control for risk preferences within the elicitation
• Another was to use an elicitation that is IC for a wider set of preferences 

  • Houssain & Okui (2013): the Binarized Scoring Rule

• Incentive compatibly. QSR (Brier, 1950), BSR (Roth and Malouf, 1979; Grether, 1980; Allen, 1987; Hossain and Okui, 2013; Schlag and van der Weele, 2013), BDM (Holt and Smith, 2009; Karni, 2009)

• Surveys (Manski, 2004; Schotter and Trevino, 2014; Schlag et al., 2015)

• Distortions and corrections (Offerman et al., 2009, Andersen et al., 2013; Harrison et al., 2013; Armantier and Treich, 2013; Schlag and van der Weele, 2013)

• Stakes and hedging (Blanco et al., 2010; Coutts, 2019)

• Does elicitation change behavior? (Croson, 2000; Wilcox and Feltovich, 2000; Rutstrom and Wilcox, 2009; Gächter and Renner, 2010)

• Does incentivization matter? (Offerman and Sonnemans, 2004; Gächter and Renner, 2010; Wang, 2011; Trautmann and van de Kuilen, 2014)

• Does properness matter? (Nelson and Bessler, 1989; Palfrey and Wang, 2009)

• Consistency with actions (Cheung and Friedman, 1995; Nyarko and Schotter, 2001; Costa-Gomes and Weizsäcker, 2008; Rey-Biel, 2009; Blanco et al., 2011; Ivanov, 2011; Hyndman et al., 2013; Armantier et al., 2013)

• Models of belief formation (Fudenberg and Levine, 1998; Camerer and Ho, 1999; Nyarko and Schotter, 2001; Hyndman et al., 2012)

• Bayesian updating (Holt and Smith, 2009; Benjamin, 2019)

• Higher-order beliefs (Dufwenberg and Gneezy, 2000; Charness and Dufwenberg, 2006; Manski and Neri, 2013)

• Each reported belief q is linked to state-contingent lottery.
• With a binary outcome:

Baseline: Information Treatment

✅

✅

✅

✅

✅

✅

✅

✅

Dominant Strategy

Payoff Description

• Randomly draw two uniform numbers between 0-100
  • If the selected urn is the Red urn: You will win the $8 prize if Your Guess is greater than or equal to either of the two Computer Numbers.
  • If the selected urn is the Blue urn: You will win the $8 prize if Your Guess is less than either of the two Computer Numbers.
• Yields the BSR lottery probabilities (Wilson & Vespa 2018)

Payoff Slider

Feedback

At the end of each round:

• Realized urn
• Earned probability on each elicitation

Guess 1: Prior Elicitation

Should report induced prior if incentivized to tell truth

Only 15 percent of participants consistently report given prior

Information: False priors report

• Confusion (inability/unwillingness to report given prior)

• Incentives:

  • Failure to reduce compound lottery (RCL)

  • Payoff structure

• Reporting a belief toward center
  • large increase in chance of winning on unlikely event
  • smaller decrease in chance of winning on likely event

What Drives False Reports?

• Confusion (inability/unwillingness to report given prior)

• Incentives:

  • Failure to reduce compound lottery (RCL)

  • Payoff structure (asymmetric and flat incentives)

• False reports more likely on non-centered than centered priors
• Deviations more likely toward center than near extreme (reports pull-to-center)

Evidence of incentives distorting reports

• False reports more likely on non-centered than centered priors
• Pull-to-center: Deviations more likely toward center than near extreme
• Survey responses discuss a hedging motive 

• Confusion

• Incentives

  • Failure to reduce compound lottery

  • flatness

  • asymmetry

• Vary information on incentives:

  • RCL-calculator : Aid reduction of compound lottery

  • No-Information: Eliminate quantitative information on incentives

Cause of false BSR reports?

✅

✅

✅

✅

❌

✅

✅

✅

✅

❌

Treatments:

Information treatment

RCL Treatment

Proportion of non-centered reports in each bin:

• Information on BSR incentives increases false reports (by 150%) (between-subject)
• Test effect of information within subject
  • Feedback Treatment
    • No-Info + scenario feedback with gradual information on incentives

• Starts out at No Information level
• Ends up at Information level
• Information on incentives distorts truthful reporting within subject

• Information on incentives increases false reports

  • Between subject: Information vs. No-Information

  • Within subject: Feedback

• What is ‘enough’ information to maintain truth telling?

  • Description Treatment

By round

By prior

• Evidence for systematic distortions over objective priors
  • When subjects are informed on the incentives they make systematic deviations
  • That deviations are purposeful distortions is clearest
  • But the environment is perhaps more artificial
• The same patterns are observed over Bayesian Posteriors
  • Data here is harder to pinpoint as we don't know the 'true' posterior beliefs
  • But distributions of response shift in same systematic way
• Two questions raised:
  1. Would this also hold for subjective beliefs?
  2. Does any of this matter for inference?

• EU in standard economic theory is typically used at an aggregate (i.e. average)  level
  • Predict a comparative static for a population
• But in the elicitation EU is used at the observation level:
  • Use the IC under EU to interpret a choice over incentives as a measured belief
  • 'Small' mistakes in the EU assumption can lead to a measurement error for any inference 
• Regressions using mismeasured data can cause biased inference

To understand the effects on inference we use a simple model of the center-bias distortions

Observed belief is:

Observed belief is:

Observed belief is:

LHS: Confidence difference between men and women:

NV-No-Information

NV-Information

NV-No-Information

NV-Information

• Original finding is that:
  • Beliefs explain a significant proportion of the gender gap
• Prediction from model for NV-Information is that gender gap in competition is more negative:
  •                 (gender gap in confidence)
  •                 (Belief effect on entry)

• The BSR does not work as an incentive-compatible elicitation
  • The less you tell the participants about the incentives, the worse the data
  • This is true for:
    • Objective priors
    • Updated posteriors
    • Subjective beliefs
• New elicitations are required. But how should we go about assessing them?

1. The mechanism should not yield worse data when participants are given precise information on the incentives on offer
2. If presented with the pure incentives available in the mechanism, the majority of participants should be choosing the theorized maximizer

BIC Diagnostic 1:

• Information vs No Information comparison

Our paper demonstrates this methodology across:

• Elicitation of an objective prior
• Elicitation of an objective posterior
• Elicitation of a subjective belief 

Have similar data showing this comparison for:

• Quadratic scoring rule
• Binarized-BDM 

BIC Diagnostic 2:

• Extract the pure incentives from the elicitation and ask the participants to choose. A majority should be choosing the theorized maximizer

We asked 120 subjects to choose their preferred lottery pair when the probability of Red was set to either 20% or 30%. Interpret choice via the EU assumption:

Same thing, but for QSR incentives:

Same thing, but for binarized-BDM (Karni)

First Price Auction, two bidders, uniform values:

\[v^\star=0.3\]

First Price Auction, two bidders, uniform values:

\[v^\star=0.7\]

\[v^\star =0.3\]

\[v^\star =0.7\]

First Price Auction:

Uniform Preferences

Correlated Preferences

Deferred Acceptance (Proposing over $6/$4/$2)

Assume all other players truthfully reveal

• Two other proposers
• Three prize amounts

Uniform Preferences

Deferred Acceptance (Proposing over $6/$4/$2)

Correlated Preferences

Deferred Acceptance (Proposing over $6/$4/$2)

Uniform Preferences

Correlated Preferences

Deferred Acceptance (Proposing over $6/$4/$2)

• On the Binarized Scoring Rule
  • Substantial false-report rate for objective prior
  • Systematic deviations driven by information on the incentives
    • Between (Information vs No Information)
    • Within (gradual Feedback)

  • Distortions generated can qualitatively affect inference

    •  Replication of Niederle & Vesterlund fails when incentive information present

• Overly content to appeal to theoretical incentive compatibility, when what is actually required are notions of behavioral compatibility
• For belief elicitation, qualitative notions are effective:
  • Both No Information and Description work well
  • But need to ask ourselves if it truly the incentives, instead  of framing/call to authority 
  • Might want to ask if it even makes sense to collect arbitrarily precise beliefs.
• Methodology offers simply diagnostic checks for behavioral incentive compatibility
  • Simple demonstrations for check on First Price auctions and Matching mechanisms