Experimenter Demand

[The participant’s] general attitude of mind is that of ready complacency and cheerful willingness to assist the investigator in every possible way by reporting to him those very things which he is most eager to find.

-A H. Pierce, 1908

The subject’s performance in an experiment might almost be conceptualized as problem-solving behavior... he sees it as his task to ascertain the true purpose of the experiment and respond in a manner which will support the hypotheses being tested.

-M. T. Orne, 1962

Experimenter Demand

the critical assumption underlying the interpretation of data from lab experiments is that the insights gained can be extrapolated to the world beyond

-S. Levitt and J. List, 2007

many reasons to suspect that these laboratory findings might fail to generalize to real markets

-S. Levitt and J. List, 2008

Design

• Eight within-subject decisions:
  • Four lottery valuations:
    • WTP and WTA
    • Lotteries with 10% and 90% chance of winning $10
  • Two donations:
    • Matched (low cost)
    • Unmatched (high cost)
  • Two intertemporal allocations:
    • Immediate (today vs a week from now)
    • Delayed  (tomorrow vs week from tomorrow)

Design

• Eight within-subject decisions:
  • Four lottery valuations
  • Two donations
  • Two intertemporal allocations

• Three between-subject treatments:
  1. No demand
  2. Positive demand
  3. Negative demand
• Three Populations:
  1. Laboratory (Pitt undergrads)
  2. Mechanical Turk
  3. Prolific 

Online Populations

Design

• Eight within-subject decisions:
  • Four lottery valuations
  • Two donations
  • Two intertemporal allocations

• Three between-subject treatments:
  1. No demand
  2. Positive demand
  3. Negative demand
• Three Populations:
  1. Laboratory (Pitt undergrads)
  2. Mechanical Turk
  3. Prolific 

Endowment Effect

Endowment Effect

Endowment Effect

Endowment Effect

Endowment Effect

This is extreme differential demand over the comparative static

Endowment Effect (High)

\(p=0.019\)

Endowment Effect (High)

\(p=0.731\)

Endowment Effect (High)

\(p=0.001\)

Endowment Effect (High)

\(p=0.127\)

False Positive in Online Samples

• All of the domains where we expect a directional result are replicated online:
  • Probability Weighting
  • Endowment effect (low probs)
  • Charitable giving
• However, we do find that extreme experimenter demand can create false positives in both online samples:
  • Present Bias
  • Charitable giving foregone amount
• Reasons:
  • Slightly more consistent demand effects in online sample
  • Larger sample sizes easier to generate significance
• With large samples, need to focus on economic size of the effects!

False Positive in Online Samples

Present Bias

Task:

Convex budget set. Have $10 to be paid at date \(t\), can move up to $9 to date \(t+7\) earning 20% interest on moved amount. Treatments are for:

• \(t=0\) (today vs week from today)
• \(t=1\) (tomorrow vs week from tomorrow)

Literature Predictions:

• Andreoni and Sprenger, 2012:
  • Compared to an immediate sooner date, participants will be no more patient when the sooner date is delayed
  • Purposeful null result: \( \text{Transfer}(t=0) = \text{Transfer}(t=1) \)

False Positive in Online Samples

Present Bias: Laboratory sample

False Positive in Online Samples

Present Bias: MTurk sample

False Positive in Online Samples

Present Bias: MTurk sample

\(p=0.039\)

False Positive in Online Samples

Present Bias: MTurk sample

\(p=0.033\)

False Positive in Online Samples

Present Bias: Prolific sample

False Positive in Online Samples

Present Bias: Prolific sample

\(p=0.043\)

False Positive in Online Samples

Present Bias: Prolific sample

\(p=0.112\)

Effect size normalization

• For each comparative static we construct a normalized effect size  \[ y_i = \hat{\beta_0} +\hat{\beta}_1\cdot 1_{\text{Treat}}+\hat{\epsilon}_i \]
• Variation normalized coefficient is: \[\hat{D}=\frac{\hat{\beta}_1}{\hat{\sigma}_{\hat{\epsilon}}} \]
  • This effect size statistic is what is referred to as Cohen's-\(D\)
    • Cohen gives the informal guidance that \(0.2\sigma\) is small, \(0.5\sigma\) medium, \(0.8\sigma\) large
• So interpretation of effect size is as a multiple of the unexplained variation over the decision \(y\) (separate from the treatment effect)
  • Many inferences require conditioning on more variables
• Here if the total data size is \(N\), then we can just think of \(\sqrt{N}\cdot\hat{D}\) as the two-sample Student's-\(t\) test statistic

Comparative Statics as \(D\)'s

Comparative Static as \(D\)'s

Effect Sizes

• Our evidences suggests even extreme experimenter demand (strong and differential across treatment) can push comparative statics by approximately \(0.2\sigma\)
• How big are effects sizes in experimental studies?
• Initial stages of data synthesis for 33 experimental studies in the AER in the last seven years:
  • Available data
  • "Important" general-interest studies
• For each paper we try to construct a simple normalized effect size  \[ y_i = \hat{\beta_0} +\hat{\beta}_1\cdot 1_{\text{Treat}}+\hat{\epsilon}_i \]
  • We generalize this approach for Diff-in-Diff designs to focus on the interaction
• Normalized coefficient is: \[\hat{D}=\frac{\hat{\beta}_1}{\hat{\sigma}_{\hat{\epsilon}}} \]

Effect Sizes

Effect Sizes

Effect Sizes

Effect Sizes (Non-null)

Effect Sizes

Effect Sizes

Effect Sizes

Probability Weighting

Probability Weighting

\(p=0.002\)

\(p<0.001\)

Probability Weighting

Probability Weighting

Probability Weighting

Chartiable Giving

Task:

Endowed with $20, and given the option to donate any of this to a local Children's Hospital. Donation cost is either Low (matched donation, \(c=\$0.50\)) or High (unmatched donation, \(c=\$1.00\)).

Literature Predictions:

• Andreoni & Miller (2002); Huck & Rasul, (2011); Karlan & List, (2007)
  • Charity receives larger donation with than without a match
    • DonatedAmount(Low)>DonatedAmount(High)

Chartiable Giving

Task:

Endowed with $20, and given the option to donate any of this to a local Children's Hospital. Donation cost is either Low (matched donation, \(c=\$0.50\)) or High (unmatched donation, \(c=\$1.00\)).

Literature Predictions:

• Andreoni & Miller (2002); Huck & Rasul, (2011); Karlan & List, (2007)
  • Charity receives larger donation with than without a match
    • DonatedAmount(Low)>DonatedAmount(High)

Chartiable Giving

Task:

Endowed with $20, and given the option to donate any of this to a local Children's Hospital. Donation cost is either Low (matched donation, \(c=\$0.50\)) or High (unmatched donation, \(c=\$1.00\)).

Literature Predictions:

• Andreoni & Miller (2002); Huck & Rasul, (2011); Karlan & List, (2007)
  • Charity receives larger donation with than without a match
    • DonatedAmount(Low)>DonatedAmount(High)

Chartiable Giving

Task:

Endowed with $20, and given the option to donate any of this to a local Children's Hospital. Donation cost is either Low (matched donation, \(c=\$0.50\)) or High (unmatched donation, \(c=\$1.00\)).

Literature Predictions:

• Andreoni & Miller (2002); Huck & Rasul, (2011); Karlan & List, (2007)
  • Charity receives larger donation with than without a match
    • DonatedAmount(Low)>DonatedAmount(High)

Present Bias

Task:

Convex budget set. Have $10 to be paid at date \(t\), can move up to $9 to date \(t+7\) earning 20% interest on moved amount. Treatments are for:

• \(t=0\) (today vs week from today)
• \(t=1\) (tomorrow vs week from tomorrow)

Literature Predictions:

• Andreoni and Sprenger, 2012:
  • Compared to an immediate sooner date, participants will be no more patient when the sooner date is delayed
  • Purposeful null result: \( \text{Transfer}(t=0) = \text{Transfer}(t=1) \)

Present Bias

\(p=0.339\)

Present Bias

\(p=0.239\)

Present Bias

\(p=0.465\)

Present Bias

\(p=0.819\)

Sensitivities

Lab:

Sensitivities

Mturk:

Sensitivities

Prolific: