Data Methods: Experiments

Data Methods:
Experiments

Karl Ho

School of Economic, Political and Policy Sciences

University of Texas at Dallas

We are limited by the impossibility of experiment. Politics is an observational, not an experimental science . . . ” (Lawrence Lowell, APSA President 1910)

Why experimental design?

Experiments facilitate causal inference through the transparency and content of experimental procedures, most no- tably the random assignment of observations to control and treatment groups.

- Druckman et al. 2006

What was the term "experiment" used in APSR?

  1. an institutional innovation such as a new constitution, electoral system, or policy process

  2. a simulation or an empirical test that involves neither an institution nor randomized trials

  3. a randomized trial in which the researcher randomly assigns units of observation to control and treatment groups. 

Why experimental design?

- Druckman et al. 2006

Purposes of experiments

(Economist Alvin Roth)

  1. Searching for facts

  2. Speaking to theory

  3. Whispering in the ears of princes

Purposes of experiments

(Economist Alvin Roth)

1. Searching for facts

“searching for facts,” where the goal is to “isolate the cause of some observed regularity, by varying details of the way the experiments were conducted. Such experiments are part of the dialog that experimenters carry on with one another.” 

Purposes of experiments

(Economist Alvin Roth)

2. Speaking to theory

where the goal is “to test the predictions [or the assumptions] of well articulated formal theories [or other types of theories]... Such experiments are intended to feed back into the theo- retical literature—–that is, they are part of a dialogue between experimenters and theorists.”

Purposes of experiments

(Economist Alvin Roth)

3. whispering in the ears of princes 

which facilitates “the dialogue between experimenters and policymakers . . . [The] experimental environment is designed to resemble closely, in certain respects, the naturally occurring environment that is the focus of interest for the policy purposes at hand.”

Observational vs. Experimental data 

In observational studies , researchers collect subject data and measure variables of interest without assigning treatments to the subjects.

In an experiment investigators apply treatments to experimental units (subject) and then proceed to observe the effect of the treatments on the experimental units.

 

Observational study 

  1. Find 100 women age 30 of which 50 watch TV every day while the other 50 do not.

  2. Measure political knowledge for each of the 100 women.

  3. Analyze, interpret, and draw conclusions from data.

Experimental study 

  1. Find 100 women age 30 who do not watch TV.

  2. Measure political knowledge for each of the 100 women

  3. Randomly assign 50 of the 100 women to watch TV for 10 days and the other 50 remain not exposed to TV.

  4. Measure political knowledge for each of the 100 women again and analyze results.

 The Experimental Method

1. What is an experiment?

Purpose: The experimental design is to demonstrate causation, that A causes B

  • A -> B

Requirements to demonstrate causality?

  • Correlation

  • Order. A must precede B.

  • Control over other variables

 The Experimental Method

2. How are experiments different from other types of research?

  • Manipulated independent variable (treatment)

  • Control of subject variables either by:

    • Random assignment of units of analysis to conditions of the independent variable, or

    • Assignment of each unit to all conditions, with controls on order of presentation

 The Experimental Method

2. How are experiments different from other types of research? (continued)

  • Control of other variables by holding them constant

  • “In an airtight experiment, there is only one rival hypothesis: chance.”

Strengths and Weaknesses of Experiments

Strengths Weaknesses
Control Artificiality
Ability to demonstrate causality Lack of external validity

On Internal Validity and External Validity

Internal validity addresses the question on whether the experimental treatments
in fact make a difference in this specific experimental instance.

External validity regards the question of generalizability: to what populations, settings, treatment variables, and measurement variables can this effect be generalized.  (Campbell and Stanley 1966)

in fact the experimental treatments

make a difference in this specific experi- mental instance?

On External Validity of Experiments

“the conventional survey interview, though well equipped to assess variations among individuals, is poorly equipped to assess variation across situations.”

 

- Sniderman et al. (1991: 265)

On External Validity of Experiments

Unlike most controlled lab settings, researchers using survey experiments have limited ability introduce contextual variations.

 

- Druckman and Kam 2009

Compare experiments with other data methods

  • Web data

  • Observations

  • Survey

  • Expert interviews

Experimental design

  • Control

  • Sampling via randomization

  • Learn from repeated experiments 

    • Adaptive Clinical Design

  • Machine Learning

Data quality

is a function of:

  • Theory-driven

  • Causality

  • Design

  • Control

  • Sampling

Statistical Learning Process

Source: Seltman, H.J., 2012. Experimental design and analysis. Pittsburgh: Carnegie Mellon University, 428.

Reference

Campbell, D.T. and Stanley, J.C., 1966. Experimental and quasi-experimental designs for research. Handbook of research on teaching (NL Gage, Ed.), pp.171-246.
Druckman, J.N. and Kam, C.D., 2009. Students as experimental participants: A defense of the 'narrow data base'.
Morton, R.B. and Williams, K.C., 2010. Experimental political science and the study of causality: From nature to the lab. Cambridge University Press.
Seltman, H.J., 2012. Experimental design and analysis. Pittsburgh: Carnegie Mellon University, 428.