Survey Design

Karl Ho

School of Economic, Political and Policy Sciences

University of Texas at Dallas

Survey Research

To start a new survey:

  • Is it an ongoing survey or a one-time survey?
  • What is the target population (whom is it studying)?
  • What is the sampling frame (how do they identify the people who have a chance to be included in the survey)?
  • What is the sample design (how do they select the respondents)?
  • What is the mode of data collection (how do they collect data)?

Budget factors

  • Staff time for planning and administration
  • Sample selection costs
    • Under/over-representative segments
  • Interview administration (if F2F)
  • Cost of “cleaning” the final data
  • Analyst costs
  • Reporting

Survey Design

  1. What is the population?
  2. How to sample?
  3. Anticipated data
  4. Single mode or mixed mode
  5. Single wave or multi-wave
  6. Pilot

Population

  1. Concept of Inference

  2. Sample and Population

  3. What is a panel?

Two Types of Survey Inference

Two Types of Survey Inference

Instrument:

  • Questionnaire design
  • Wording
  • Order
  • Scale

Two Types of Survey Inference

Sampling:

  • Target population
  • Sampling frame
  • Sampling strategy & methods
  • Panel

Two Types of Survey Inference

Two needed characteristics of a survey:

  1. Answers people give must accurately describe characteristics of the respondents.

  2. The subset of persons participating in the survey must have characteristics similar to those of a larger population.

Population

It is imperative to understand the concept of inference and how much the respondents provide:

  1. Answers accurately describe characteristics of individual respondent
  2. ​Answers representative of the population

 

Survey Process

Launch

Launch

Instrument

Sampling

Survey Measurement Design

  • Construct refers to a representation of a concept, which could be observed or latent.   
  • For example, identity ambiguity, apathy, political efficacy.
  • Construct may not be directly observable or measurable. 

Construct

Survey Measurement Design

  • Measurements are more concrete than constructs.
  • Measurements in surveys are ways to gather information about constructs.  
  • Questions used for measuring the construct provide data for "indicators".

Measurement

Survey Measurement Design

Sources of error in a survey 

Instrument

Sampling

Confirmatory Factor Analysis (CFA)

Missing value analysis and multiple imputations 

Weighting (e.g. raking)

  • Confirmatory Factor Analysis (CFA) is a theory-driven factor analysis.  Unlike Exploratory Factor Analysis (EFA), CFA verifies the factor structure taking into account the measurement errors, theory guided factor loadings and the iterative process of building a “measurement” model.
  • EFA atheoretically provides the number of factors based on different rotation methods and variable communality/uniqueness.  It regards no concept of measurement error.
  • In the SEM framework, researchers are concerned not only about the number of factors, but also the covariance structure and what we make of the structure.  This approach investigates the measurement errors, latent constructs and their structural relations. 

Illustration: CFA

Illustration: CFA

Illustration: CFA - two factors

Illustration: Structural model

  1. Valid measure
  2. Reliable measure                                  

Survey Measurement Design

  1. Valid measure
    • Validity is measuring what is supposed to be measured.
    • Example:   
      • Wealth vs. Income
      • Happiness vs. Satisfaction             

Survey Measurement Design

2. Reliable measure

  • Reliability is measuring well what is supposed to be measured.
  • Consistency
  • Methods
    • ​Cronbach alpha

Survey Measurement Design

  1. Question wording
    1. Closed-ended questions
    2. Open-ended questions
  2. Question order
  3. Answer order

 

Questionnaire bias

  1. Avoid leading questions
    1. E.g.
      The government should force you to pay higher taxes.

Question wording

Keep the questions short

  • Try under 25 words

  • Short and easy to understand

  • Ensure there is no errors in spelling

 

Question design

    Start designing by thinking of the answers
    1. Scale (Likert, Thermometer)
    2. Mutually exclusive choices
    3. Allow multiple selection?

Question wording

  1. Exhaust all possible answers
    • Add an open ended choice

Question wording

  1. Use even number of choices
    1. To avoid middle choice inertia
    2. Allow nonresponse

Question wording

Avoid beginning questions with answers:

  • Do you very often, frequently, seldom or never.....

  • Use:

    How often do you ____?  Very often, frequently, seldom or never. 

 

Question design

Open-ended question(s)

Places open-ended question at the end of a section.

Question design

Open-ended question(s)

Places open-ended question at the end of a section.

Question design

Open-ended answer(s)

When the question may not provide an exhaustive list of answers, provide open-ended choice.

Question design

Likert scale: middle choice?

  • Some researchers prefer even number of choices to avoid tendency to choose the middle answer. If the question begs for a more definite positioning of the respondent, use even number.

  • Some questions are more difficult for choosers, allow odd number in this case.

Question design

  • Bipolar scale

    • Scales in agreement with bipolar requests and bipolar concepts should measure two poles: positive to negative or active to passive.
    • Example:
    • "How satisfied or dissatisfied are you with the present state of the economy in your country?"
      • Bipolar scale:
        1. Extremely dissatisfied
        2. Dissatisfied
        3. Neither satisfied nor dissatisfied
        4. Satisfied
        5. Extremely satisfied

Scale

  • Unipolar scale

    • Scales in which only one of the poles is used in the responses.
    • Example:
    • "How satisfied are you with the present state of the economy in your country?"
      • Unipolar scale:
        1. Not at all satisfied
        2. Fairly satisfied
        3. Very satisfied
        4. Extremely satisfied

Scale

  • The concept to be measured can be either bipolar or unipolar. Bipolar concepts have two theoretical opposite poles (e.g. positive/negative or active/passive), while unipolar concepts have only one theoretical pole. 

Bipolar and Unipolar concepts

  1. Comprehension

  2. Retrieval

  3. Judgement

  4. Response 

Survey response process

Four major components

- Tourangeau, Rips, and Rasinski (2000)
The Psychology of Survey Response

Learn from the giants

 

Borrow from existing surveys such as GSS, ANES, TEDS since they are well tested and crafted by experienced researchers.  Alternatively, modify or adapt from the question wording styles.

Question design

Aesthetic consideration

 

Allow space for respondents to feel comfortable.  Make the page look professional with elegant design.  Try multiple color (but not too many).

Question design

Aesthetic consideration

 

Allow space for respondents to feel comfortable.  Make the page look professional with elegant design.  Try multiple color (but not too many).

Question design

"Ambiguity is the ghost most difficult to exorcise from survey questions."


–Czaja & Blair (2005)

Question design

Respondents must use mental processes to read and understand a question (along with any relevant instructions), inferring the main idea of the question and identifying what the researcher is looking for regarding a response. 

Survey response process

Comprehension

Survey response process

Comprehension

Problems with comprehension arise when respondents:

  1. do not notice, do not read, or misinterpret instructions;

  2. encounter unfamiliar vocabulary in a question stem or response options;

  3. interpret words or phrases differently than the way the researcher intended; or

  4. find a question worded in an overly complex or detailed way.

Survey response process

Retrieval

Retrieval “requires recalling relevant information from long-term memory. This component encompasses such processes as adopting a retrieval strategy, generating specific retrieval cues to trigger recall, recollecting individual memories, and filling in partial memories through inference” (Tourangeau et al., 2000, p. 9). 

Survey response process

Retrieval

A retrieval strategy might be the particular way in which we attempt to remember something. For example, if you were asked to think of as many parties as you can, you might think in terms of political leaders. 

Survey response process

Judgement

Tourangeau et al. (2000) divide judgement into three distinct types:

  1. factual questions

  2. dates and durations

  3. frequencies. 

Survey response process

Judgement

Tourangeau et al. (2000) divide judgement into three distinct types:

  1. factual questions

  2. dates and durations

  3. frequencies. 

Survey response process

Response

Response requires selecting and reporting an answer to a survey question. Responding to a survey question involves groups of processes around “mapping” the answer to available response options (as with a multiple-choice type question) and “editing” the answer to meet certain criteria.

  1. Willingness

    1. Question types

    2. Question wording

    3. Social desirability (Hawthorne effect)

  2. Ability

    1. Memory

    2. Comprehensibility

    3. Culture

Respondent Willingness and Ability to Participate in a Survey

Robinson, S.B. and Leonard, K.F., 2018. Designing Quality Survey Questions. SAGE Publications.