Designing a Test

FYS 100, Module 2

Some observable behavior

Some insight about the person who performed the behavior

What is it that makes this a test?

Today we will talk about arguably the most structured test:

the questionnaire, survey, or self-report instrument.

 

A few notes:

  • We will use these words interchangeably.
  • The basic definition we will use is: a structured testing instrument which consists of prompts for obtaining information from respondents.
  • Can contain open-ended and closed-ended questions
  • Can be written, spoken, or even administered pictorially 

 

Question: Do academic tests or tests of aptitude count?

Answer: For right now, yes.

Steps to making a test

  1. Decide what you are interested in
  2. Decide how to ask it

 

This is pretty much the only set of steps that encompasses every test-making process. Unfortunately, neither step is simple!

Deciding what you are interested in

  • This isn't as simple as it sounds
  • In particular, you want to define and operationalize the construct
    • Construct: The clearly-defined trait, state, or experience that underlies our more observable behaviors
    • Operationalization: As complete a set as possible of the observable, measureable manifestations of that construct
  • For example, let's use the example of agreeableness. A recent encyclopedia on the topic says:
    • Construct: Agreeableness
    • Definition: Agreeableness is defined as the motivation to maintain smooth interpersonal relationships.
    • Operationalization:  Individuals high in Agreeableness are kind, warm, helpful, and accommodating to others... Agreeableness has been linked to prosocial behaviors including cooperation, adaptive conflict resolution, emotion regulation, and holding fewer prejudices.

Deciding what you are interested in

  • You may draw from your personal experiences to get the idea, but lots of research goes into this!
    • Talking to experts
    • Differentiating from other constructs
      • You probably aren't the first person to think about this
    • Reading a lot
  • The basic idea is to figure out whether and how to make a "thing" of the thing you're interested in!

Deciding how to measure it

  • You build a test the same way you build anything else: iteratively

(Re-)write the test

Administer the test

Here you're getting all of the reliability and validity evidence we will talk about!

Anatomy of a questionnaire*

  • Instructions
    • Typically precede some set of items, often the whole test, telling the respondent how to interpret and respond to the items
  • Item stem
    • The statement or question to which the respondent must respond
  • Response options
    • The set of options the respondent may use to respond in a closed-ended test
    • Note that in an open-ended questionnaire there are no response options -  the respondent can just respond!

*We'll focus on questionnaires today but not always!

Below you will see a number of statements about how you have been feeling this past week. Please circle the response which best represents your level of agreement or disagreement with each statement. 

I have been tearful this past week. 

Strongly disagree       Disagree         Neither agree nor disagree        Agree          Strongly agree

Instructions

Item stem

Response options

Things to consider

  • Response options
    • "strongly disagree" to "agree"? "never" to "always"?
    • How many should there be?
  • Conditions in which the test will be administered
    • On paper? In conversation?
  • Length
    • Too short = not enough information
    • Too long = respondent gets exhausted or frustrated
  • Trade-offs between face validity and obviousness of question
    • How obvious do you want your construct to be? 
    • Think of the Rosenberg Self-Esteem Scale

Writing good questions: Do's

  • Use language your target audience can understand
  • Ask for only one piece of information at a time
    • Opposite of this is "double-barreled question" which we will see in the next slide
  • Ask questions that are at the appropriate level of specificity
    • The most specific option isn't always the right one!
      • e.g., "On how many days last month did you feel anxious?" vs. "How often did you feel anxious last month?"
  • Ask the respondent to recall concrete facts when possible
    • ...definitely not always possible (e.g., the previous bullet point)
  • Consider reverse-coded items
  • Measure social responding
    • People want to look good, even if you tell them the test is just for research or is anonymous!

Writing good questions: Don't's

  • Use negatives except when necessary
    • Note that this is not the same as using a reverse-scored item
  • Use double-barreled items
    • e.g., "I am typically respectful of people's time and am seldom late"
  • Rely on content knowledge the respondent might not have
    • This includes asking participants to speculate about something they might not think much about
    • Do not assume that it will make sense to them if it makes sense to you!

Below you will see a number of statements about how you have been feeling this past week. Please circle the response which best represents your level of agreement or disagreement with each statement. 

I am frequently kind and solicitous to others. 

Strongly disagree       Disagree         Neither agree nor disagree        Agree          Strongly agree

Suppose we are trying to measure agreeableness. Let's rewrite the following item.