Validity

FYS 100, Module 2

Are we measuring what we intend to measure?

This is the question of validity*. 

* Confusingly enough, it is sometimes referred to as construct validity, but here we will use that term to refer to something more specific. More on that in a minute.

Does our scale give us the correct weight?

Does our heart risk screening tool predict heart attacks?

Does our IQ test tell us how intelligent people are?

Does our tool for measuring well-being actually find people with high levels of well-being?

Does our test for agreeableness accurately tell us how agreeable people are?

• Question: Does our test look like it measures what we intend it to measure?
  • This is typically the first question we ask, but it isn't necessary that a test have face validity to have the other kinds of validity!
  • Think of tests involving deception. Do they have face validity?
• Examples
  • We are attempting to measure extraversion. Participants see a bunch of questions that ask about how much they enjoy socializing or spending time with others. It looks like a test that measures extraversion. It has high face validity.
  • We are attempting to measure math ability. Participants are required to do an interpretive dance and name the 50 states in the United States. It has low face validity.

Face validity

• Question: Does our test measure every aspect of what it intends to measure, as opposed to just some aspects?
• It can be hard to determine this one. Often we have to have a panel of experts look at the test or consult the literature.
  • Can't rely on correlations like we can with the next few types
• Examples
  • We are attempting to measure extraversion. Participants see a bunch of questions that ask about how much they enjoy going to large group gatherings such as parties or sporting events. This may measure some aspect of extraversion, but there are other aspects, but not all aspects. It has low content validity.

Content validity

• Question: Does the score on this test correlate with other test results with which it should correlate?
• Logic: If we're measuring what we intend to measure, the results of our test should be related to certain things. 
• Examples
  • We are attempting to measure extraversion. We know that in real life, people who are more extraverted are also more sociable. So we take the results of our test and obtain a correlation between that test and a test of sociability.
  • We are attempting to measure math ability. We know that people who are good at math also tend to have higher IQ's overall. So we take the results of our test and obtain a correlation between that test and an IQ test.

Convergent validity

• Question: Does the score on this test not correlate with other test results with which it should not correlate?
• Logic: If our test is correlated with certain things with which it shouldn't be correlated, it may be the case that it's picking up on something else - not the thing we're trying to measure!
  • This is a bit more subtle than convergent validity
• Examples
  • We are attempting to measure math ability. However, we are worried that some of our word problems require too much verbal ability. So we obtain a correlation between scores on our test and scores on a test of verbal ability. When we find that the correlation is quite low (maybe r = .2 or so) we are reassured that our test isn't just measuring how well someone reads.

Discriminant validity

• Question: Does the score on this test correlate with other external criteria, measured at the same time as our test, with which it should correlate?
• Logic: If we're measuring what we intend to measure, the results of our test should pick out people who possess certain real world characteristics.
• Examples
  • We are attempting to measure extraversion. We know that in real life, people who are more extraverted tend to talk to more people over the course of their day. So we take the people to whom we administered our test and ask them how many people they talk to over the course of their day. There is a high correlation between extraversion and the number of people someone speaks to in a day.

Concurrent validity

• Question: Does the score on this test correlate with other external criteria, measured after we administer our test, with which it should correlate?
• Logic: If we're measuring what we intend to measure, the results of our test should pick out people who do certain things (i.e., things related to the content of our test) later on.
• Examples
  • We are attempting to measure math ability. We administer our math test to high school seniors. We then follow up with those high school seniors once they are seniors in college. It should be the case that more of the respondents who were high in math ability are math majors, relative to those who were low in math ability. They should also have higher math GPA's.

Predictive validity