Usability Testing

CMPT 363

Use testing to guide design, not to grade designers.
– Jeff Johnson

 

Where Are We?

What is Usability Testing?

 
Simply put, having actual users try to complete specific tasks
with your product or service

Usability Testing Matters

e.

Copyright by respective copyright owners. Used without permission under the Fair Use Doctrine. Source: http://civicdesigning.org/

Types of Usability Testing


  • Summative (verification)
    • Usability tests (often more formal) at the end of a development process or on a released product to assess if the interface meets pre-defined standards
    • Quantitative measurements (stats) are often the focus

Types of Usability Testing


  • Formative (diagnostic)
    • Usability tests (often less formal) during the design process to help identify usability issues before development is finalized or a product is released
    • Qualitative insights (details) are often the focus

Why Conduct Usability Tests?


Learning opportunity for even the most experienced designers

Why Conduct Usability Tests?


A very effective form of risk mitigation

Why Conduct Usability Tests?


Insight into the critical “first-use” user experience

Why Conduct Usability Tests?


Often the process of preparing for the test will result in discovering areas which can be improved

Why Conduct Usability Tests?


All products and services will be usability tested once released!

What are the Limitations of Usability Testing?


  • Often done in artificial test environments (unless conducted remotely or on-location)
  • Not really proof that the product works
  • Participants are often not actual users
  • Time consuming and difficult to properly conduct extensive comparative tests
  • Measures learnability rather than overall usability
  • Not a replacement for interface design skills

Forms of Usability Tests


  • Paper and pencil
  • 5 second test
  • “Greeked” layout/X-test
  • Observational test

Paper and Pencil (aka Sketches)


  • Representative users are shown very early versions of interface design elements
  • Users are then asked a series of task oriented questions

Example Task-oriented Questions 

 
  • What do you think is the primary purpose of this screen?
  • What do you think about the overall organization of this screen?
  • What might you do to start the task of …?
 

Activity: Task-oriented Questions

Review your assignment #4 sketches​ in terms of how well the interface supports the following questions:

 

Primary purpose of this screen?
Overall organization of this screen?
Start the task of …?

5 Second Test


Show an image of your design to users for only 5 (five) seconds, and ask for their initial impressions

Greeked” Layout/X-test

(Formative Tests)


The visual placement of elements on a screen
alone contributes to usability

Greeked” Layout/X-test

(Formative Tests)


This technique is sometimes also called the X-test, where all characters on the screen are replaced with X’s
 


Observational Test


A representative user is given a task to perform with the interface, and the evaluator passively observes

Observational Test


If other colleagues are to be present, be sure to outline procedural expectations for the test

Hawthorne Effect


A famous series of experiments were conducted in 1920’s and are often cited as proof that people being observed may try harder than usual to achieve their tasks

Hawthorne Effect


This interpretation as applied to usability studies remains controversial, as the original study was longitudinal (same task over time) and participants were also task experts

Time for Questions


  • What we’ve covered so far
    • What is usability testing?
    • Paper and pencil
    • 5 second test
    • “Greeked” layout/X-test
    • Observational test
  • Coming up
    • Pre-test activities

Pre-Test Activities


  • Define test purpose and concerns
  • Identify test participants
  • Create demographics (user profile) questionnaire
  • Decide on test method
  • Select tasks to be performed
  • Create usability test tasks
  • Conduct pilot test

Define Test Purpose


When defining the purpose for a usability test, you are identifying the overall goal for the test

Define Test Purpose


For example, “The purpose of this usability test is to get an early assessment of the overall usability of the software product XYZ”

Define Test Concerns

 
When defining concerns for a usability test, you are identifying the specific aspects of the interface you want to learn more about
 

Define Test Concerns

 
For example, “Can participants successfully submit an order form?” or “Do participants have any difficulty with terms and phrases?”
1

Define Usability Targets
(Summative Tests)


  • Elements of a Usability Target:
    • Condition (a)
    • Criteria (b)
    • Performance (c)
  • Example Usability Target (e.g. effectiveness):
    • With at least 3 months of Windows experience (a), at least 80% of participants (b) should be able to complete the task of adding a new contact name (c)

Key Usability Target Metrics


  • Effectiveness (e.g. 80% of participants can complete the task without assistance)
  • Efficiency (e.g. 70% of participants can complete the task in under 10 minutes)
  • Attitude (e.g. after performing three tasks, 85% of participants rated their overall satisfaction with the software at least 5.5 out of 7)

Identify Test Participants


Actual users of your product are the best participants for observational usability tests

Identify Test Participants


If actual users are not available, the test participants chosen should be as representative of actual users as possible

Sources for Test Participants

 
  • Contacts via marketing or customer support
  • Colleagues who are representative users
  • Recent employees of your company
  • “Dash and grab” company hallway method
  • Anywhere you can find them (i.e. Guerilla usability testing)​ 

Number of Test Users vs.
Usability Problems

 

Copyright by respective copyright owners. Used without permission under the Fair Use Doctrine. Source: http://www.nngroup.com/articles/

How Many Users, Really?


Rolf Molich (co-inventor with Jakob Nielsen of original usability heuristics), more recently stated that “five users are enough to drive a useful iterative cycle” (based on his CUE studies)

User Demographics (User Profile)

Questionnaire

 
A questionnaire should be used to collect and summarize demographic information about all the participants
involved in the test
 

Decide on Test Method


  • Simple observation
  • Think-aloud method

Simple Observation Method


Most basic type of observationalusability test
Test participant is observed with as little interaction as possible

Think-aloud Method


  • A variation of the simple observational test, where participants are asked to think out loud (describe what they are thinking) while performing tasks
  • Most widely used method in thesoftware industry

Think-aloud Method


Method is not appropriate if the efficiency (i.e. time on task) is being measured, unless done in a retrospective manner

Prompting Test Participants
to Think-aloud


  • What they believe is happening
  • What they are trying to do
  • Questions that arise in their mind
  • Things that they find confusing
  • Why they decided to try a specific action

Think-aloud Method


(+) Can reveal user’s mental model
(+) Can be used early in design process
(+) Precise pointers to problems
(–) Unnatural situation
(–) Measurement of time on task not reliable
(–) Requires more participant effort

    Co-discovery Method
    (Formative Tests)


    A variation of the think-aloud method, where two participants jointly work together on specified tasks

    Co-discovery Method
    (Formative Tests)


    Normal conversation between the two participants provides a more natural way of revealing internal thought processes

    Decide on Test Environment/Equipment

     
    • In-person vs. remote (via screen sharing software)
    • Paper-based mockups vs. prototype vs. actual system

    Select Tasks to Perform


    In general, either frequent tasks and/or important tasks should be selected for inclusion into usability tests

    Select Tasks to Perform


    Ensure that a very simple task is at the beginning of the usability test

    Create Usability Test Tasks


    • Tasks should start with a written scenario, describing one or more specific goals
    • Communicate intent, but not the operations of what is required to be done
    • Avoid using words contained in any user interface elements

    Example Usability Test Task


    “You are planning to visit family on the east coast, but before you do so you need to temporarily stop your newspaper delivery”

    “Cancel your Vancouver Sun newspaper delivery between Oct 10th to 29th.

    Activity: Usability Test Tasks

    Review your assignment #4 sketches​ and create a possible usability test task based on a core job story

    Video Taping (Summative)


    (+) Written notes are not required
    (+) Capture of minute details
    (+) Easily shared with other team members
    (+) Effective tool for upper-management support
    (–) Consent must be obtained
    (–) Participants likely to be less comfortable
    (–) Time consuming to analyze (3-10X times)
    (–) Often done in artificial environments (e.g. lab)

      Conduct Pilot Test


      • Identifies issues with test materials (e.g. sequence, wording, etc.)
      • Validates time estimates for tasks/session
      • Ensures that the mock-up/software is capable of completing specified tasks

      Create a Usability Test Plan


      • Test purpose
      • Test concerns
      • Usability targets (summative tests)
      • Participant user profile
      • Test method
      • Test environment and equipment
      • Test tasks
      • Evaluation measurements

      Time for Questions


      • What we’ve covered in this section
        • Defining test purpose and concerns
        • Identifying test participants
        • Creating user profile questionnaire
        • Deciding on test method
        • Selecting tasks to be performed
        • Creating usability test tasks
        • Conducting a pilot test
      • Coming up
        • Observational test activities

      Observational Test Activities


      • Welcome participant
      • Introduce the software
      • Provide written tasks
      • Observe participant
      • Participant suggestions
      • Test completion

      Welcome the Participant


      Greet the participant, and provide a high-level overview of the software product being tested

      Welcome the Participant


      Inform participants that the product’s user interface is being tested, and not them

      Welcome the Participant


      Explain that you cannot provide help to the participant during the test, as this would bias the test results

      Welcome the Participant


      Provide a signed consent form, if appropriate

      Welcome the Participant


      • Other ethical considerations
        • Don’t use the term “test subject”
        • Test participants can end the test at any time
        • Results will remain anonymous (use a participant coding scheme)

      Introduce the Product


      A brief preview or demonstration of the software product might be appropriate

      Introduce the Product


      Be sure that none of the elements included in the usability test are demonstrated or discussed

      Activity: Compare Two 
      Usability Test Introductions

       

      Moderating Usability Tests:
      Principles and Practices for Interacting - Videos 

      Introduction

      Pre-Test Script 
      Pre-Test No Script
      Reactions Pretests 

      Provide Written Tasks


      Provide participants with a written copy of each task (one at a time) that you want them to try to perform

      Provide Written Tasks


      Remember, these tasks should be written more as scenarios describing specific goals, and not operational instructions

      Observe the Participant


      Carefully observe the participants as they try out your software, and have them indicate when they have completed each task

      Observe the Participant


      Consider having another person on your team take notes of actions each participant tries to perform with the software

      Things to Watch Out For


      • Inconsistencies between software and participant’s vocabulary
      • Screens that participants visibly paused on
      • The type and number of mistakes eachparticipant makes

      How to Make Your
      Observations Effective?


      Let participants make mistakes

      How to Make Your
      Observations Effective?


      Record, but do not answer, questions (alternative is to answer them with questions)

      How to Make Your
      Observations Effective?


      Do not tell participants what to do!

      How to Make Your
      Observations Effective?


      Be careful not to give subtle cues, for example raising your voice often signals agreement

      How to Make Your
      Observations Effective?


      Try not to be defensive, remember you are trying to create a better user experience!

      Handling Requests for Help


      Reiterate in a very friendly manner that you cannot provide assistance, as doing so would bias the test results (note help requests)

      Handling Requests for Help


      If your initial response is not sufficient,
      state “Please remember, there’s no rush– take your time...”

      Handling Requests for Help


      If assistance is provided, clearly indicate in your notes when and where participant received help

      Activity: Moderating a Usability Test — Asking for Assistance

      Moderating Usability Tests:
      Principles and Practices for Interacting - Videos
       

      Participant Suggestions


      Participants will often make specific suggestions regarding the usability issues that they encounter

      Participant Suggestions


      These suggestions should be reviewed, but not blindly followed as solutions to problems

      Task Completion


      • Once the test is complete, thank participants for their time and have them immediately fill out a post-test questionnaire before engaging in any follow-up discussions
      • Provide participants with at least a token gift for their assistance

      Post-Test Questionnaire


      • A small number of rating scales
      • Features to add/remove or best/worst features of interface (open ended)
      • Blank “other” or “comments” field
      • Consider including questions on each task sheet/card
      • Use existing questionnaire when possible (e.g. Single Ease Question after each task, System Usability Scale after all tasks)

      Single Ease of Use Question


      • Overall, this task was?
        • Very difficult
        • Somewhat difficult
        • Neither difficult nor easy
        • Somewhat easy
        • Very easy

      Create a Usability Test Script For More Consistent Test Activities


      • Introduction
      • Session details, including participant rights
      • Check if participant has any questions
      • Pre-test survey/questions (if applicable)
      • Usability test tasks
      • Post-test survey/questions (if applicable)
      • Session wrap-up, including thanking participant

      Time for More Questions


      • What we’ve covered in this section
        • Welcoming participant
        • Introducing the software
        • Providing written tasks
        • Observing participant
        • Participant suggestions
        • Test completion
      • Coming up
        • Post-test activities

      Post-test Activities


      Summarize and present the results to your team, and consider sharing them with all project stakeholders

      Post-test Activities


      Ensure that there is time to use the results when conducting formative tests!

      Sharing Results - Brief Report


      • Executive summary, including highlighting of positive attributes
      • Methodology (participant profiles, test activities, and data collected)
      • Key findings and recommendations

      Key Findings and Recommendations


      • List of major issues
      • Prioritized recommendations (if appropriate)
      • Task success matrix
      • User quotes
      • Post-test questionnaire results

      Task Success Matrix


        Task List Task #1 Task #2 Task #3
        User #1 Success Success Success
        User #2 Success Failure Partial
        User #3 Partial Success Success

      Overall success rate = (6 + (2 *.5)) / 9 = 78% (7/9)

      Success vs. Failure


      • Success
        • Participant was able to complete task without using on-line help or requesting assistance
      • Partial
        • Participant had to use on-line help, or had to try one or more actions before succeeding
      • Failure
        • Participant required help from facilitator, or was unable to complete requested task

      Usability Test Report for
      a Summative Test


      • Common Industry Format (ISO/IEC 25062:2006)
      • Title page
      • Executive summary
      • Introduction (full product description and objectives)
      • Context of test (participants, tasks, environment, etc.)
      • Experimental design (independent/dependant variables, measurements, procedure, general instructions, etc.)
      • Results (data analysis, presentation of results , etc.)
      • Appendices (detailed instructions, surveys, etc.)

      Summary

       
      • What is usability testing?
      • Paper and pencil
      • 5 second test
      • “Greeked” layout/X-test
      • Observational test
      • Pre-test activities
      • Observational test activities
      • Post-test activities
      • Sharing results

      References and Suggested Books

       
      • Elements of User Interface Design, by Theo Mandel
      • The Usability Kit, by Gerry  Gaffney and Daniel  Szuc
      • Usability Engineering, by Jacob Nielsen
      • Comparative Usability Evaluation - CUE, by Rolf Molich
        (
        http://www.dialogdesign.dk/cue.html)
       

      CMPT-363-153 Usability Testing

      By Paul Hibbitts

      CMPT-363-153 Usability Testing

      • 3,351