Paul Hibbitts
Educator, interaction design practitioner and software developer.
Usability Testing
CMPT 363
“Use testing to guide design, not to grade designers.”
– Jeff Johnson
Where Are We?
What is Usability Testing?
Simply put, having actual users try to complete specific tasks
with your product or service
with your product or service
Usability Testing Matters
e.
Copyright by respective copyright owners. Used without permission under the Fair Use Doctrine. Source: http://civicdesigning.org/
Types of Usability Testing
-
Summative (verification)
- Usability tests (often more formal) at the end of a development process or on a released product to assess if the interface meets pre-defined standards
- Quantitative measurements (stats) are often the focus
Types of Usability Testing
-
Formative (diagnostic)
- Usability tests (often less formal) during the design process to help identify usability issues before development is finalized or a product is released
- Qualitative insights (details) are often the focus
Why Conduct Usability Tests?
Learning opportunity for even the most experienced designers
Why Conduct Usability Tests?
A very effective form of risk mitigation
Why Conduct Usability Tests?
Insight into the critical “first-use” user experience
Why Conduct Usability Tests?
Often the process of preparing for the test will result in discovering areas which can be improved
Why Conduct Usability Tests?
All products and services will be usability tested once released!
What are the Limitations of Usability Testing?
-
Often done in artificial test environments (unless conducted remotely or on-location)
-
Not really proof that the product works
-
Participants are often not actual users
-
Time consuming and difficult to properly conduct extensive comparative tests
-
Measures learnability rather than overall usability
-
Not a replacement for interface design skills
Forms of Usability Tests
-
Paper and pencil
-
5 second test
-
“Greeked” layout/X-test
-
Observational test
Paper and Pencil (aka Sketches)
-
Representative users are shown very early versions of interface design elements
-
Users are then asked a series of task oriented questions
Example Task-oriented Questions
- What do you think is the primary purpose of this screen?
- What do you think about the overall organization of this screen?
- What might you do to start the task of …?
Activity: Task-oriented Questions
Review your assignment #4 sketches in terms of how well the interface supports the following questions:
Primary purpose of this screen?
Overall organization of this screen?
Start the task of …?
5 Second Test
Show an image of your design to users for only 5 (five) seconds, and ask for their initial impressions
“ Greeked” Layout/X-test
(Formative Tests)
The visual placement of elements on a screen
alone contributes to usability
alone contributes to usability
“ Greeked” Layout/X-test
(Formative Tests)
This technique is sometimes also called the X-test, where all characters on the screen are replaced with X’s
Observational Test
A representative user is given a task to perform with the interface, and the evaluator passively observes
Observational Test
If other colleagues are to be present, be sure to outline procedural expectations for the test
Hawthorne Effect
A famous series of experiments were conducted in 1920’s and are often cited as proof that people being observed may try harder than usual to achieve their tasks
Hawthorne Effect
This interpretation as applied to usability studies remains controversial, as the original study was longitudinal (same task over time) and participants were also task experts
Time for Questions
-
What we’ve covered so far
- What is usability testing?
- Paper and pencil
- 5 second test
- “Greeked” layout/X-test
- Observational test
-
Coming up
- Pre-test activities
Pre-Test Activities
-
Define test purpose and concerns
-
Identify test participants
-
Create demographics (user profile) questionnaire
-
Decide on test method
-
Select tasks to be performed
-
Create usability test tasks
-
Conduct pilot test
Define Test Purpose
When defining the purpose for a usability test, you are identifying the overall goal for the test
Define Test Purpose
For example, “The purpose of this usability test is to get an early assessment of the overall usability of the software product XYZ”
Define Test Concerns
When defining concerns for a usability test, you are identifying the specific aspects of the interface you want to learn more about
Define Test Concerns
For example, “Can participants successfully submit an order form?” or “Do participants have any difficulty with terms and phrases?”
1
Define Usability Targets
(Summative Tests)
-
Elements of a Usability Target:
- Condition (a)
- Criteria (b)
- Performance (c)
-
Example Usability Target (e.g. effectiveness):
- With at least 3 months of Windows experience (a), at least 80% of participants (b) should be able to complete the task of adding a new contact name (c)
Key Usability Target Metrics
-
Effectiveness (e.g. 80% of participants can complete the task without assistance)
-
Efficiency (e.g. 70% of participants can complete the task in under 10 minutes)
-
Attitude (e.g. after performing three tasks, 85% of participants rated their overall satisfaction with the software at least 5.5 out of 7)
Identify Test Participants
Actual users of your product are the best participants for observational usability tests
Identify Test Participants
If actual users are not available, the test participants chosen should be as representative of actual users as possible
Sources for Test Participants
- Contacts via marketing or customer support
- Colleagues who are representative users
- Recent employees of your company
- “Dash and grab” company hallway method
- Anywhere you can find them (i.e. “Guerilla” usability testing)
Number of Test Users vs.
Usability Problems
Copyright by respective copyright owners. Used without permission under the Fair Use Doctrine. Source: http://www.nngroup.com/articles/
How Many Users, Really?
Rolf Molich (co-inventor with Jakob Nielsen of original usability heuristics), more recently stated that “five users are enough to drive a useful iterative cycle” (based on his CUE studies)
User Demographics (User Profile)
Questionnaire
A questionnaire should be used to collect and summarize demographic information about all the participants
involved in the test
involved in the test
Decide on Test Method
-
Simple observation
-
Think-aloud method
Simple Observation Method
Most basic type of observationalusability test
Test participant is observed with as little interaction as possible
Think-aloud Method
-
A variation of the simple observational test, where participants are asked to think out loud (describe what they are thinking) while performing tasks
-
Most widely used method in thesoftware industry
Think-aloud Method
Method is not appropriate if the efficiency (i.e. time on task) is being measured, unless done in a retrospective manner
Prompting Test Participants
to Think-aloud
-
What they believe is happening
-
What they are trying to do
-
Questions that arise in their mind
-
Things that they find confusing
-
Why they decided to try a specific action
Think-aloud Method
(+) Can reveal user’s mental model
(+) Can be used early in design process
(+) Precise pointers to problems
(–) Unnatural situation
(–) Measurement of time on task not reliable
(–) Requires more participant effort
Co-discovery Method
(Formative Tests)
A variation of the think-aloud method, where two participants jointly work together on specified tasks
Co-discovery Method
(Formative Tests)
Normal conversation between the two participants provides a more natural way of revealing internal thought processes
Decide on Test Environment/Equipment
- In-person vs. remote (via screen sharing software)
- Paper-based mockups vs. prototype vs. actual system
Select Tasks to Perform
In general, either frequent tasks and/or important tasks should be selected for inclusion into usability tests
Select Tasks to Perform
Ensure that a very simple task is at the beginning of the usability test
Create Usability Test Tasks
-
Tasks should start with a written scenario, describing one or more specific goals
-
Communicate intent, but not the operations of what is required to be done
-
Avoid using words contained in any user interface elements
Example Usability Test Task
“You are planning to visit family on the east coast, but before you do so you need to temporarily stop your newspaper delivery”
“Cancel your Vancouver Sun newspaper delivery between Oct 10th to 29th.
Activity: Usability Test Tasks
Review your assignment #4 sketches and create a possible usability test task based on a core job story
Video Taping (Summative)
(+) Written notes are not required
(+) Capture of minute details
(+) Easily shared with other team members
(+) Effective tool for upper-management support
(–) Consent must be obtained
(–) Participants likely to be less comfortable
(–) Time consuming to analyze (3-10X times)
(–) Often done in artificial environments (e.g. lab)
Conduct Pilot Test
-
Identifies issues with test materials (e.g. sequence, wording, etc.)
-
Validates time estimates for tasks/session
-
Ensures that the mock-up/software is capable of completing specified tasks
Create a Usability Test Plan
-
Test purpose
-
Test concerns
-
Usability targets (summative tests)
-
Participant user profile
-
Test method
-
Test environment and equipment
-
Test tasks
-
Evaluation measurements
Time for Questions
-
What we’ve covered in this section
- Defining test purpose and concerns
- Identifying test participants
- Creating user profile questionnaire
- Deciding on test method
- Selecting tasks to be performed
- Creating usability test tasks
- Conducting a pilot test
-
Coming up
- Observational test activities
Observational Test Activities
-
Welcome participant
-
Introduce the software
-
Provide written tasks
-
Observe participant
-
Participant suggestions
-
Test completion
Welcome the Participant
Greet the participant, and provide a high-level overview of the software product being tested
Welcome the Participant
Inform participants that the product’s user interface is being tested, and not them
Welcome the Participant
Explain that you cannot provide help to the participant during the test, as this would bias the test results
Welcome the Participant
Provide a signed consent form, if appropriate
Welcome the Participant
-
Other ethical considerations
- Don’t use the term “test subject”
- Test participants can end the test at any time
- Results will remain anonymous (use a participant coding scheme)
Introduce the Product
A brief preview or demonstration of the software product might be appropriate
Introduce the Product
Be sure that none of the elements included in the usability test are demonstrated or discussed
Activity: Compare Two
Usability Test Introductions
Moderating Usability Tests:
Principles and Practices for Interacting - Videos
Provide Written Tasks
Provide participants with a written copy of each task (one at a time) that you want them to try to perform
Provide Written Tasks
Remember, these tasks should be written more as scenarios describing specific goals, and not operational instructions
Observe the Participant
Carefully observe the participants as they try out your software, and have them indicate when they have completed each task
Observe the Participant
Consider having another person on your team take notes of actions each participant tries to perform with the software
Things to Watch Out For
-
Inconsistencies between software and participant’s vocabulary
-
Screens that participants visibly paused on
-
The type and number of mistakes eachparticipant makes
How to Make Your
Observations Effective?
Let participants make mistakes
How to Make Your
Observations Effective?
Record, but do not answer, questions (alternative is to answer them with questions)
How to Make Your
Observations Effective?
Do not tell participants what to do!
How to Make Your
Observations Effective?
Be careful not to give subtle cues, for example raising your voice often signals agreement
How to Make Your
Observations Effective?
Try not to be defensive, remember you are trying to create a better user experience!
Handling Requests for Help
Reiterate in a very friendly manner that you cannot provide assistance, as doing so would bias the test results (note help requests)
Handling Requests for Help
If your initial response is not sufficient,
state “Please remember, there’s no rush– take your time...”
state “Please remember, there’s no rush– take your time...”
Handling Requests for Help
If assistance is provided, clearly indicate in your notes when and where participant received help
Activity: Moderating a Usability Test — Asking for Assistance
Moderating Usability Tests:
Principles and Practices for Interacting - Videos
Principles and Practices for Interacting - Videos
Participant Suggestions
Participants will often make specific suggestions regarding the usability issues that they encounter
Participant Suggestions
These suggestions should be reviewed, but not blindly followed as solutions to problems
Task Completion
-
Once the test is complete, thank participants for their time and have them immediately fill out a post-test questionnaire before engaging in any follow-up discussions
-
Provide participants with at least a token gift for their assistance
Post-Test Questionnaire
-
A small number of rating scales
-
Features to add/remove or best/worst features of interface (open ended)
-
Blank “other” or “comments” field
-
Consider including questions on each task sheet/card
-
Use existing questionnaire when possible (e.g. Single Ease Question after each task, System Usability Scale after all tasks)
Single Ease of Use Question
-
Overall, this task was?
- Very difficult
- Somewhat difficult
- Neither difficult nor easy
- Somewhat easy
- Very easy
Create a Usability Test Script For More Consistent Test Activities
-
Introduction
-
Session details, including participant rights
-
Check if participant has any questions
-
Pre-test survey/questions (if applicable)
-
Usability test tasks
-
Post-test survey/questions (if applicable)
-
Session wrap-up, including thanking participant
Time for More Questions
-
What we’ve covered in this section
- Welcoming participant
- Introducing the software
- Providing written tasks
- Observing participant
- Participant suggestions
- Test completion
-
Coming up
- Post-test activities
Post-test Activities
Summarize and present the results to your team, and consider sharing them with all project stakeholders
Post-test Activities
Ensure that there is time to use the results when conducting formative tests!
Sharing Results - Brief Report
-
Executive summary, including highlighting of positive attributes
-
Methodology (participant profiles, test activities, and data collected)
-
Key findings and recommendations
Key Findings and Recommendations
-
List of major issues
-
Prioritized recommendations (if appropriate)
-
Task success matrix
-
User quotes
-
Post-test questionnaire results
Task Success Matrix
| Task List | Task #1 | Task #2 | Task #3 | |
| User #1 | Success | Success | Success | |
| User #2 | Success | Failure | Partial | |
| User #3 | Partial | Success | Success
|
Overall success rate = (6 + (2 *.5)) / 9 = 78% (7/9)
Success vs. Failure
-
Success
- Participant was able to complete task without using on-line help or requesting assistance
-
Partial
- Participant had to use on-line help, or had to try one or more actions before succeeding
-
Failure
- Participant required help from facilitator, or was unable to complete requested task
Usability Test Report for
a Summative Test
-
Common Industry Format (ISO/IEC 25062:2006)
-
Title page
-
Executive summary
-
Introduction (full product description and objectives)
-
Context of test (participants, tasks, environment, etc.)
-
Experimental design (independent/dependant variables, measurements, procedure, general instructions, etc.)
-
Results (data analysis, presentation of results , etc.)
-
Appendices (detailed instructions, surveys, etc.)
Summary
- What is usability testing?
- Paper and pencil
- 5 second test
- “Greeked” layout/X-test
- Observational test
- Pre-test activities
- Observational test activities
- Post-test activities
- Sharing results
References and Suggested Books
- Elements of User Interface Design, by Theo Mandel
- The Usability Kit, by Gerry Gaffney and Daniel Szuc
- Usability Engineering, by Jacob Nielsen
-
Comparative Usability Evaluation - CUE, by Rolf Molich
(http://www.dialogdesign.dk/cue.html)
CMPT-363-153 Usability Testing
By Paul Hibbitts
