What does your app really do?…and other adventures in instrumentation
Product Designer at VividCortex
We are hiring!
"I'm hearing great things about our product comparison pages."
"Nobody is doing previews anymore, it's all edit; that's what our app should be."
"Our product detail page is by far the most visited, we can't make any changes to that page."
"People just don't use our app on the phone, we get only 0.5% of hits from mobile devices."
The Case for Instrumentation
Design decisions have expensive and ethical consequences.
You need systematic and cost effective data collection in your decision making process.
Instrumentation is research
Apply sound research practices
Use statistical methods to understand your data better
the design, provision, or use of measuring instruments.
cost effective source of information about users
Instrumentation is not direct observation.
More like this
Be more thoughtful about instrumentation
Improve the quality of your existing methodology
Are your questions
Are my users...
able to find the page that let's them change their date of birth?
Did my users...
enjoy this article?
spend enough time on this article to read all of it?
How many times did my users...
hover over their profile picture?
go the previous page using browser's back button?
Click filter input
Enter 'xyz' word
Click search box
Enter same word
Actions vs constructs
User is feeling generous
User clicks the button to increase donation dollar amount from $2 to $3
|Total number of hitting back and forward in browser||7||1|
|average time spent on pages||10(sec)||150|
|total number of pages||12||3|
|total click events on pages||24||12|
|total times hitting 'cancel' or 'esc'||6||1|
Where are you getting this from?
- Review feedback
- Look back at past users with desired outcome
- From literature
- Past studies, linear regression
Read up on: Scale construction
Consistency of a measure
Mostly not an issue on the web if your interaction is programmed for it.
i.e. Lists that appear in an order defined by algorithm
Are you measuring what you intend to
Very commonly an issue
- Calendar view is a top visited page and therefore very important for users... or is it?
- There are over 4000 events collected in time picker over 5 hours!
- Most common list size is 20 because 80% of people choose 20
Is your sample representative?
Is your sample size big enough?
Make a habit of looking at your data
Box whisker plot
You can't simply compare numbers
What is the likelihood that the result is by chance?
"After we pushed this change our 200 more people signed up per week"
Does not mean the difference is 'important'.
If your sample size is smaller, the likelihood of chance is higher.
What is your research plan?
What is your data like?
Are you asking the right questions?
What matters to your team?
How do you define success?
What is your desired outcome?
Can you instrument qualitative data?
Depends on your content
Requires different methodologies
Potentially sensitive data
Works best with categorical inputs
Photo by PNG Design on Unsplash
Ask your own database
Run scripts that give you aggregated content about what your users generate
Use case: are your charts empty?
Very accurate false data
"Our page hits doubled!
... after you introduced forced pagination
Badly defined constructs
"Our users are very engaged with our content"
... because you made it difficult to navigate
"90% of our users read stories about political party A"
... because you decide which articles to show users
"We made our prefix menu shorter"
... by removing Dr. from the list for women "because it literally is never used"
"We increased viewership in more of our movies"
... because you made it harder to find favourites.
Illegal activities and grey zones
"User A seems to be buying a lot of bags, let's show them 10% higher prices than others"
"User B has a great fashion sense, I will keep track of what they are buying"
Don't be Facebook
Is instrumentation surveillance?
Photo by Siarhei Horbach on Unsplash
It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do.
However, not everything that can be counted counts, and not everything that counts can be counted.
William Bruce Cameron 1963 text “Informal Sociology: A Casual Introduction to Sociological Thinking
What does your app really do?
By Caner Uguz