Caner Uguz
Product Designer at VividCortex
@caneruguz
We are hiring!
Design decisions have expensive and ethical consequences.
You need systematic and cost effective data collection in your decision making process.
CONCEPT
DESIGN
DEVELOPMENT
MAINTENANCE
INFRASTRUCTURE
MANAGEMENT
MARKETING
Apply sound research practices
Use statistical methods to understand your data better
the design, provision, or use of measuring instruments.
cost effective source of information about users
Instrumentation is not direct observation.
Not this
More like this
Be more thoughtful about instrumentation
Improve the quality of your existing methodology
Specific
Measurable
Meaningful
Are my users...
like... happy?
able to find the page that let's them change their date of birth?
Specific
Did my users...
enjoy this article?
spend enough time on this article to read all of it?
Measurable
How many times did my users...
hover over their profile picture?
go the previous page using browser's back button?
Meaningful
Click filter input
Enter 'xyz' word
Hit clear
Click search box
Enter same word
User is feeling generous
User clicks the button to increase donation dollar amount from $2 to $3
Lost?
Action | Value | Average |
---|---|---|
Total number of hitting back and forward in browser | 7 | 1 |
average time spent on pages | 10(sec) | 150 |
total number of pages | 12 | 3 |
total click events on pages | 24 | 12 |
total times hitting 'cancel' or 'esc' | 6 | 1 |
https://pdfs.semanticscholar.org/5bb5/57fdfffd8e186b7fd72ca81b6f1a43fb162b.pdf
https://pdfs.semanticscholar.org/245e/18045e05363a12e163acf388e245e202ec40.pdf
Consistency of a measure
Mostly not an issue on the web if your interaction is programmed for it.
i.e. Lists that appear in an order defined by algorithm
Are you measuring what you intend to
Very commonly an issue
i.e
Photo by Kelly Neil on Unsplash
https://www.surveymonkey.com/mp/sample-size-calculator/
https://www.dummies.com/software/microsoft-office/excel/how-to-use-excels-descriptive-statistics-tool/
https://www.khanacademy.org/math/statistics-probability/summarizing-quantitative-data/box-whisker-plots/a/box-plot-review
What is the likelihood that the result is by chance?
"After we pushed this change our 200 more people signed up per week"
Does not mean the difference is 'important'.
If your sample size is smaller, the likelihood of chance is higher.
What matters to your team?
How do you define success?
What is your desired outcome?
Photo by Eunice Lituañas on Unsplash
Yes but...
Depends on your content
Requires different methodologies
Potentially sensitive data
Works best with categorical inputs
Photo by PNG Design on Unsplash
Run scripts that give you aggregated content about what your users generate
"Our page hits doubled!
... after you introduced forced pagination
"Our users are very engaged with our content"
... because you made it difficult to navigate
"90% of our users read stories about political party A"
... because you decide which articles to show users
"We made our prefix menu shorter"
... by removing Dr. from the list for women "because it literally is never used"
"We increased viewership in more of our movies"
... because you made it harder to find favourites.
"User A seems to be buying a lot of bags, let's show them 10% higher prices than others"
"User B has a great fashion sense, I will keep track of what they are buying"
Photo by Siarhei Horbach on Unsplash
It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do.
However, not everything that can be counted counts, and not everything that counts can be counted.
William Bruce Cameron 1963 text “Informal Sociology: A Casual Introduction to Sociological Thinking