| M | E | T |
|---|---|---|
| R | I | C |
| S |
creating products that are
made to measure
regan davis
VP of Product, Data @Jellyvision
Makers of ALEX, an interactive cartoon for explaining HR & benefits to employees
the lies we tell (ourselves)
reality distortion field
Inconvenient Truths
(not Marty Cagan)
Truth 1:
Truth 2:
It typically takes several iterations to get to the point where it actually delivers the expected business value.
At least half of our ideas are just not going to work.
- Not valuable
- Not usable
- Not feasible
If you've got data, let's go with data.
If all you have are opinions, let's go with mine."
—Jim Barksdale, CEO Netscape
| 1. | Have Goals |
| 2. | Measure Your Bets |
| 3. | Proceed or Pivot |
| 4. | Find the One Metric That Matters |
Have Goals
goals give you focus
goals give you focus
Shift you to outcomes, not features
Stem from your customers' problems
Define a "measure of success"
Your customer does not care about your roadmap
They care about their problem.
you're gonna have to talk to people.
You're looking for
- a problem
- that's painful
- so painful that they try to fix it
- in a really awkward/costly way
your customer's problem
your business goal
meets
how to find the Conversion Point
Ask yourself, "So what?"
Example: Craigslist
- User posted a "For Sale" item
- So what? The user might sell their stuff.
- User got 2 replies to their listing
- So what? The user probably sold their stuff.
- User closed post, confirmed they sold their stuff.
- So what? The user lives happily ever after.
Conversion ("Success") Funnel
My Goal: 3x Sales
Measure your bets
Track Your baseline
Understand what things look like right now
You need to know how you’re growing without intervention, to know what growth actually looks like
Check for Fluctuations
How do your KPIs change over time?
- Are they seasonal? Do they fluctuate by day, week, month?
- Do they fluctuate with your PR or marketing efforts?
Maintenance Goal
80 / 20 Rule (Pareto principle)
Growth Goal
Set as a "check" metric
Need to beat baseline growth
Place for you to experiment and really move the needle
Maintenance Goal
Growth Goal
The 1% rule
Be Aggressive.
Your team will only think as big as you do.
Incremental goals deliver incremental results.
It’s often easier to make something 10x better than it is to make it 10% better.
—Astro Teller, Google[x]
Proceed or pivot
shiny new feature
empirical evidence
all of humanity
Hypothesis-driven development
Each product change should include a hypothesis
important
V
place your bets
IF we (make this change/build X)
THEN we will see (this KPI go up or down)
BECAUSE (user research/analytics/gut that suggest why this outcome will occur)
measure your bets
Proceed: If we reach 70-80% of this goal.
Pivot: Try a different approach if we don't improve by >50%
Push: Tweak and modify if between these numbers
Hypothesis:
"Hosts with professional photography will get more business."
Experiment:
offer hosts professional photographers (concierge MVP)
Metric: average # of bookings is 1.5x vs. non-photographer hosts
Learning:
professional photographer hosts get 2-3x the number of bookings
The one metric that matters
The "One metric that matters" (OMTM)
The one number you're completely focused on above all else for your current stage.
Benefits of OMTM
- Answers your riskiest question
- Forces you to create goals and a line in the sand
- Inspires experimentation
- Focuses the entire company
This metric will change
Solving your most pressing problem -> discovering the next problem
Vanity metrics
beware of
What Makes a good metric?
a rate or ratio
comparative to other time periods or groups
simple and easy to communicate
either:
- makes your predictions more accurate (accounting)
- will significantly change your behavior (experimentation)
The "right" Metric depends on your biz model & stage
Chapter 20, table 20-1
(Honestly, you should just buy the book.)
You will always track multiple #s
Some will be important — these are your KPIs (Key Performance Indicators)
- Track and report on these every day.
Some will help you explore and understand what's behind the metric that matters
Others help you tell your story for investors or customers —store these till later
Capture everything
focus on what's important
but
to learn / explore
to tell your story
to monetize your exhaust
only what you can affect
"would i do something different with this info?"
get business moving in the same direction
Demo
Conduit Labs
Demo
Jellyvision
| Q | U | E |
|---|---|---|
| S | T | I |
| O | N | S |
?
[appendix]
The Four Villains of Decision Making
- we look too narrowly at a problem
- we look for evidence that supports our beliefs
- we let short-term emotions affect our decisions
- we are overconfident
what color should this button be?
we can only get 2 of these 3 before the deadline. is that okay?
sales requested a, b, c, and d. which should we do?
the CEO has a new idea.
Metrics: Creating Products that are Made to Measure
By Regan Davis
Metrics: Creating Products that are Made to Measure
This workshop covers how to set the right measure of success for your users, and use your existing analytics to define the path that successful users are taking. We'll also look at common metrics for different types of products, like marketplaces, apps, and SaaS.
