Joel Maher
Open Source hacker for the Mozilla project.
Metrics for success
How to measure the success of your project
Mobile Test Automation
Development:
- Deliver a first class mobile browser on Android
- Ensure we run tests on every build
Ateam:
- Provide tools to run tests on Android
- Ensure reliable and actionable results
All your tests belong to us
Tests are running, MFBT!
yes, even I have watched this movie
Feel the love
- We found a bug with automation!
- Is anybody looking at these results?
- Why are we getting worse and worse?
- I got backed out due to failures, what now?
Time for a meeting
- Weekly mobile testing meeting
- Cross team attendees
- Discuss priorities, debugging
- Goal of meeting unclear ^ ?
Is this proactive or reactive?
Why am I here again?
Remember this:
We want this to be
We want this to be
...
We want this to be GREEN
AHA!!!
Time for a metric
- Mission: green tests
- Measure how many failures
- Goal: < 5% total failure rate
Total failure rate 7.72%
What does this mean?
Blue links
https://wiki.mozilla.org/Mobile/Testing/06_05_13
Now we understand
http://goo.gl/pyS7KB
What am I missing?
If you are looking for the patentable piece of data there is nothing, this metric is successful because:
- It is easy to understand
- It provides a quarterly goal to track to
- "in Q2 we will have <5% tfr"
- Actionable data can be derived from it
- "mochitest 1 has regressed"
- This is communicated weekly to all parties
Encore
Demo
Metrics for dummies
By Joel Maher
