Firefox OS
Test Strategy
2013 and Beyond
QA Challenges
Mozilla is Open Source, and the rest of the world isn't.
Mozilla owns the software Stack (Gaia, Gecko, Gonk).
Mozilla doesnt do hardware, firmware.
The developer ratio is about 15:1. maybe more.
Developers dont always recognize our work.
A growing community that we havent touched yet.
Release work
Environment: Devices
- Pre Merge work
- Participate in acceptance criteria for sprint planning
- Identify and critique user stories
- Write test cases in moztrap during sprints
- And automate them in Gaia UI tests (python or javascript)
- Executing mini test runs at the end of each sprint
- Functional Testsuite test runs during the last week of FC date complete (2-3?)
- Branch merge, run a smoke test and
- Execute 3-4 full testruns during the cycle
- 1-2 Localization testruns
- FOTA update testing
- 3rd party apps smoke test (but trying to hand this off)
- Daily Bug Triage work
Our Test Strategy
Release support
Platform integration
Automation
Community
Other stuff
release work
Environment: Devices
- Participate in acceptance criteria for sprint planning
- Identify User stories and Writing test cases in moztrap during sprints
- And automate them in Gaia UI tests (python or javascript)
- Executing mini test runs at the end of each sprint
- Functional testruns during the week of FC (2-3?)
- Branch merge, run a smoke test and
- Execute 3-4 full testruns during the cycle
- 1-2 Localization testruns
- FOTA update testing
- 3rd party apps smoke test (but trying to hand this off)
- Daily Bug Triage work
Platform work
Environment: Emulators
- enable platform tests for B2G (mochitests, layout, reftests, crash, new APIs)
- create device API tests for CRB and test apps
- we need a place to converge these App tests so it can be run by automation and community
Automation
Environment: Devices, Desktop Builds
- Gaia UI tests on current production branch (ie. 1.2, 1.3, etc..)
- Gaia UI tests on master/m-c
- MTBF stability testing across multiple devices
- Eideticker performance testing measure FPS startup and pan/zoom
- Endurance testing on core apps
- future: fuzzing on b2g with marionette (MariFuzz)
- CRB test guidance
Miscellaneous
Environment: Various
- Support Flashing tools for nightly and CI machines
- Assisting partners with automation and test case setup
- Support live issues from partners
- test prototypes
Community
Environment: Devices, Desktop builds, Apps
- Target device testers
- execute moztrap tests
- download and run test Apps, submit test data
- have meet ups and testdays
- Document! (irc, blogs, social networking, Planet, QMO)
- Target automation testers
- set up test environments
- Mentor and review contributing testcases
- Document! (github readmes, comments, bugs)
- Target app developers
- How to properly test apps in HTML5
- How to clone their own test environments we use
January Workweek!
Knowledge Transfer.
Cross product updates and sharing.
Tools Exploration and Enhancements.
Co-locating test environments.
Community Roadmap.
Brainstorm here:
https://etherpad.mozilla.org/b2g-qa-workweek-brainstorm