Firefox OS

Test Strategy

2013 and Beyond




QA Challenges


Mozilla is Open Source, and the rest of the world isn't.

Mozilla owns the software Stack (Gaia, Gecko, Gonk).

Mozilla doesnt do hardware, firmware.

The developer ratio is about 15:1. maybe more.

Developers dont always recognize our work.

A growing community that we havent touched yet.


Release work

Environment: Devices

  • Pre Merge work
  • Participate in acceptance criteria for sprint planning
  • Identify and critique user stories
  • Write test cases in moztrap during sprints
    • And automate them in Gaia UI tests (python or javascript)
  • Executing mini test runs at the end of each sprint
  • Functional Testsuite test runs during the last week of FC date complete (2-3?)
  • Branch merge, run a smoke test and
  • Execute 3-4 full testruns during the cycle
  • 1-2 Localization testruns
  • FOTA update testing
  • 3rd party apps smoke test (but trying to hand this off)
  • Daily Bug Triage work


Our Test Strategy


Release support


Platform integration


Automation


Community


Other stuff



release work

Environment: Devices

  • Participate in acceptance criteria for sprint planning
  • Identify User stories and Writing test cases in moztrap during sprints
    • And automate them in Gaia UI tests (python or javascript)
  • Executing mini test runs at the end of each sprint
  • Functional testruns during the week of FC (2-3?)
  • Branch merge, run a smoke test and
  • Execute 3-4 full testruns during the cycle
  • 1-2 Localization testruns
  • FOTA update testing
  • 3rd party apps smoke test (but trying to hand this off)
  • Daily Bug Triage work

Platform work

Environment: Emulators


    • enable platform tests for B2G (mochitests, layout, reftests, crash, new APIs)
    • create device API tests for CRB and test apps
      • we need a place to converge these App tests so it can be run by automation and community

    Automation

    Environment: Devices, Desktop Builds


    • Gaia UI tests on current production branch (ie. 1.2, 1.3, etc..)
    • Gaia UI tests on master/m-c
    • MTBF stability testing across multiple devices
    • Eideticker performance testing measure FPS startup and pan/zoom
    • Endurance testing on core apps
    • future: fuzzing on b2g with marionette  (MariFuzz)
    • CRB test guidance

    Miscellaneous

    Environment: Various


    • Support Flashing tools for nightly and CI machines
    • Assisting partners with automation and test case setup
    • Support live issues from partners
    • test prototypes

    Community

    Environment: Devices, Desktop builds, Apps

    • Target device testers
      • execute moztrap tests
      • download and run test Apps, submit test data
      • have meet ups and testdays
      • Document!  (irc, blogs, social networking, Planet, QMO)
    • Target automation testers
      • set up test environments 
      • Mentor and review contributing testcases
      • Document!  (github readmes, comments, bugs)
    • Target app developers
      • How to properly test apps in HTML5
      • How to clone their own test environments we use

    January Workweek!


    Knowledge Transfer.

    Cross product updates and sharing.

    Tools Exploration and Enhancements.

    Co-locating test environments.

    Community Roadmap.

    Brainstorm here:
    https://etherpad.mozilla.org/b2g-qa-workweek-brainstorm

    Firefox OS Test Strategy

    By dknite

    Firefox OS Test Strategy

    • 555
    Loading comments...

    More from dknite