Stockwell

#stockwell:

* :gbrown

* :jmaher

Turning our intermittent test failures into a manageable problem

2017 - what we planned

  • Define policy (30+ failures == needswork)
  • Triage (and handoff to component owners)
  • Identify ownership (all files map to bug component)
  • Test-verify
  • Identify and Automate useful data
  • Dashboards
  • Identify problems faster
  • Auto classification

2017 Triage

  • 1283 bugs triaged
  • 603 bugs fixed
    • 220 test/harness/tool fixes
    • 110 product fixes
    • 29 backouts
  • 308 bugs disabled
  • 265 bugs fixed themselves
  • 74 bugs related to infrastructure
  • 67 components owner triaged
>= 35 bugs >= 20 bugs <=13 bugs
Core::DOM Core::WebRTC Core::DOM: Security
Core::Layout Firefox::General DevTools: Console
WebExtensions: General Testing::Talos RelEng::Buildduty
Audio/Video: Playback DevTools: Debugger Core::Canvas: WebGL
Testing::Marionette Android::Testing Firefox::Session Restore
Testing::web-platform Testing::Firefox UI Tests Graphics: WebRender
Core::CSS Parsing and Computation Core::Build Config Core::Disability Access APIs

Components with high frequency bugs

Components top fixed/disabled

Fixed Disabled
Firefox for Android::Testing Firefox::Session Restore
WebExtensions: Untriaged Core::WebRTC
Testing::Firefox UI Tests DevTools: Debugger
Testing::Talos Testing::web-platform-tests
Core::Build Config Core::Canvas: WebGL

2018 - the plan

  • 100% owner triage
  • Auto classification
  • More test-verify
  • Ensure accuracy on ownership data
  • Dashboards/Data
  • Policy adjustment: disable if 100 failures/21 days

Stockwell

By Joel Maher

Made with Slides.com

Stockwell

What we have done and learned in 1 year of focusing on intermittent failures

  • 1,204

More from Joel Maher