Engineering Productivity Strategy and Roadmap

Automation & Tools

Mandate: anything related to automation or tooling

Engineering Productivity

Mandate: tooling and automation which directly impacts engineering productivity

Automation & Tools

• MozTrap
• Mozmill tests, media tests
   
• Pulse
• Autophone, MozBench

Engineering Productivity

• No; tool for another team
• No; test cases are best owned by developers
• No? infrastructure
• Maybe - automation yes, but optimally not maintaining hardware pools

What impacts engineering productivity?

Build Speeds

• local and in automation

• C++ and JavaScript developers
• "Build Faster" project with 8+ resources, 5 from Eng Productivity, goal to dramatically improve build times across the board [in progress]

Version Control

• Fast clones and fast pushes

  • partially done (bundle clones)

  • narrow and shallow clones [Q3+]

• Support for both hg and git users [in progress]

  • primarily through git support in MozReview, mach commands that interact with version control

• Creation of a unified gecko repo [in progress]

Reviewing Code and Landing Patches

• MozReview

  • improved UX [in progress]

  • static analysis [in progress]

  • code coverage [Q3+]

  • performance data [Q3+]

  • try runs and automatic landing [deployed, refinements and improvements in progress]

Reviewing Code and Landing Patches

• Autoland

  • land code à la checkin-needed [done]

  • pre-qualify landings with try runs [in progress]

  • handle branch uplifts [Q3+]

  • integration with performance data [TBD]

Continuous Integration

• Treeherder: automatic starring [in progress]
  • enables a lot of other intelligent automation
• Treeherder: viewing data in more ways [Q2+]
  • e.g., viewing test results by test files, instead of by
    job

Continuous Integration

• Perfherder
  • Talos test sheriffing integrated with Treeherder [in progress]
  • Integration of AWSY, AWFY data [in progress]

Q2 and beyond

Continuous Integration

• E2E times are fast
  • greater chunking utilized on non-OSX platforms
    • depends on TaskCluster
  • automation build times are reduced via artifact builds
    • depends on integrating some of the "build faster" work with automation
  • SETA is made to operate on a per-directory basis
    • depends on automatic starring

Continuous Integration

• Try runs fail fast
  • re-use build artifacts when appropriate for fast builds
    • depends on "build faster"
  • automatic smart test selection via static and dynamic analysis
    • depends on code coverage
  • multiple-stage runs with highest value tests run first
    • depends on TaskCluster, smarter test selection

Continuous Integration

• Code coverage
  • C++ and JavaScript
  • Make it easy for developers to get code coverage data for their patches
  • Generate code coverage data for all in-tree tests
  • Integrate with try runs/MozReview so it's easy to see coverage deltas per patch

Debugging Tests

• Make test machine checkouts more useful
  • Provide test machines with ready-to-go dev environments
  • Allow interaction with remote test machines via local mach commands
  • Make checkout of AWS-based testers self-serve

Debugging Tests

• Make it easy to move between local and remote debugging
  • Test invocation locally and via try operates via the same set of local mach commands
  • Expose TaskCluster's ability to connect to running tasks
  • Chunking/job details aren't necessary to reproduce test failures

The Future

• MozReview: inline reviewer edits?
• Try runs: display unique test failures along with the  chance that the failure was caused by that change
• Intermittent failures: ability to automatically bisect to find root cause; detection of change of intermittent failure rate?
• Test debugging: integrate debuggers like gdb with try runs
• Remote dev environments: provide fully functional and configured remote dev environments for developing/debugging on various OS's

Open Questions

• Bugzilla: invest in new views, UI?
• On-device automation: keep it small and manage ourselves, or try to scale?
• Performance tests: do current frameworks provide developers with the tools they need to measure all the kinds of performance they care about?
• Sheriffing automation: spend resources trying to automate things like tree closures, backouts, etc?
• Resource allocation: how many resources will be needed for bringing up new platforms, like TaskCluster, Windows in the cloud, etc?