Developer Confidence

or: How I learned to stop worrying and care about the website

Mike Sherov

Core UX Team @ Skillshare

Do you care about the website?


  • +1mil visits/day

  • Responsive Website

  • Syndicated White Label "CCN" sites

  • Native iOS/tvOS/Android app

  • 10+ 2nd party API clients (Adobe products)

  • 100+ 3rd party API clients


Deployments as a Service

Moonbeam - PR Review

  • Enforces a reviewers file
  • Allows teams to define PR acceptance criteria
  • Only reviewed PRs can be deployed
  • Admins can deploy unreviewed PRs (with a warning)

Moonbeam - Build Gaurdrails

  • Breaks deployments into steps: build, stage, prod
  • CI enforced at every step
  • Checklists enforce manual tests and watching dashboards
  • Build reruns CI same (or stronger) checks that Github PR CI
  • Stage runs critical e2e tests and asks for manual verification
  • Prod asks for manual verification: "mark as good"

Moonbeam - Keep it Green

  • PRs merged to master only after "mark as good" in PROD
  • Any time before "mark as good", 1 click to rollback builds
  • After "mark as good", admin can emergency rollback to any previous "mark as good", and lockdown the build queue

Moonbeam - Make it fast

  • Contribute to critical build pipeline projects: npm, webpack, babel, uglify
  • Build server caching: npm, uglify, babel
  • Drafting! PR A in PROD, PR A + B in stage, PR A + B + C in buildĀ 
  • PR clustering: User A can can take up multiple approved PRs at a time (although this is discouraged)

Moonbeam - Scale it Out

  • What started as a napkin drawing is used
    by 100+ teams at Adobe to ship code
  • Fully dedicated RE team
  • Continual improvements to UX, as it's a
    critical step of modern development

Code Review

  • Small PRs encouraged. Minimal change that can be considered complete.
  • Say something positive on every PR.
  • Quick tight rounds of feedback. PRs shouldn't sit for days.
  • Ignore style comments at all costs, use robots for that.

Feature Flags

  • New features are incrementally developed and shipped to PROD behind flags.
  • Design and community and team use the product in production environment before real users see it.
  • Allows for continuous design review, and enables small PRs, even for new features.

Testing - Unit Tests

  • No coverage enforcement yet
  • Minimal Mocking: test behavior, not implementation
  • Culture of testing

Testing - e2e Tests

  • Critical suite runs in Moonbeam on stage
  • Full suite runs against prod with alerting out of band
  • on Saucelabs (no Cypress yet)

Testing - QA tests

  • "Continuous QA" via rainforest QA
  • Mechanical Turk style integration tests


  • New Relic (is amazing)
  • Sumologic (Splunk) to aggregate and query logs
  • Datadog for measuring common ops metrics


  • Pagerduty rotations for devs
  • alerting from e2e tests, error rate, key metrics
  • Maintain Error Zero for it's psychic and cultural effects

Community Team

  • Support team was called "Community"
  • Community team helps fix bugs in realtime in our "issues" slack channels. Emphasis on empathy for the user.
  • Community participates in sprint planning, and has lots of input into prioritization.
  • Team leaders are outspoken advocates for users and for fixing stuff with community.
  • Deployments are announced to community team as code is shipped to keep them continuously in the loop.

Support each other

  • Devs are humans!
  • Blameless postmortems
  • No one gets in trouble just for breaking the site
  • Team needs to feel unjudged and comfortable deploying
  • Yes, it's scary to deploy to a big site! But that fear should be about ruining things for users, not for our jobs or shaming in front of peers.

Developer Confidence

By mikesherov

Developer Confidence

  • 1,677