Developer Confidence

or: How I learned to stop worrying and care about the website

Mike Sherov

Core UX Team @ Skillshare

Do you care about the website?

YMMV

Behance.net

• +1mil visits/day

• Responsive Website

• Syndicated White Label "CCN" sites

• Native iOS/tvOS/Android app

• 10+ 2nd party API clients (Adobe products)

• 100+ 3rd party API clients

Moonbeam

Deployments as a Service

Moonbeam - PR Review

• Enforces a reviewers file
• Allows teams to define PR acceptance criteria
• Only reviewed PRs can be deployed
• Admins can deploy unreviewed PRs (with a warning)

Moonbeam - Build Gaurdrails

• Breaks deployments into steps: build, stage, prod
• CI enforced at every step
• Checklists enforce manual tests and watching dashboards
• Build reruns CI same (or stronger) checks that Github PR CI
• Stage runs critical e2e tests and asks for manual verification
• Prod asks for manual verification: "mark as good"

Moonbeam - Keep it Green

• PRs merged to master only after "mark as good" in PROD
• Any time before "mark as good", 1 click to rollback builds
• After "mark as good", admin can emergency rollback to any previous "mark as good", and lockdown the build queue

Moonbeam - Make it fast

• Contribute to critical build pipeline projects: npm, webpack, babel, uglify
• Build server caching: npm, uglify, babel
• Drafting! PR A in PROD, PR A + B in stage, PR A + B + C in build 
• PR clustering: User A can can take up multiple approved PRs at a time (although this is discouraged)

Moonbeam - Scale it Out

• What started as a napkin drawing is used
  by 100+ teams at Adobe to ship code
• Fully dedicated RE team
• Continual improvements to UX, as it's a
  critical step of modern development

Code Review

• Small PRs encouraged. Minimal change that can be considered complete.
• Say something positive on every PR.
• Quick tight rounds of feedback. PRs shouldn't sit for days.
• Ignore style comments at all costs, use robots for that.

Feature Flags

• New features are incrementally developed and shipped to PROD behind flags.
• Design and community and team use the product in production environment before real users see it.
• Allows for continuous design review, and enables small PRs, even for new features.

Testing - Unit Tests

• No coverage enforcement yet
• Minimal Mocking: test behavior, not implementation
• Culture of testing

Testing - e2e Tests

• Critical suite runs in Moonbeam on stage
• Full suite runs against prod with alerting out of band
• Webdriver.io on Saucelabs (no Cypress yet)

Testing - QA tests

• "Continuous QA" via rainforest QA
• Mechanical Turk style integration tests

Monitoring

• New Relic (is amazing)
• Sumologic (Splunk) to aggregate and query logs
• Datadog for measuring common ops metrics

Alerting

• Pagerduty rotations for devs
• alerting from e2e tests, error rate, key metrics
• Maintain Error Zero for it's psychic and cultural effects

Community Team

• Support team was called "Community"
• Community team helps fix bugs in realtime in our "issues" slack channels. Emphasis on empathy for the user.
• Community participates in sprint planning, and has lots of input into prioritization.
• Team leaders are outspoken advocates for users and for fixing stuff with community.
• Deployments are announced to community team as code is shipped to keep them continuously in the loop.

Support each other

• Devs are humans!
• Blameless postmortems
• No one gets in trouble just for breaking the site
• Team needs to feel unjudged and comfortable deploying
• Yes, it's scary to deploy to a big site! But that fear should be about ruining things for users, not for our jobs or shaming in front of peers.