Some Lessons I Learned At Netflix

Lessons From Netflix

• Build Lots of Small Services.

  • Make Them Easy To Work With.

• Build fast, Build Often.
   

• Documentation Is Non-Negotiable.
   
• Tolerate Failure.
   
• Errors Are Part Of Documentation.
   

Documentation Is Non Negotiable

• An API that isn't documented is no API at all
   

• In an Ideal World?
  • Code is self documenting
     
• At Netflix we used (and loved) Swagger
  • Describe API endpoints as part of code
  • Documentation generated
  • Easy to generate client libraries from Swagger endpoint

Swagger At A (Brief) Glance

Swagger At A (Brief) Glance

Tolerate Failure

• Tolerating failure in a Microservice environment is crucial.
   
• Small independent pieces should mean one thing failing doesn't take out the whole system.
   
• Lots of small pieces, loosely coupled.
   
• Key Term: Graceful Degradation

Failing Gracefully

• Microservices let us choose what constitutes a true system outage
   
• Search feature down? You can still serve the rest of your site
   
• To Fail/Degrade Gracefully We Need To Use a Forbidden Word...
   
• DESIGN PATTERN!

Circuit Breakers

• In his book Release It, Michael Nygard introduced the Circuit Breaker pattern
   
• Designed like an electrical circuit breaker
   
• When things are working: Closed
   
• When things aren't working: Open
   
• When trying to reset: Half-Open

The Circuit Breaker Pattern

[ credit: akka.io ]

Errors

• Errors!
   

• Errors!
   

• Errors!

Errors

• Meaningless Errors & Exceptions lead to long days (and nights!) slaving over error logs
   

• Nothing sucks quite so much as spending long hours digging through error logs...
   

• ... Only to find out it was something stupid that could have been provided in an error message

Good Practices with Errors

• If you can validate an entire request object, do so.

  • Return every error you find immediately, and clearly.
     

• If you can pass multiple errors up your stack and return them, do so...

  • JSON supports Arrays, after all

Good Practices with Errors

• Provide useful data, not just a generic 500 Internal Server Error

  • If you have an exception, extract some useful message from it
     

  • REMINDER: Exceptions shouldn't be used for flow control

Provide Errors That Make Sense

• PHP Provides a particularly useful error message...
   

• Parse error: syntax
  error, unexpected
  T_PAAMAYIM_NEKUDOTAYIM
   

• W. T. F. ?

Provide Errors That Make Sense

• 'Paamayim
  Nekudotayim' ( פעמיים נקודתיים ) is apparently
  Hebrew for "Double Colon"
   

• The main authors of PHP 3+
  are Israeli, and probably
  thought this was funny.
   

• Imagine how many hapless
  devs have misplaced a '::'
  and spent hours scratching
  their heads...

Sensible Errors

• Most API platforms have
  some sort of Status Code
  mechanism
   

• For REST, Pick & Document
  HTTP Codes
   

• BE CONSISTENT

Sensible Errors

• Even if you make up your
  own silly HTTP codes, make
  sure you document them so
  people know what they mean.
   

• There's a proposed HTTP
  Error Code "759 -
  Unexpected
  T_PAAMAYIM_NEKUDOTAYIM"
   

• I also like "763 - Under-
  Caffeinated"

Testing Matters!

• Your API Framework of
  choice should provide good
  tools for testing it

  • If it doesn't, consider finding something better
     

• Good testing contributes greatly to feeling comfortable that
  what you've built works as
  documented

Write Lots of Tests

• Positive Tests

• Negative Tests

• Service Level Tests (HTTP
  Calls)

• Helper Level Tests (backend
  objects sans HTTP)

• Parser Tests

• Regression Tests

• Voight-Kampff Tests

One Test
Two Test
Red Test
Blue Test

Never trust a test you haven't seen fail.