Load TEsting Your APIs

Day Camp 4 Developers / Plumbing THe Internet

Ian Littman / @iansltx

follow along at https://ian.im/dc4d18

About You

  • You're a developer, doing devops, or close thereto
  • You can build a near-production environment that's billed hourly (optional, but highly helpful)
  • You like human-readable scripts and CLIs/APIs over fancy GUIs spitting out XML (because the tools we'll be looking at do the former)

In this presentation we will Learn...

  • ...when it makes sense to test
  • ...the difference between a smoke test, a load test, a stress test, and a spike test
  • ...how to better match your load test with (anticipated) reality for more useful results
  • ...what bottlenecks to look for when stress testing
  • ...that a bunch of free, open-source utilities exist to load test your application
  • ...how to use a couple of them

In this presentation we wOn'T Learn...

  • ...about every load test application out there
  • ...how to set up clustered load testing (read load test application docs)
  • ...how to load test all the way to the browser (we're focusing on APIs)
    • Slow connections tie up server/load balancer resources for longer
    • Solutions for slow connections (e.g. compression) may revise system capacity elsewhere
  • ...how to do deep application profiling, e.g. Blackfire
  • ...about single-user load testing (e.g. running an import with a larger data set than usual)

A Challengr Appears

This will be our system under test

Why?

No, Seriously, you need to answer this question.

When?

  • When your application performance may change
    • Adding/removing features
    • Refactoring
    • Infrastructure changes
  • When your load profile may change
    • Initial app launch
    • Feature launch
    • Marketing pushes/promotions

#IFNDEF

Load TEst

  • <= peak traffic
  • Your system shouldn't break
  • If it does, it's a stress test

Stress Test

  • Trying to break your system
  • Surfaces bottlenecks
  • Increase traffic above peak or decrease available resources
  • Capacity Test is a subset

Soak Test

  • Extended test duration
  • Watch behavior on ramp down as well as ramp up
  • Memory leaks
  • Disk space exhaustion (logs!)
  • Filled caches

Spike Test

  • Stress test with quick ramp-up
  • Woot.com at midnight
  • TV ad "go online"
  • System comes back online after downtime
  • Everyone hits your API via on-the-hour cron jobs

Smoke test

  • An initial test to confirm the system operates properly without a large amount of generated load
  • May be integration tests in your existing test suite
  • May be your load test script, turned down to one (thorough) iteration and one Virtual User
  • Do this before you load test

Rule #1: Strive for accuracy

What are your metrics?

  • Speed - response latency
  • Scalability - throughput, resource utilization
  • Stability - % failed calls/transactions/flows

What should I test?

  • Flows, not just single endpoints
  • Frequently used
  • Performance intensive
  • Business critical

Concurrent Requests != Concurrent Users

  • Think Time
  • API client concurrency
  • Caching (client-side or otherwise)

Don't Oversimplify

  • No starting data in database
  • No parameterization
  • No abandonment at each step in the process
  • No input errors
  • No think times
  • Static think times
  • Uniformly distributed think times
  • Assuming you have one type of user
  • Assuming that a distribution is normal

Vary Your Testing

  • Best Case vs. Anticipated Case vs. Worst Case
    • Worst case: heavier endpoints get refreshed more often
    • Best case: validation failures + think time decreases writes per second
  • Turn off think time for stress tests

Things that help you test more effectively

  • Run your APM (e.g. New Relic, Tideways) on your load test env
    • Better profiling info
    • You'll have the same perf hit as production
  • Is your environment code-ified?
    • Easier to copy envs
    • Cheaper to set up an env for an hour to run a load test
  • Decide whether testing from near your environment is accurate enough (save on bandwidth costs vs. testing outside the cloud)
  • Use logs/analytics to figure out how long your users are spending

at some point enough is enough

Your simulation won't be perfect

Rule #2: Don't Misinterpret Your Data

Aggregating Your Metrics

  • Average
  • Median (~50th percentile)
  • 90th, 95th, 99th percentile
  • Standard Deviation
  • Distribution of results
  • Explain your outliers

Things to Keep In Mind

  • Just because a request is heavy doesn't mean it's the biggest source of load
  • As a system reaches capacity you'll see nonlinear performance degradation
    • Turn off features to shed load
    • Autoscale
    • Test scaling as part of your load test

Bottlenecks

  • Web Server
    • FPM workers/Apache processes
    • CPU + RAM utilization
    • Network utilization
    • Disk utilization
  • Load balancer
    • Network utilization/warmup
    • Connection count
  • External Services
    • Rate limits (natural or artificial)
    • Latency
    • Network egress
  • Queues
    • Per-job spin-up latency
    • Worker count
    • CPU + RAM utilization
      • Workers
      • Broker
    • Queue depth
  • Caches
    • Thundering herd
    • Churning due to cache evictions

Load Test Tools We'll Look At


* I've used this on a project significantly more real than Challengr, so that's a big reason we're looking at it today.

More Tools

  • Tsung
    • Erlang (efficient, high volume from a single box)
    • Flexible (not just HTTP)
    • XML based config
  • The Grinder
    • Java-based
    • Java, Jython or Clojure scripts

Even More Tools!

  • Artillery.io
    • Node-based
    • Simple stuff in Yaml, can switch to JS (including npm)
  • Molotov
    • Python 3.5+, uses async IO via coroutines
  • Locust
    • Python based
    • Can be run clustered (as can many others)
  • Wrk2
    • Built in C
    • Fed via CLI args, or scriptable via Lua

Thanks! Questions?