Load TEsting Your APIs

Day Camp 4 Developers / Plumbing THe Internet

Ian Littman / @iansltx

follow along at https://ian.im/dc4d18

About You

• You're a developer, doing devops, or close thereto
• You can build a near-production environment that's billed hourly (optional, but highly helpful)
• You like human-readable scripts and CLIs/APIs over fancy GUIs spitting out XML (because the tools we'll be looking at do the former)

In this presentation we will Learn...

• ...when it makes sense to test
• ...the difference between a smoke test, a load test, a stress test, and a spike test
• ...how to better match your load test with (anticipated) reality for more useful results
• ...what bottlenecks to look for when stress testing
• ...that a bunch of free, open-source utilities exist to load test your application
• ...how to use a couple of them

In this presentation we wOn'T Learn...

• ...about every load test application out there
• ...how to set up clustered load testing (read load test application docs)
• ...how to load test all the way to the browser (we're focusing on APIs)
  • Slow connections tie up server/load balancer resources for longer
  • Solutions for slow connections (e.g. compression) may revise system capacity elsewhere
• ...how to do deep application profiling, e.g. Blackfire
• ...about single-user load testing (e.g. running an import with a larger data set than usual)

A Challengr Appears

This will be our system under test

Why?

No, Seriously, you need to answer this question.

When?

• When your application performance may change
  • Adding/removing features
  • Refactoring
  • Infrastructure changes
• When your load profile may change
  • Initial app launch
  • Feature launch
  • Marketing pushes/promotions

#IFNDEF

Load TEst

• <= peak traffic
• Your system shouldn't break
• If it does, it's a stress test

Stress Test

• Trying to break your system
• Surfaces bottlenecks
• Increase traffic above peak or decrease available resources
• Capacity Test is a subset

Soak Test

• Extended test duration
• Watch behavior on ramp down as well as ramp up
• Memory leaks
• Disk space exhaustion (logs!)
• Filled caches

Spike Test

• Stress test with quick ramp-up
• Woot.com at midnight
• TV ad "go online"
• System comes back online after downtime
• Everyone hits your API via on-the-hour cron jobs

Smoke test

• An initial test to confirm the system operates properly without a large amount of generated load
• May be integration tests in your existing test suite
• May be your load test script, turned down to one (thorough) iteration and one Virtual User
• Do this before you load test

Rule #1: Strive for accuracy

What are your metrics?

• Speed - response latency
• Scalability - throughput, resource utilization
• Stability - % failed calls/transactions/flows

What should I test?

• Flows, not just single endpoints
• Frequently used
• Performance intensive
• Business critical

Concurrent Requests != Concurrent Users

• Think Time
• API client concurrency
• Caching (client-side or otherwise)

Don't Oversimplify

• No starting data in database
• No parameterization
• No abandonment at each step in the process
• No input errors
• No think times
• Static think times
• Uniformly distributed think times
  • Normal distribution
  • Exponential distribution
• Assuming you have one type of user
• Assuming that a distribution is normal

Vary Your Testing

• Best Case vs. Anticipated Case vs. Worst Case
  • Worst case: heavier endpoints get refreshed more often
  • Best case: validation failures + think time decreases writes per second
• Turn off think time for stress tests

Things that help you test more effectively

• Run your APM (e.g. New Relic, Tideways) on your load test env
  • Better profiling info
  • You'll have the same perf hit as production
• Is your environment code-ified?
  • Easier to copy envs
  • Cheaper to set up an env for an hour to run a load test
• Decide whether testing from near your environment is accurate enough (save on bandwidth costs vs. testing outside the cloud)
• Use logs/analytics to figure out how long your users are spending

at some point enough is enough

Your simulation won't be perfect

Rule #2: Don't Misinterpret Your Data

Aggregating Your Metrics

• Average
• Median (~50th percentile)
• 90th, 95th, 99th percentile
• Standard Deviation
• Distribution of results
• Explain your outliers

Things to Keep In Mind

• Just because a request is heavy doesn't mean it's the biggest source of load
• As a system reaches capacity you'll see nonlinear performance degradation
  • Turn off features to shed load
  • Autoscale
  • Test scaling as part of your load test

Bottlenecks

• Web Server
  • FPM workers/Apache processes
  • CPU + RAM utilization
  • Network utilization
  • Disk utilization
• Load balancer
  • Network utilization/warmup
  • Connection count
• External Services
  • Rate limits (natural or artificial)
  • Latency
  • Network egress

• Queues
  • Per-job spin-up latency
  • Worker count
  • CPU + RAM utilization
    • Workers
    • Broker
  • Queue depth
• Caches
  • Thundering herd
  • Churning due to cache evictions

Thanks! Questions?

• ian.im/dc4d18 - these slides
• github.com/iansltx/challengr - this code
• twitter.com/iansltx - me
• github.com/iansltx - my code
• Performance Testing Guidance for Web Applications (from Microsoft)
• Blazemeter Blog