Load TEsting Your APIs
Day Camp 4 Developers / Plumbing THe Internet
follow along at https://ian.im/dc4d18
About You
- You're a developer, doing devops, or close thereto
- You can build a near-production environment that's billed hourly (optional, but highly helpful)
- You like human-readable scripts and CLIs/APIs over fancy GUIs spitting out XML (because the tools we'll be looking at do the former)
In this presentation we will Learn...
- ...when it makes sense to test
- ...the difference between a smoke test, a load test, a stress test, and a spike test
- ...how to better match your load test with (anticipated) reality for more useful results
- ...what bottlenecks to look for when stress testing
- ...that a bunch of free, open-source utilities exist to load test your application
- ...how to use a couple of them
In this presentation we wOn'T Learn...
- ...about every load test application out there
- ...how to set up clustered load testing (read load test application docs)
- ...how to load test all the way to the browser (we're focusing on APIs)
- Slow connections tie up server/load balancer resources for longer
- Solutions for slow connections (e.g. compression) may revise system capacity elsewhere
- ...how to do deep application profiling, e.g. Blackfire
- ...about single-user load testing (e.g. running an import with a larger data set than usual)
A Challengr Appears
This will be our system under test
Why?
No, Seriously, you need to answer this question.
When?
- When your application performance may change
- Adding/removing features
- Refactoring
- Infrastructure changes
- When your load profile may change
- Initial app launch
- Feature launch
- Marketing pushes/promotions
#IFNDEF
Load TEst
- <= peak traffic
- Your system shouldn't break
- If it does, it's a stress test
Stress Test
- Trying to break your system
- Surfaces bottlenecks
- Increase traffic above peak or decrease available resources
- Capacity Test is a subset
Soak Test
- Extended test duration
- Watch behavior on ramp down as well as ramp up
- Memory leaks
- Disk space exhaustion (logs!)
- Filled caches
Spike Test
- Stress test with quick ramp-up
- Woot.com at midnight
- TV ad "go online"
- System comes back online after downtime
- Everyone hits your API via on-the-hour cron jobs
Smoke test
- An initial test to confirm the system operates properly without a large amount of generated load
- May be integration tests in your existing test suite
- May be your load test script, turned down to one (thorough) iteration and one Virtual User
- Do this before you load test
Rule #1: Strive for accuracy
What are your metrics?
- Speed - response latency
- Scalability - throughput, resource utilization
- Stability - % failed calls/transactions/flows
What should I test?
- Flows, not just single endpoints
- Frequently used
- Performance intensive
- Business critical
Concurrent Requests != Concurrent Users
- Think Time
- API client concurrency
- Caching (client-side or otherwise)
Don't Oversimplify
- No starting data in database
- No parameterization
- No abandonment at each step in the process
- No input errors
- No think times
- Static think times
- Uniformly distributed think times
- Assuming you have one type of user
- Assuming that a distribution is normal
Vary Your Testing
- Best Case vs. Anticipated Case vs. Worst Case
- Worst case: heavier endpoints get refreshed more often
- Best case: validation failures + think time decreases writes per second
- Turn off think time for stress tests
Things that help you test more effectively
- Run your APM (e.g. New Relic, Tideways) on your load test env
- Better profiling info
- You'll have the same perf hit as production
- Is your environment code-ified?
- Easier to copy envs
- Cheaper to set up an env for an hour to run a load test
- Decide whether testing from near your environment is accurate enough (save on bandwidth costs vs. testing outside the cloud)
- Use logs/analytics to figure out how long your users are spending
at some point enough is enough
Your simulation won't be perfect
Rule #2: Don't Misinterpret Your Data
Aggregating Your Metrics
Average- Median (~50th percentile)
- 90th, 95th, 99th percentile
- Standard Deviation
- Distribution of results
- Explain your outliers
Things to Keep In Mind
- Just because a request is heavy doesn't mean it's the biggest source of load
- As a system reaches capacity you'll see nonlinear performance degradation
- Turn off features to shed load
- Autoscale
- Test scaling as part of your load test
Bottlenecks
- Web Server
- FPM workers/Apache processes
- CPU + RAM utilization
- Network utilization
- Disk utilization
- Load balancer
- Network utilization/warmup
- Connection count
- External Services
- Rate limits (natural or artificial)
- Latency
- Network egress
- Queues
- Per-job spin-up latency
- Worker count
- CPU + RAM utilization
- Workers
- Broker
- Queue depth
- Caches
- Thundering herd
- Churning due to cache evictions
Load Test Tools We'll Look At
- Siege - a quick, rather simple command line utility
-
k6 - write your tests in JS, run via a Go binary*
- HAR import for in-browser recording
- Fix for Grafana install (or use the 5.0.4 image)
* I've used this on a project significantly more real than Challengr, so that's a big reason we're looking at it today.
More Tools
-
Tsung
- Erlang (efficient, high volume from a single box)
- Flexible (not just HTTP)
- XML based config
-
The Grinder
- Java-based
- Java, Jython or Clojure scripts
-
Bees With Machine Guns
- Uses EC2 instances
- Python-based
- Goad (Go inside Lambda)
-
Gatling (Java-based)
- Tests in Scala...
- ...or use the recorder
- ab
- httperf
- Apache JMeter
Even More Tools!
-
Artillery.io
- Node-based
- Simple stuff in Yaml, can switch to JS (including npm)
-
Molotov
- Python 3.5+, uses async IO via coroutines
Thanks! Questions?
- ian.im/dc4d18 - these slides
- github.com/iansltx/challengr - this code
- twitter.com/iansltx - me
- github.com/iansltx - my code
- Performance Testing Guidance for Web Applications (from Microsoft)
- Blazemeter Blog
Load Testing Your API - DC4D
By Ian Littman
Load Testing Your API - DC4D
- 2,093