QUestions We'll Answer
- What types of tests exist, and when should I use them?
- How can I match my load test with (anticipated) reality?
- What does a real load test script look like on a small system?
- How do I properly analyze results during and after my test?
Questions we won't answer
- How do I use $otherPerfTestTool (!== 'k6')?
- How can I set up clustered load testing?
- How can I simulate far-end users?
- How can I do deep application profiling? (Blackfire)
- What about single-user load testing?
This is what we'll test with*
- Siege - a quick, rather simple command line utility
- k6 - write your tests in JS**, run via a Go binary***
* More tools are listed at the end of this presentation.
** It's JS, but it uses goja, not V8 or Node, and doesn't have a global event loop yet.
*** I've used this on a project significantly more real than the example in this presentation, so that's a big reason we're looking at it today.
This will be our system under test
- Load Test vs. Stress Test
- Soak Test vs. Spike Test
- Smoke Test
- <= expected peak traffic
- Your system shouldn't break
- If it does, it's a...
- Increase traffic above peak || decrease available resources
- Trying to break your system
- Surfaces bottlenecks
- Extended test duration
- Watch behavior on ramp down
as well as ramp up
- Memory leaks
- Disk space exhaustion (logs!)
- Filled caches
- Stress test with quick ramp-up
- Woot.com at midnight
- TV ad "go online"
- System comes back online after downtime
- Everyone hits your API via on-the-hour cron jobs
- An initial test to confirm the system operates properly without a large amount of generated load
- Do this before you load test
- Pick your implementation...
- Integration tests in your existing test suite
- Load test script, turned down to one (thorough) iteration and one Virtual User (VU)
When should you run a load test?
- When your application performance may change
- Adding or removing features
- Infrastructure changes
- When your load profile may change
- Initial app launch
- Feature launch
- Marketing pushes and promotions
How should I test?
How should I test?
What should I test?
- Flows, not just single endpoints
- Frequently used
- Performance intensive
- Business critical
Concurrent Requests != Concurrent Users
- Think Time
- API client concurrency
- Caching (client-side or otherwise)
How not to model think time
- Uniformly distributed
(use a normal distribution instead)
- Assume you have one type of user
Other Oversimplifications to avoid
- No starting data in database
- No parameterization
- No abandonment at each step in the process
- No input errors
Vary Your Testing
- High-load Case: more expensive endpoints get called more often
- Anticipated Case
- Low-load Case: validation failures + think time
Auth + Fixture Data
Probabilities and input specs
Trends: How long did it take?
Now that our setup is done...
Let's make an HTTP Call
Make sure we fail successfully
Make sure we succeed successfully
Making simultaneous requests
checking simultaneous responses
Timing simultaneous responses
let's create a challenge
Understand your load test tool
Aggregate your metrics repsonsibly
- Median (~50th percentile)
- 90th, 95th, 99th percentile
- Standard Deviation
- Distribution of results
- Explain (don't discard) your outliers
Keep it real
- Use logs and analytics to determine your usage patterns
- Run your APM (e.g. New Relic, Tideways) on your system under test
- Better profiling info
- Same performance drop as instrumenting production
- Is your infrastructure code? (e.g. Terraform, CloudFormation)
- Easier to copy environments
- Cheaper to set up an environment for an hour to run a load test
- Decide whether testing from near your env is accurate enough
- Test autoscaling and/or load-shedding facilities
Warning: Tricky bottlenecks ahead
- Just because a request is expensive
doesn't mean it's the biggest source of load
- As a system reaches capacity
you'll see nonlinear performance degradation
Bottlenecks: Web Server + DAtabase
- FPM workers/Apache processes
- DB Connections
- CPU + RAM utilization
- Network utilization
- Disk utilization (I/O or space)
Bottlenecks: Load Balancer
- Network utilization/warmup
- Connection count
Bottlenecks: External Services
- Rate limits (natural or artificial)
- Network egress
- Per-job spin-up latency
- Worker count
- CPU + RAM utilization
- Queue depth
- Thundering herd
- Churning due to cache evictions
Bonus material: More Tools
BONUS MATERIAL: Even More Tools!
What We Learned
- What types of tests exist, and when you should use them
- How to match load test with (anticipated) reality
- What a real load test script looks like in K6
- How to analyze results during and after your test
Load Testing Your App - CascadiaPHP 2019
By Ian Littman