Load TEsting Your App

PHP Serbia 2019

Ian Littman / @iansltx

follow along at https://ian.im/loadsrb19

QUestions We'll Answer

• What's the difference between
  • Load test
  • Stress test
  • Soak test
  • Spike test
  • Smoke test
• When should I load test?
• How can I match my load test with (anticipated) reality
  for more useful results?
• What does a real load test script look like on a small system?
• What bottlenecks should I be looking for when testing?

Questions we won't answer

• How do I use {JMeter|Gatling|Molotov}?
• How can I set up clustered load testing?
• How can I simulate far-end users?
  • Slow connections tie up server/load balancer resources for longer
  • Solutions for slow connections (e.g. compression) may affect system capacity elsewhere
• How can I do deep application profiling? (Blackfire)
• What about single-user load testing?
  (e.g. running an import with a larger data set than usual)

This is what we'll test with*

• Siege - a quick, rather simple command line utility
  • GitLab Large Staging Collider
• k6 - write your tests in JS**, run via a Go binary***
  • HAR import for in-browser recording

* More tools are listed at the end of this presentation.
** It's JS, but it uses goja, not V8 or Node, and doesn't have a global event loop yet.
*** I've used this on a project significantly more real than the example in this presentation, so that's a big reason we're looking at it today.

A Challengr Appears

This will be our system under test

#IFNDEF

Load TEst

• <= expected peak traffic
• Your system shouldn't break
• If it does, it's a...

Stress Test

• Increase traffic above peak || decrease available resources
• Trying to break your system
• Surfaces bottlenecks

Soak Test

• Extended test duration
• Watch behavior on ramp down as well as ramp up
• Memory leaks
• Disk space exhaustion (logs!)
• Filled caches

Spike Test

• Stress test with quick ramp-up
• Woot.com at midnight
• TV ad "go online"
• System comes back online
  after downtime
• Everyone hits your API via
  on-the-hour cron jobs

Source: https://twitter.com/troyhunt/status/1102312963401109504

Smoke test

• An initial test to confirm the system operates properly without a large amount of generated load
• May be integration tests in your existing test suite
• May be your load test script, turned down to one (thorough) iteration and one Virtual User (VU)
• Do this before you load test

Speed
Scalability
Stability

When should you run a load test?

• When your application performance may change
  • Adding or removing features
  • Refactoring
  • Infrastructure changes
• When your load profile may change
  • Initial app launch
  • Feature launch
  • Marketing pushes and promotions

How should I test?

How should I test?

Accurately.

What should I test?

• Flows, not just single endpoints
• Frequently used
• Performance intensive
• Business critical

Concurrent Requests != Concurrent Users

• Think Time
• API client concurrency
• Caching (client-side or otherwise)

Oversimplifications to avoid

• No starting data in database
• No parameterization
• No abandonment at each step in the process
• No input errors
• No think times
• Static think times
• Uniformly distributed think times
  (see normal distribution)
• Assuming you have one type of user

Vary Your Testing

• High-load Case: more expensive endpoints get called more often
• Anticipated Case
• Low-load Case: validation failures + think time

Let's see what that looks like WIth K6

Understand your load test tool

For example, arrival rate vs. looping

k6 is working on it

Keep it real

• Use logs and analytics to determine your usage patterns
• Run your APM (e.g. New Relic, Tideways) on your system under test
  • Better profiling info
  • Same performance drop as instrumenting production
• Is your infrastructure code? (e.g. Terraform, CloudFormation)
  • Easier to copy environments
  • Cheaper to set up an environment for an hour to run a load test
• Decide whether testing from near your env is accurate enough
  
• Test autoscaling and/or load-shedding facilities

Aggregate your metrics repsonsibly

• Average
• Median (~50th percentile)
• 90th, 95th, 99th percentile
• Standard Deviation
• Distribution of results
• Explain (don't discard) your outliers

Let's run another test!

Warning: Tricky bottlenecks ahead

• Just because a request is expensive
  doesn't mean it's the biggest source of load
   
• As a system reaches capacity
  you'll see nonlinear performance degradation

Bottlenecks

• Web Server + Database
  • FPM workers/Apache processes
  • DB Connections
  • CPU + RAM utilization
  • Network utilization
  • Disk utilization (I/O or space)
• Load balancer
  • Network utilization/warmup
  • Connection count
• External Services
  • Rate limits (natural or artificial)
  • Latency
  • Network egress

• Queues
  • Per-job spin-up latency
  • Worker count
  • CPU + RAM utilization
    • Workers
    • Broker
  • Queue depth
• Caches
  • Thundering herd
  • Churning due to cache evictions

Let's fix some bottlenecks...

Thanks! Questions?

• ian.im/loadsrb19 - these slides
• joind.in/talk/2f6a2 - rate this talk!
• github.com/iansltx/challengr - this code
• twitter.com/iansltx - me
• github.com/iansltx - my code
• Performance Testing Guidance for Web Applications (from Microsoft)
• Blazemeter Blog