Improved experiment workload accuracy while lowering overhead
Simon Frey
B.Sc. Computer Science
Technische Universität Berlin
Non-functional Regression Testing on Serverless Applications Using Controlled Online Experiments
20.11.2020
If nothing else, remember this
Online experiments in FaaS are possible
They have a high cost overhead
and require provisioned concurrency
Why this topic is relevant
- FaaS is used in more complex applications
- Regression testing ensures application quality
- Distributed nature of FaaS makes offline testing difficult
- Shadow traffic is not support at all by the cloud providers
Research question
How can experimenters use specialized workloads in online experiments on serverless applications on
demand?
Online experiment - Definition
| Offline experiment | Online experiment | |
|---|---|---|
| Infrastructure | Replica of live system | Live system |
| User facing side effects | None | Possible |
| Workload quality | Mediocre | High |
| Workload expressiveness | High | Low |
Experiment setup
ExSys design challenges
| Design challenge | Solution |
|---|---|
| Workload expressiveness | Various forms of routing configurations |
| Low client visible side effects | Asynchronous execution of shadow functions |
| Usability | REST User API for experiment control |
| Infrastructure cost overhead | Ephemeral Proxy function only being deployed for the duration of experiment |
| Custom traffic routing | Intermediate Proxy function function controlling experiment state |
| Measure incoming traffic | Distributed counter via database stream |
| Elasticity issues | Provisioned concurrency |
ExSys system overview
Proxy inner workings
Control function inner workings
Elasticity issues
Weighted Distributed counter
$$ratio_{r/s} = \frac{weigthed\_total\_request\_count}{\sum\limits_{i=1}^{ss} i}$$
$$ss = \text{seconds since experiment start}$$
$$weigthed\_total\_request\_count = request * ss$$
Distributed counter issues
- Setup with DynamoDB streams might be too slow
- A different implementation might work better (e.g. a central counter service)
Counter design problem
ExSys cost overhead
- At least 100%
- Provisioned concurrency is expensive