Serverless Benchmarks
Round 2
Sydney Lambda Meetup September 2018
About Me Us
Round 1
Round 2
Winner
2018 Round 1 Multi Lingual
Round 1 Feedback
Done
- Lambda Signatures were inconsistent
- Java had way too much overhead, log4j etc
- No cold start
- No detailed metrics on median, p95, p99 etc
Pending
- No actual workload, add downstream service call like DynamoDB or S3
- .Net Core is fast? That's unpossible!
- Add other JVM flavours Scala/Kotlin since F# maded it in
- Add Google Cloud and Azure
Not Doing
- Not "realistic"! Add universal "realistic" scenarios
Round 1 Results
- .Net 2.0 Outperforms
- Go and Java is about the same
- Consistent Performance between Dynamic vs Static
- Go Packages are bigger than Java
Round 2
Including Cold Starts
Round 2 Detailed Results
| Runtime | Min | Max | Median | P95 | P99 |
|---|---|---|---|---|---|
| aws-csharp2 | 242ms | 8.447s | 331.6ms | 1.132s | 5.332s |
| aws-csharp2.1 | 243ms | 11.078s | 328ms | 1.247s | 5.297s |
| aws-fsharp2 | 237ms | 10.938s | 338ms | 1.340s | 5.328s |
| aws-fsharp2.1 | 238ms | 11.120s | 329ms | 1.762s | 5.306s |
| aws-go | 243ms | 11.390s | 332ms | 1.262s | 5.301s |
| aws-java8 | 241ms | 7s | 335ms | 1.095s | 5.284s |
| aws-node6 | 238ms | 9.132s | 327ms | 1.078s | 5.285s |
| aws-node8 | 231ms | 8.558s | 324ms | 1.064s | 5.272s |
| aws-python2 | 239ms | 8.516s | 326ms | 1.316s | 5.307s |
| aws-python3 | 242ms | 10.852s | 337ms | 1.430s | 5.326s |
| gcp-node8 | 267ms | 17.127s | 305ms | 7s | 11.342s |
The Winner
Is You!
The median results are quite consistent between runtimes
But the real benefit of Serverless is in the paradigm shift towards Simplified Operating Model and Focus on Business Value
Questions
Serverless Benchmarks 2018 Round 2
By Yun Zhi Lin