West LA DevOps
June 13, 2019
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6005427/pasted-from-clipboard.png)
meetup.com/West-LA-DevOps
Agenda
- Job Board
- Industry Updates
- Talk #1: Geodesic Cloud Automation Shell
- Talk #2: Prometheus: how we ditched our legacy monitoring systems
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218718/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218722/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218732/pasted-from-clipboard.png)
About GumGum
- Computer Vision company
- Advertising division
- Context-aware ads
- Brand safety technology
- Sports division
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1015598/images/5871586/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033910/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033915/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033917/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033921/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033922/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033924/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033927/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033930/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033932/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033948/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033949/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033950/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033952/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033955/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033960/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033964/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033970/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033977/pasted-from-clipboard.png)
>20M RPM
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033984/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6033992/pasted-from-clipboard.png)
Online Advertising
![](https://s3.amazonaws.com/media-p.slid.es/uploads/91024/images/2272749/maybelline-gimp.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6254097/pasted-from-clipboard.png)
Did you know?
GumGum Invented In-Image advertising in 2008
GumGum Sports
Job Board
- DevOps Engineer @ GumGum
- Your company?
Industry Updates
Since the last time we met..
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218820/pasted-from-clipboard.png)
1) Ubuntu 14.04 EOL
- No more updates as of April 30, 2019
- Original release: April 2014
- Did you know? Versions are YY.MM.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6228711/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6228733/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6228736/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6228740/pasted-from-clipboard.png)
2) Shopify tests Istio
- Benchmarking Istio & Linkerd CPU by Michael Kipper
- Shopify was working on deploying Istio as our service mesh. But they hit a wall: cost
- From Istio's docs: "As of Istio 1.1, a proxy consumes about 0.6 vCPU per 1000 requests per second."
- This equated to 1,200 cores for the proxy alone, per million requests per second which, in GCP, would cost Shopify $50k/month/1MRPS
- Istio control plane: ~750 mcores
- Linkerd control plane: ~22 mcores
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236003/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236010/pasted-from-clipboard.png)
3) Stack Overflow breach
- On May 5, 2019 attackers managed to access the development tier for stackoverflow.com using a bug deployed the same day
- The hackers spent 5 days exploring and then escalated their access to the production systems
- Internal investigation revealed the attackers obtained names, email addresses and IP addresses of Stack Exchange users
- Stack Overflow has contracted a third-party forensics and incident response firm to assist its investigation, and says it’s resetting passwords and taking other “precautionary measures” in response to the incident
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6235998/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236012/pasted-from-clipboard.png)
4) Intel ZombieLoad exploit
- CPU hardware exploit similar to Meltdown and Spectre
- Allows arbitrary in-flight data from CPU-internal buffers (Line Fill Buffers, Load Ports, Store Buffers), including data never stored in CPU caches
- According to HN, Intel attempted to play down the issue by trying to award the researchers with the $40k tier reward and a separate $80k reward as a "gift" (which the researchers kindly denied) instead of the maximum $100k reward for finding a critical vulnerability (source)
- Check out mdsattacks.com for the attack details
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236127/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236158/pasted-from-clipboard.png)
5) DockerHub breach
- On April 25, 2019 DockerHub detected a brief period of unauthorized access to a production database
- Sensitive data from ~190k accounts could have been exposed (<5% of total users)
- Leaked data includes usernames and hashed passwords for a small percentage of these users, as well as GitHub and Bitbucket tokens for autobuilds
- If you use Docker Hub autobuilds, please check if your GitHub/BitBucket API tokens have been used to push unexpected changes to your integrated repos
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6235997/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236203/pasted-from-clipboard.png)
5) DockerHub breach
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6235997/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236203/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6236207/pasted-from-clipboard.png)
- DNS outage for over 2 hours (19:43 -22:35 UTC)
- Caused by a migration from legacy DNS system to Azure DNS
- Affected many Microsoft services
- SQL servers, Azure Postgres, Storage, Azure Active Directory among services rendered unused
- Outage affected all regions and availability zones, being region-redundant would not have helped
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241029/pasted-from-clipboard.png)
6) Azure outage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241035/pasted-from-clipboard.png)
6) Azure outage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6253968/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6253971/pasted-from-clipboard.png)
- Iteration Construct: for operator introduced
- Type system: allows for complex types and improves usage of data structures (nested maps and lists)
- First class expressions: removing need for string interpolation syntax i.e. "${aws_vpc.this.id}"
- Terraform team provides a migration script to ease migrating from 0.11 to 0.12
7) Terraform 0.12 Released
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241127/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241170/pasted-from-clipboard.png)
- Lyft, Uber, PagerDuty, Zoom, Pinterest are some of the well-known Tech IPOs in 2019
- Uncertainty in the market might be causing some VCs to want to cash out now
- Slack, Airbnb, Crowdstrike amongst those highly anticipated in this year as well
- Ride-sharing/Consumer apps have not faired well in this market, but B2B Enterprise SaaS has done very well
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241060/pasted-from-clipboard.png)
8) Tech IPO season
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241075/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241079/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241081/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241086/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241087/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241090/pasted-from-clipboard.png)
- "Two normally benign misconfigurations, and a specific software bug, combined to initiate the outage"
- Outage lasted for over four hours starting at 2:58 p.m EST
- Google services affected included Gmail, Youtube, Docs, Drive, and Hangouts
- Many users of GCP like Snap, Shopify, and Pokemon Go were also affected
- Cluster management software (Borg? K8s?) were accidentally included in a maintenance event causing many clusters to be de-scheduled
9) Google Cloud Outage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241212/pasted-from-clipboard.png)
10) DigitalOcean kills startup
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241252/Screen_Shot_2019-06-11_at_12.10.53_AM.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241262/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241265/pasted-from-clipboard.png)
10) DigitalOcean kills startup
- @w3nicolas runs a startup that hosts their service on DO
- They periodically need to run a script requiring them to scale up to use more droplets to run the script in parallel
- DigitalOcean's automation flagged this as crypto mining
- The account was locked on two separate occasions, the first human Abuse Operations agent failed to flag the account as approved
- As with all automation, the account was flagged again, in which the second Abuse Operations agent denied access
- DigitalOcean reached out to the customer to apologize and offer more credits
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241267/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1038755/images/6241269/pasted-from-clipboard.png)
Geodesic Cloud Automation Shell
The easy way to automate everything
By Erik Osterman, Cloud Architect @ Cloud Posse
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218783/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218732/pasted-from-clipboard.png)
Prometheus
How we ditched our legacy monitoring systems
By Florian Dambrine, MLOps Engineer @ GumGum
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218718/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6218810/pasted-from-clipboard.png)
Next WLAD Meetup
Date: mid-August 2019
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6005440/pasted-from-clipboard.png)
Next Meetup Preview
- Continuously delivering Terraform
- ?? Maybe you ??
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1017015/images/6005459/pasted-from-clipboard.png)
Getting Involved
- Have a handy tip?
- Want to speak at WLAD?
Email us: westladevops@gmail.com
...or message us on Meetup
...or talk to us right now.
West LA DevOps: The 2nd Meetup
By Corey Gale
West LA DevOps: The 2nd Meetup
- 925