West LA DevOps
August 29, 2019
meetup.com/West-LA-DevOps
Agenda
- Job Board
- Industry Updates
- Tech talk: CI/CD Pipelines for Microservices Best Practices by Dan Garfield
- Closing notes
About GumGum
- Computer Vision company
- Advertising division
- Context-aware ads
- Brand safety technology
- Sports division
>20M RPM
Job Board
DevOps Engineer @ GumGum (3)
Data Engineer, Senior Python Dev @ GumGum
Job Board
DevOps Engineer @ AuditBoard
Backend/Frontend Engineer @ AuditBoard
Job Board
DevOps Engineer @ Your company?
Since the last time we met..
1) Recent security fails
CapitalOne Hack
- 100m people affected, 140k socials numbers leaked
- The hacker (Paige Thompson) bragged on Meetup, Slack, Twitter
- Meetup: "Seattle Warez Kiddies"
- Paige previously worked for AWS
- Posted CapitalOne data on GitHub, which was detected and reported by a whitehat hacker
- CapitalOne says the breach will cost them $150m
- Equifax will pay $650m to settle their 2017 data breach that exposed data of 147 million consumers
Kubernetes Exploit
- Security issue has been found in the net/http library of the Go language
- Allows untrusted clients to allocate an unlimited amount of memory, until the server crashes
- This vulnerability can result in a DoS against any process with an HTTP or HTTPS listener
- Fixed in Kubernetes versions 1.13.10, 1.14.6, 1.15.3
2) AWS Advances DevOps Agenda at Summit Event
- CTO Werner Vogels said ".. these latest services are part of an ongoing effort to automate DevOps processes to the point at which devs will need to write only the code for business logic"
- Starts with a tool for automating VPC creation called CDK (Cloud Development Kit)
- Launched aside AWS EventBridge, an event bus for integrating data on AWS with data residing in software-as-a-service (SaaS)
3) New AWS EC2 Features
-
New EC2 instances: I3en, C5n bare metal instances
- Direct access to the Xeon CPU feature set
- Amazon EC2 now supports Diagnostic Interrupts
- Helps debug unresponsive instances and conduct RCAs (can view memory dumps)
-
New Capacity-Optimized Allocation Strategy for provisioning Amazon EC2 Spot instances
- Considers spot instance market availability
- Enforce cost control: EC2 Fleet now lets you set a max spend for a fleet of instances
4) GDPR exposed
- Security researcher did a talk at the Black Hat conf on how he used GDPR to gather information about his fiancee
- Used GDPR time limit to put pressure on companies to provide the data
- Drew personal information he gathered from an initial set of companies to answer questions from the later set of companies
- Replies included credit card info, passwords, travel data, login data, and social security number
- Official Vault Helm Chart is released (Vault on K8s in minutes!)
- HashiCorp team has mentioned lots of effort in aiding with secrets management much more easily in K8s via Vault
- Vault integrated storage backend released allowing Vault to be run as a standalone system
5) Vault Improvements
- "every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide"
- Cloudflare was down for 27 minutes and this was their first global outage in 6 years
- At its worst traffic dropped by 82%
- A single misconfigured rule within the Cloudflare Web Application Firewall (WAF)
- Pegged CPUs resulted in customers getting 502s
- Because it was a global outage, their initial speculation was that it could be an attack (who wouldn't think this?)
6) Cloudflare Outage
- WAF rule contained a regular expression that caused CPU to spike to 100%
- CPU usage protection accidentally removed week prior with a change to improve CPU usage
- Non-emergency WAF rule deployed globally
- WAF Regex that caused CPU exhaustion:
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))
10) Cloudflare Postmortem
CI/CD Pipelines
for Microservices
Best Practices
By Dan Garfield, Chief Technology Evangelist at Codefresh
Next WLAD Meetup
Date: mid-October 2019
Getting Involved
- Have a handy tip?
- Want to speak at WLAD?
Email us: westladevops@gmail.com
...or message us on Meetup
...or talk to us right now :)
West LA DevOps: The Third Meetup
By Corey Gale
