West LA DevOps
August 29, 2019
meetup.com/West-LA-DevOps
Agenda
- Job Board
- Industry Updates
- Tech talk: CI/CD Pipelines for Microservices Best Practices by Dan Garfield
- Closing notes
About GumGum
- Computer Vision company
- Advertising division
- Context-aware ads
- Brand safety technology
- Sports division
>20M RPM
Online Advertising
Did you know?
GumGum Invented In-Image advertising in 2008
Job Board
DevOps Engineer @ GumGum (2)
Job Board
DevOps Engineer @ AuditBoard
Backend/Frontend Engineer @ AuditBoard
Job Board
DevOps Engineer @ Your company?
Since the last time we met..
1) Twilio adds DevOps tool
- At it's annual SIGNAL conference, Twilio announced a new CLI tool for deeper telemetry integrations
- Senior director describes it as a server-less tool that "will make it possible to incorporate Twilio cloud services within a larger DevOps toolchain"
- Provides access to SMS, audio and video services
-
What can we do with this?
- SMS approvals for prod deploys
- ???
2) Recent security fails
CapitalOne Hack
- 100m people affected, 140k socials numbers leaked
- Hacker (Paige Thompson, 33) previously worked for AWS
- Bragged on Meetup, Slack, Twitter
- Meetup: "Seattle Warez Kiddies"
- Posted Capital One data on GitHub, which was detected and reported by a whitehat hacker
- CapitalOne says the breach will cost them $150m
- Equifax will pay $650m to settle their 2017 data breach that exposed data of 147 million consumers
Kubernetes Exploit
- Security issue has been found in the net/http library of the Go language
- Allows untrusted clients to allocate an unlimited amount of memory, until the server crashes
- This vulnerability can result in a DoS against any process with an HTTP or HTTPS listener
- Fixed in Kubernetes versions 1.13.10, 1.14.6, 1.15.3
3) AWS Advances DevOps Agenda at Summit Event
- CTO Werner Vogels said ".. these latest services are part of an ongoing effort to automate DevOps processes to the point at which devs will need to write only the code for business logic"
- Starts with a tool for automating VPC creation called CDK (Cloud Development Kit)
- Launched aside AWS EventBridge, an event bus for integrating data on AWS with data residing in software-as-a-service (SaaS)
4) VMware goes shopping
- Announced plans to acquire Pivotal Software, a cloud-native platform provider, for $2.7b
- VMware also announced an acquisition of cybersecurity firm Carbon Black for about $2.1b
- Both VMware and Pivotal are owned by Dell Technologies
- VMware CEO Pat Gelsinger: "With these actions we meaningfully accelerate our subscription and SaaS offerings and expand our ability to enable our customers' digital transformation."
- Pivitol's stock (NYSE: PVTL) shot up 69%
5) New AWS EC2 Features
-
New EC2 instances: I3en, C5n bare metal instances
- Direct access to the Xeon CPU feature set
- Amazon EC2 now supports Diagnostic Interrupts
- Helps debug unresponsive instances and conduct RCAs (can view memory dumps)
-
New Capacity-Optimized Allocation Strategy for provisioning Amazon EC2 Spot instances
- Considers spot instance market availability
- Enforce cost control: EC2 Fleet now lets you set a max spend for a fleet of instances
- Huge focus on extensibility and work on Custom Resource Definitions (CRDs)
- Improvements to cluster lifecycle stability (installation and upgrades of kubernetes clusters)
- Future CVE patches will only be for v1.13, v1.14, v1.15
- Who is still running on v1.11 or v1.12?
- Latest Versions:
- Google GKE: 1.13.7
- Amazon EKS: 1.13.8
- Azure AKS: 1.14.6, 1.13.10
6) Kubernetes v1.15
7) GDPR exposed
- Security researcher did a talk at the Black Hat conf on how he used GDPR to gather information about his fiancee
- Used GDPR time limit to put pressure on companies to provide the data
- Drew personal information he gathered from an initial set of companies to answer questions from the later set of companies
- Replies included credit card info, passwords, travel data, login data, and social security number
- Official Vault Helm Chart is released (Vault on K8s in minutes!)
- HashiCorp team has mentioned lots of effort in aiding with secrets management much more easily in K8s via Vault
- Vault integrated storage backend released allowing Vault to be run as a standalone system
8) Vault Improvements
- GitHub is finally introducing their own CI/CD solution
- General availability will be on November 13th, Beta is out now
- YAML based config using container actions
- Huge shift in the community towards GitOps
- Coming soon: self-hosted GitHub Actions runners (free!)
9) GitHub Actions CI/CD
- "every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide"
- Cloudflare was down for 27 minutes and this was their first global outage in 6 years
- At its worst traffic dropped by 82%
- A single misconfigured rule within the Cloudflare Web Application Firewall (WAF)
- Pegged CPUs resulted in customers getting 502s
- Because it was a global outage, their initial speculation was that it could be an attack (who wouldn't think this?)
10) Cloudflare Outage
- WAF rule contained a regular expression that caused CPU to spike to 100%
- CPU usage protection accidentally removed week prior with a change to improve CPU usage
- Non-emergency WAF rule deployed globally
- WAF Regex that caused CPU exhaustion:
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))
10) Cloudflare Postmortem
CI/CD Pipelines
for Microservices
Best Practices
By Dan Garfield, Chief Technology Evangelist at Codefresh
Next WLAD Meetup
Date: mid-October 2019
Next Meetup Preview
- <AudidBoard guy?>
- ?? Maybe you ??
Getting Involved
- Have a handy tip?
- Want to speak at WLAD?
Email us: westladevops@gmail.com
...or message us on Meetup
...or talk to us right now :)
West LA DevOps: The Third Meetup
By Corey Gale
West LA DevOps: The Third Meetup
- 1,210