West LA DevOps

August 29, 2019

meetup.com/West-LA-DevOps

Agenda

  1. Job Board
  2. Industry Updates
  3. Tech talk: CI/CD Pipelines for Microservices Best Practices by Dan Garfield
  4. Closing notes

About GumGum

  • Computer Vision company
  • Advertising division
    • Context-aware ads
    • Brand safety technology
  • Sports division

>20M RPM

Job Board

DevOps Engineer @ GumGum (3)

Data Engineer, Senior Python Dev @ GumGum

Job Board

DevOps Engineer @ AuditBoard

Backend/Frontend Engineer @ AuditBoard

Job Board

DevOps Engineer @ Your company?

Since the last time we met..

1) Recent security fails

CapitalOne Hack

  • 100m people affected, 140k socials numbers leaked
  • The hacker (Paige Thompson) bragged on Meetup, Slack, Twitter
    • Meetup: "Seattle Warez Kiddies"
  • Paige previously worked for AWS
  • Posted CapitalOne data on GitHub, which was detected and reported by a whitehat hacker
  • CapitalOne says the breach will cost them $150m
  • Equifax will pay $650m to settle their 2017 data breach that exposed data of 147 million consumers

Kubernetes Exploit

  • Security issue has been found in the net/http library of the Go language
  • Allows untrusted clients to allocate an unlimited amount of memory, until the server crashes
  • This vulnerability can result in a DoS against any process with an HTTP or HTTPS listener
  • Fixed in Kubernetes versions 1.13.10, 1.14.6, 1.15.3

2) AWS Advances DevOps Agenda at Summit Event

  • CTO Werner Vogels said ".. these latest services are part of an ongoing effort to automate DevOps processes to the point at which devs will need to write only the code for business logic"
  • Starts with a tool for automating VPC creation called CDK (Cloud Development Kit)
  • Launched aside AWS EventBridge, an event bus for integrating data on AWS with data residing in software-as-a-service (SaaS)

3) New AWS EC2 Features

4) GDPR exposed

  • Security researcher did a talk at the Black Hat conf on how he used GDPR to gather information about his fiancee
  • Used GDPR time limit to put pressure on companies to provide the data
  • Drew personal information he gathered from an initial set of companies to answer questions from the later set of companies
  • Replies included credit card info, passwords, travel data, login data, and social security number
  • Official Vault Helm Chart is released (Vault on K8s in minutes!)
  • HashiCorp team has mentioned lots of effort in aiding with secrets management much more easily in K8s via Vault
  • Vault integrated storage backend released allowing Vault to be run as a standalone system

5) Vault Improvements

  • "every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide"
  • Cloudflare was down for 27 minutes and this was their first global outage in 6 years
  • At its worst traffic dropped by 82%
  • A single misconfigured rule within the Cloudflare Web Application Firewall (WAF)
  • Pegged CPUs resulted in customers getting 502s
  • Because it was a global outage, their initial speculation was that it could be an attack (who wouldn't think this?)

6) Cloudflare Outage

  • WAF rule contained a regular expression that caused CPU to spike to 100%
  • CPU usage protection accidentally removed week prior with a change to improve CPU usage
  • Non-emergency WAF rule deployed globally
  • WAF Regex that caused CPU exhaustion:
    (?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))

10) Cloudflare Postmortem

CI/CD Pipelines

for Microservices

Best Practices

By Dan Garfield, Chief Technology Evangelist at Codefresh

Next WLAD Meetup

Date: mid-October 2019

Getting Involved

  • Have a handy tip?
  • Want to speak at WLAD?

 

Email us: westladevops@gmail.com

...or message us on Meetup

...or talk to us right now :)

Made with Slides.com