West LA DevOps

October 24, 2019



  1. Job Board
  2. Industry Updates
  3. Tech talk #1: Dancing with Data and Distributed Systems by Jasmine Dahilig
  4. Tech talk #2: Service Mesh for Everyone by Jake Lundberg
  5. Closing notes

About GumGum

  • Computer Vision company
  • Advertising division
    • Context-aware ads
    • Brand safety technology
  • Sports division

>20M RPM

Online Advertising

Did you know?

GumGum Invented In-Image advertising in 2008

Job Board

DevOps Engineer @ GumGum (x2)

Job Board

DevOps Engineer @ AuditBoard

Backend/Frontend Engineer @ AuditBoard

Job Board

DevOps Engineer @ Your company?

Since the last time we met..

1) NordVPN Breach

  • An unknown attacker gained access to a VPN server for "about a month" by exploiting an insecure remote management system installed by the datacenter provider
  • NordVPN was unaware such a system existed
  • A NordVPN spokesperson claimed “the server itself did not contain any user activity logs" and added "usernames and passwords couldn’t have been intercepted either”
  • Security researcher warned that NordVPN was ignoring the larger issue of network access. “Your car was just stolen and taken on a joy ride and you’re quibbling about which buttons were pushed on the radio?”

Solution: Algo

  • Ansible scripts that simplifies the setup of a personal VPN
  • Lightweight
  • Uses the most secure defaults available
  • Works on most public clouds

2) New AWS Features

2) New AWS Features Cont'

3) DockerHub Outage

  • On October 15, 2019 10:06 AM DockerHub went down
    • DockerHub Web, DockerHub Registry 😲, DockerHub Automated Builds
  • Many of our Docker builds broke 😧
  • Some of our systems were unable to scale-out 😵
  • At 11:04 AM the DockerHub Registry service was restored and at 2:48 PM all services were restored 🙏🏻 
  • Reminder: DockerHub is a free service with no service SLA 🤔
  • Solutions: pull-through cache, ECR, kraken 👍

4) HTTP Desync Attacks

  • HTTP/1.1 vulnerability exploited allowing an attacker to prepend content to another legitimate request. This is known as request smuggling
  • This vulnerability exists because of the introduction of keep-alives in HTTP/1.1, sending multiple requests over a single TCP/SSL socket
  • Because HTTP/1.1 requests are placed back to back, it is crucial that the frontend and backend agree on where each request ends and starts
  • Content-Length and Transfer-Encoding headers typically used to determine when a request ends

4) HTTP Desync Attacks

4) HTTP Desync Attacks

Host: example.com
Content-Length: 6
Transfer-Encoding: chunked


Host: example.com



Host: example.com
Content-Length: 47
Transfer-Encoding: chunked


POST /users/{my_id}/make_admin

X-Ignore-Header: POST / HTTP/1.1

Host: example.com

Access-Token: {superuser_access_token}

5) GitHub Actions CI/CD

  • GitHub is finally introducing their own CI/CD solution
  • General availability will be on November 13th, Beta is out now
  • YAML based config using container actions 
  • Huge shift in the community towards GitOps
  • Coming soon: self-hosted GitHub Actions runners (free!)
  • Out-of-the-box remote state management
    • Previously needed to set up remote state + state locking for collaborative work
  • Performing terraform plan + apply remotely
    • This introduces CI/CD to your Infra-As-Code, which is naturally the next step
  • VCS Webhooks allows GitOps based infrastructure deployments
    • PRs into master with a terraform plan check
    • Merge into master kicks off a terraform apply

6) Terraform Cloud

Dancing with Data & Distributed Systems

By Jasmine Dahilig, Software Engineer at HashiCorp

Service Mesh for Everyone

By Jake Lundberg, Staff Solutions Engineer at HashiCorp


Next WLAD Meetup

Date: January 2020

Getting Involved

  • Interested in speaking at WLAD?
  • Have an idea for a future talk?


Email us: westladevops@gmail.com

...or message us on Meetup

...or talk to us right now :)

West LA DevOps: The 4th Meetup

By Corey Gale

West LA DevOps: The 4th Meetup

  • 828