Giant bags of mostly water

Konstantin Ryabitsev

First presented at Linux Security Summit 2015

Securing your IT infrastructure by expecting user error

Who am I

  • Programmer since 1989
  • Linux administrator since 1998
  • InfoSec since 2008
  • Linux Foundation since 2011

and, most importantly...

I'm a giant bag of mostly water

(and so are you)

This is a 1957 Chevy Bel Air

1957 Chevy Bel Air

  • 283 HP "Super Turbo Fire V8" engine
  • TurboGlide automatic transmission
  • Air Conditioning
  • Roomy seats
  • Smooth ride

Raw power and comfort

1957 Chevy Bel Air (contd)

Safety features:

  • Seat belts with separate shoulder harnesses
  • Triple-locking door latches
  • Padded dashboard
  • Steering wheel with a "safety-styled recessed hub"

"Vehicle interiors are so poorly constructed from a safety standpoint that it is surprising that anyone escapes from an automobile accident without serious injury."
- Journal of the American Medical Association, 1955

Unsafe at any speed

Ralph Nader's 1965 book famously blamed car manufactures for "designed-in dangers of the American automobile."

1960's engineers response:

What the hell are you talking about?

1960's car engineers:

  • We have designed the components to run safely!
  • Nothing leaks, explodes, or jabs you in the face
  • Nothing catches on fire under most conditions
  • Harmful fumes are kept well away from occupants
  • Components fail safely by stalling the engine
    and letting the vehicle coast to a stop

Most crashes are due to bad drivers!

The car is designed to drive, not to crash!

(sounds familiar?)

Car company responses:

  • Protecting drivers against crashes is expensive
  • Adding safety features sacrifices performance
  • Safety features significantly complicate everything
  • Safety features look ugly and reduce comfort
  • Problem much better solved by driver education
  • Customers are just not asking for it!

Fast-forward 50 years

This is a 2015 BMW i3

2015 BMW i3

  • 168 HP all-electric drive
  • Single speed fixed transmission
  • 0-60 in 7 seconds

Cute and green
but still plenty of torque

2015 BMW i3 (contd)

Safety features:

  • Carbon fibre roll cage ("life module")
  • Driver, passenger, side air bags
  • Entire front is crumple space
  • Seat belts with pre-tensioners
  • Collision sensors stop the car

Only 4-star safety rating!

Modern cars are all about driver errors!

See the results?

Wait, I thought we were going to talk about Linux security?

We still design IT infrastructure like we designed cars in the 1960s.

Raw power and comfort

  • More HZs!
  • More RAMs!
  • More cores!
  • Larger, faster disks!
  • Faster, lower-latency networks!
  • One-click deployment!
  • Containers, so anyone can deploy!

As an industry, we don't care about user security. We will gladly ship products with known security failings and no plans to update them.

- Matthew Garrett, 2015

This is as damning of a quote as the one from the American Medical Association

2015 IT engineers:

  • We have designed the components to run safely!
  • Nothing leaks, explodes, or jabs you in the face
  • Nothing catches on fire under most conditions
  • Harmful traffic is kept well away from users
  • Components fail safely by saving state and
    dumping core before crashing

Yes, but what about dumb things humans do?

IT company responses:

  • Protecting users against their own mistakes is expensive
  • Adding safety features sacrifices UI and usability
  • Safety features significantly complicate testing
  • Problem much better solved by user education
  • Customers are just not asking for it!

We are not responsible for user errors!


So we create an IT Security post and hire a smart person to help protect our infrastructure.

(that's you!)

You design the infrastructure like it's Helm's Deep

(and you should!)

Helm's Deep

  • First line of defence (the Wall)
  • Second line of defence (Elf Archers)
  • Third line of defence (Gimli)
  • Fourth line of defence (the Keep)
  • Final line of defence (Two dudes on horses)
  • Disaster Recovery (Gandalf)


But then one day...

And your boss is like...

"How did they get in?"

And you're like...

(Oh, darn, these memes are all from 2015!)

So, the forensic consultants are called and they find that:

  • The "backdoor" has always been there
  • In fact, this is how your admins get in
  • It's called, you know, "the VPN"
  • The attacker got in with one of the admin's credentials
  • And you can't just turn the VPN off
  • Because then your admins can't get in
  • Or they got in via an internal PHP app
  • ...which shouldn't have been remotely accessible
  • ...but was exposed because of a firewall rule
  • ...with a comment "temporary for Bob"
  • ...made 2 years ago
  • ...and nobody remembers who Bob was any more

You laugh, but I know your darkest fears.

Because they are also my own.

So, in the end PR will put out a statement making it sound like this happened:

Source: DUH

But you still have a problem.

How to prevent human mistakes from becoming security issues in the future.

Let's talk airbags

How can we reduce damage when something has already gone
terribly wrong?

We'll talk about:

  • Airbags for your networks
  • Airbags for your servers
  • Airbags for your admins
  • Seat belts for your team


  • Stateful firewalls
  • VLANs and zoning
  • VPNs
  • Routine NMAP scans

What you're probably already doing:

  • Payload inspection
  • IP reputation
  • Actually bothering with DNSSec
  • Actually checking TLS certs
  • Heck, actually using TLS

What you're probably NOT doing:


What you're probably already doing:

  • Virtualization for service isolation
  • Routine security updates
  • Centralized logging and monitoring
  • ...
  • SSH keys vs. just passwords?

We can do better than this!


  • Yes, SELinux! (or GrSec/AppArmor, pick one)
  • It's for when something has gone wrong
  • That PHP app has no business connecting to other sites
    • Or looking in /proc
    • Or in /etc/passwd
    • Or scanning your network
    • Or sending mail (spam)
  • If you could learn IPTables, you can learn SELinux
  • At least use it on your hypervisors!

Stop turning it off!

Use 2-factor auth

  • Make your admins enter an OATH token when performing sudo
    • Best to use hardware tokens, not soft-tokens running on phones
    • Look up Yubikeys, they are awesome
  • Put their ssh keys on smartcards
    • Yubikey NEO can do this, too
    • We publish detailed docs

Embrace containers

Yes, I did just say that.

Sysadmins at ContainerCon

Containers are not going to:

  • Make sysadmins a thing of the past
  • Let you "run any app safely"
  • Make security updates unnecessary
  • Replace VMs

They help to confine and isolate complex software stacks

Containers are like whole-OS static linking

Containers as airbags:

  • Bundle away complex stacks
  • Confine them with SELinux
    • You can even write detailed policies
      using MCS, but that's hardcore
  • Add extra network isolation via SDN

Containers as crash test dummies:

  • Easy to set up test/production environments
  • Easy to write a battery of pre-deployment tests
    • Check what ports it listens on
    • Check that it writes to the right log files
    • Check that it's (un)able to send mail
    • Check that it correctly rejects malicious input
  • Deploy to production if all tests pass

This is what DevOps is about: running Ops like you're Developing an app, not letting your devs run your ops.

That's when you end up with
curl | sudo sh


Woefully inadequate safety

  • Confining workstation apps is really hard
  • SELinux is near-useless on the desktop
  • X is the manifestation of pre-safety design
    • Security only partitions between users
    • Any app gets full access to all of user's X
    • Any app gets all keystrokes
    • Any app gets access to the whole screen
  • Any app gets access to microphone, camera

Workstations are notoriously bad at protecting us from harm when things go wrong.

A single vulnerable X app spells complete compromise of a user's desktop (and all their files).

Attackers don't even need root access.

X must die

Yay Wayland, amiright?

X won't die

For quite some time

That said, I've been running Wayland for months with excellent results. Try it!

Q: What do you call software written with the sole purpose of downloading and executing arbitrary code on a user's system, without their explicit consent?

A: Web browser

Web browsers:

  • Fantastically complex applications
  • ... with huge attack surfaces
  • ... that are extended with proprietary plugins
  • ... and add-ons that run with minimal protection
  • ... from un-signed sources

And we can't live without them.

The PDF reader in Mozilla Firefox allows remote attackers to read arbitrary files or gain privileges, as exploited in the wild in August 2015.

Don't get cocky, Webkit users, this could have happened to you.

Things that are not safe enough on a user's workstation:

  • Keystrokes (passwords)
  • Files (private keys)

The most unprotected system on your network is your sysadmin's workstation.

Sysadmin's workstations:

  • Are probably on the VPN, with full access
  • Probably have access to the password vault
  • Have privileged ssh keys on them
    • (never allow to ssh in as root, even with a pubkey)
  • Have a list of systems they can ssh to
  • Are awash in dangerous keystrokes
    • sudo passwords
    • root passwords
    • password vault master passwords


  • Require One-Time Passwords to elevate privileges
    • Best if not a phone app, but better than nothing
  • Keep private keys on smartcards
    • this includes ssh private keys
    • we publish a detailed guide on how to do it
  • Create policies and educate


  • The only serious attempt at workstation security
  • Works by creating virtualized "AppVMs"
    • Apps are isolated from others via Xen
    • Sidesteps most X vulnerabilities
    • Minimizes impact of when things go wrong
  • Is the Volvo of Linux Distros


  • Similar concept to QubesOS, but with containers
    • QubesOS offers more hardware isolation
  • Built-in TOR integration for all networking
  • GrSecurity/PAX kernel hardening
  • Secure mail and IM clients developed by Subgraph

XDG-app and Snap

  • Application-level sandboxing
  • Preview technology, but rapidly maturing
  • Useless unless Wayland/Mir are also used

To watch in 2016-2017


We still have to largely rely on user education.

So prepare for failures.

Establish secure comms

  • Set up and maintain a web of trust
  • Or use an exclusively trusted CA
  • Set up and require trusted email exchange
  • Set up and require trusted IM exchange
  • Establish a policy on what can never go into cleartext
  • Enforce that policy to show that you mean business
    • Less "hey, try not to send passwords in an email"
    • More "stick with the security policy, copy attached"

Establish workstation security guidelines

  • Create a checklist for workstation hardening
  • Provide examples of secure workflows, e.g.:
    • PGP and SSH keys on smartcards
    • Use a dedicated browser for work
    • Run the other browser in a VM
  • Have a recommended configuration
    • For hardware
    • For distro
    • For applications

Use code review

  • Use dedicated software or just pull requests
  • Require sign-off (for later shaming)
  • Do the same for sysadmins
    • Use config management backed by git

Write tests and use CI

  • These are your crash test dummies
  • You can use CI with sysadmins, too
    • This is where containers are great
    • But can be used with VMs, too
    • (this won't be easy!)

FYI, sysadmins will hate you for this.

Create and use checklists

  • For system deployment
  • For code review
  • For staff on-boarding
  • For staff close-out
  • For rapid admin lockout

Checklists are your most powerful tool to avoid the "oh sh*t moments"

Mistakes will happen

It is our job to make sure they are not fatal

In conclusion

We are at a fork in the road

We are here ⇒

We build systems that are great to drive

But that do not forgive mistakes

We are overdue

For our industry's own "unsafe at any speed" moment.


(And with in-vehicle computing taking off, it's not going to be a metaphor for much longer.)

Our current approach:

  • Padded dashboards
  • Optional seatbelts
  • "Safety-styled recessed hubs"

And a large dose of victim blaming.

The technology is there!

  • Seccomp
  • Namespaces
  • Safer languages
  • SELinux/AppArmor/PaX
  • Decent crypto

Use them in your stuff!

Kernel devs:

We need more airbags!

Distro devs:

Put in those airbags!


Stop turning them off!


Stop asking sysadmins to turn them off!


Foster secure team practices

And learn how encrypted email works,
it won't kill you.

We are opening up and sharing the policies the Linux Foundation IT team uses.

Help us improve them.

Let's not take 50 years to get to the point where computing is fun, powerful, and is a lot less likely to maim you when you make a mistake.

Thank you!


Giant bags of mostly water

By Konstantin Ryabitsev

Giant bags of mostly water

(Generic) Securing your infrastructure by securing your humans.

  • 4,733