Kubernetes Best Practises for Cluster admins (not creators)

Wait. What's the difference?

Today, with managed Kubernetes becoming more common, your cluster creator could be a public cloud provider, a third party company or, indeed, perhaps you. As time goes on, I believe these roles will continue to diverge as Kubernetes becomes more commoditised. 

Authentication, Authorization and Admission Control

AuthN

• AuthN and AuthZ are strictly separated in K8S. It's only who you are and the groups you're in.
• Types of Authentication include:
  • Client Certificates
  • Bearer Tokens
  • Basic Auth
  • OpenID Tokens
  • Webhook Authenticators
• AuthN has a username, UID and Groups that are passed to AuthZ

AuthZ

• AuthZ defines what you're allowed to do in the cluster, either cluster wide or namespace restricted
• RBAC is the default authorization system today, replacing ABAC
• There are also third party webhooks you can use for AuthZ with Open Source projects allowing you more complex control over your resources

Admission Controllers

• Admission controllers intercept incoming requests and can make decisions on them
• Special Admission Controllers are the mutating and validating webhook controllers, which allows for controllers not compiled into the apiserver

Recommended Controllers to enable

Note: all of these are default, but v1.13 also adds PersistentVolumeClaimResize to the default

Pod Security Policies

• Allows for fine grained control over what pods can and cannot do
• For example, they can restrict things like:
  • Which host paths can be mounted, if any
  • If a pod is allowed to be privileged
  • Which hostPorts are allowed to be used
  • Which volume types are allowed to be used
  • Plus more

Network Policies

• Allows for fine grained control over what pods are allowed to talk to
• Effectively an ACL and allows specification of source, destination, ingress, egress, port numbers and protocols
• Requires a network provider that supports them, which include (alphabetically)
  • Calico
  • Canal
  • Kube-Router
  • Romana
  • Weave Net 

Ok great! You've done everything possible as an admin to secure with best practises

Why, that's a lovely cluster you have there. Sure would be a shame if something were to... happen to it

CVE-2018-1002105

Pretty much every Kubernetes admin on Dec. 3rd

It comes in two flavours, using the same underlying exploit

• Anonymous Information leak via aggregated API Servers (metrics server or service catalog are prime targets here)
• Privilege escalation to the Kubelet API

It's a 9.8 out of 10 on the severity score for a CVE

Demo time!

How do I fix it?

Option 1: disable anonymous-auth on the kube-apiserver. The default is true:

Option 2: ACL your ApiServer

Option 3 (the best): Upgrade!

Privilege escalation via the Kubelet API

Communication from Kubelet to ApiServer is almost always Kubelet -> Apiserver

For example, creating a pod:

But if you need to exec to a pod, the apiserver needs to talk to the Kubelet. Usually via port 10250

We can abuse this port to gain access if Kubelet Authentication has not been set up

Demo time number 2!

How do I fix it?

Option 1: disable anonymous-auth on the kubelet. Once again, the default is true:

Option 2: Add authentication and authorization to your kubelets using either Certs or Tokens by following the steps here: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/

Privilege escalation to the Kubelet via the second version of CVE-2018-1002105

Similar to our first exploit, we are using the open upgraded connection to escalate privileges

Demo time!

Come on WiFi and Demo Gods, don't let me down now

How do I fix it?

Option 1: disallow your users to use exec, attach etc. privileges

Option 2: upgrade ASAP!

Privilege escalation via the cloud provider

IAM in public clouds is provided via the metadata server, so your pods appear to the metadata server as the nodes

Demo time!

Yes, I'm crazy enough to attempt 4 demos in 45 minutes

How do I fix it?

Ensure that your nodes are created with read only permissions to your cloud provider

Review your permissions! If someone accidentally downloaded a docker image with this kind of code inside it you could face a hefty bill