Pilfering Kubernetes Clusters

Kubernetes is one of the most used container orchestration frameworks for people building, deploying and managing scalable micro-services. I won’t explain how it works since the goal of this post is not to describe how to use it but more precisely why it may be a threat to your organization (and others) when not properly secured.

Kubernetes consists of multiple elements split between master and node components (see https://kubernetes.io/docs/concepts/overview/components/ for more). The most interesting component for this post is kubelet (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/). It is an agent service running on each node that is part of a cluster and is responsible for making sure each specified container deployed on the node is healthy and running. Kubelet exposes a REST API that allows other nodes to manage the cluster, it listens on TCP/10250 and is reachable using HTTPS. Interestingly, no authorization or authentication is enabled per default, and anyone with access to the host (from the internet for example) can communicate with it. Recent versions of the software have fixed this issue. Still, it seems not everyone is updating.

This vulnerability is not new and was already reported several times in the past (see some references below). What makes it interesting is that not only can anyone retrieve information on the running cluster through this API (running pods, containers information, statistics, and metrics) but also get the ability to interact with any container running on the node. Tokens and information retrieved from the containers could potentially allow for escalation and compromise of the entire cluster.

How many of those open endpoints are actually reachable on IPv4? We started by running a port scan to identify all TCP/10250 ports opened on IPv4. There is a bit more than 2.5 million hosts with ports TCP/10250 opened out there, however, this does not mean each of those has a kubelet listening on it. We thus launched a fingerprinting scan using scannerl to identify which of those were kubelet services.

We identified a total of 31k kubelet services available. Among those 1495 are vulnerable endpoints, that is, services which allow anyone to query it and send commands to containers. Well, you might think that 1495 is not much (it is only around 5% of all reachable kubelet) but well, only one is enough for malicious users to use and potentially create their own little botnet.

A few workarounds are described on the kubelet-exploit page on how to quickly secure these endpoints: https://github.com/kayrus/kubelet-exploit#possible-workaround. A better solution is to update Kubernetes and correctly firewall the services.

kubelet_vuln_map

The US and China had the highest numbers of vulnerable kubelet services. The US represented 38% of all vulnerable endpoints.

kubelet_by_country

The ISPs from which those endpoints are reachable is also interesting. With no surprise, the big cloud providers (Amazon, DigitalOcean, Google, Microsoft, …) which provide managed Kubernetes services are in the top 10.

kubelet_by_isp

If you are testing micro-services using Kubernetes or running a production service with it, make sure to have a recently updated version that is adequately firewalled.

Thanks to my colleagues Benoît Knecht and François Deppierraz for their help.

References:

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s