Kubernetes is one of the most used container orchestration frameworks for people building, deploying and managing scalable micro-services. I won’t explain how it works since the goal of this post is not to describe how to use it but more precisely why it may be a threat to your organization (and others) when not properly secured.
Kubernetes consists of multiple elements split between master and node components (see https://kubernetes.io/docs/concepts/overview/components/ for more). The most interesting component for this post is kubelet (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/). It is an agent service running on each node that is part of a cluster and is responsible for making sure each specified container deployed on the node is healthy and running. Kubelet exposes a REST API that allows other nodes to manage the cluster, it listens on TCP/10250 and is reachable using HTTPS. Interestingly, no authorization or authentication is enabled per default, and anyone with access to the host (from the internet for example) can communicate with it. Recent versions of the software have fixed this issue. Still, it seems not everyone is updating.
This vulnerability is not new and was already reported several times in the past (see some references below). What makes it interesting is that not only can anyone retrieve information on the running cluster through this API (running pods, containers information, statistics, and metrics) but also get the ability to interact with any container running on the node. Tokens and information retrieved from the containers could potentially allow for escalation and compromise of the entire cluster.
How many of those open endpoints are actually reachable on IPv4? We started by running a port scan to identify all TCP/10250 ports opened on IPv4. There is a bit more than 2.5 million hosts with ports TCP/10250 opened out there, however, this does not mean each of those has a kubelet listening on it. We thus launched a fingerprinting scan using scannerl to identify which of those were kubelet services.
We identified a total of 31k kubelet services available. Among those 1495 are vulnerable endpoints, that is, services which allow anyone to query it and send commands to containers. Well, you might think that 1495 is not much (it is only around 5% of all reachable kubelet) but well, only one is enough for malicious users to use and potentially create their own little botnet.
A few workarounds are described on the kubelet-exploit page on how to quickly secure these endpoints: https://github.com/kayrus/kubelet-exploit#possible-workaround. A better solution is to update Kubernetes and correctly firewall the services.
The US and China had the highest numbers of vulnerable kubelet services. The US represented 38% of all vulnerable endpoints.
The ISPs from which those endpoints are reachable is also interesting. With no surprise, the big cloud providers (Amazon, DigitalOcean, Google, Microsoft, …) which provide managed Kubernetes services are in the top 10.
If you are testing micro-services using Kubernetes or running a production service with it, make sure to have a recently updated version that is adequately firewalled.
Thanks to my colleagues Benoît Knecht and François Deppierraz for their help.
References:
- https://github.com/kayrus/kubelet-exploit
- https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers
- https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c
- https://blog.rapid7.com/2018/06/27/analyzing-the-kubernetes-hack-backdooring-through-the-kubelet-api/
- https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/
- https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go
Kudelski Security Research 