[*]Kubeflow, the machine learning toolkit for Kubernetes, is the victim of a security breach. An attack vector that hurts the securing of the clusters of this container orchestrator.
A few days after the discovery of Kubernetes network implementations vulnerable to attack, Microsoft alerted this Wednesday on another vector of compromise affecting this container orchestrator. Although it is not directly a Kubernetes security flaw as such, the latter has already undermined several dozen Kubernetes clusters according to the Microsoft Azure Security Center (ASC) team. The origin of the flaw lies in Kubeflow, an open source framework dedicated to machine learning in Kubernetes.
“In April, we observed the deployment of a suspicious image from a public repository on many different clusters. The image is ddsfdfsaadfs / dfsdf: 99, “alerted Microsoft. “By inspecting the image layers, we can see that this image is running an XMRIG miner.” As part of its investigations, the team of security researchers was able to establish that the Kubeflowx framework can be used as an attack vector. “Kubeflow creates several CRDs in the cluster which expose certain functionalities on the API server”, continues the research team. But that’s not all because Kubeflow also exposes its user interface functionality via a dashboard that is deployed in the cluster.
A backdoor deployable in a container to compromise a cluster
“The dashboard is exposed by the Istio entrance gateway, which is by default accessible only internally. As a result, users must use port forwarding to access the dashboard that tunnel traffic through the Kubernetes API server, ”said Microsoft. “In some cases, users change the setting of the Istio service on Load-Balancer which exposes the service (istio-ingressgateway in the istio-system of the namespace) to the Internet. We believe that some users have chosen to do so for convenience: without this action, access to the dashboard requires tunneling via the Kubernetes API server and is not direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation allows insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including the deployment of new containers in the cluster. ”
Once the Kubeflow dashboard, hackers are then able to deploy a backdoor in one of the containers contained in a new cluster specially created for malicious purposes. To deploy it, hackers can for example choose to create a personalized image of a Jupyter notebook server, or even use a real Jupyter notebook to run their Python code. “The code runs from the notebook server, which is a container in itself with a mounted service account. This service account (by default) has permissions to deploy containers in its namespace. Therefore, attackers can use it to deploy their backdoor container in the cluster, “says Microsoft.
Vigilance on load-balancing shared with a public IP address
To protect yourself, several checks are necessary such as ensuring that no malicious container is deployed in the cluster via the command kubectl get pods –