December 12, 2022
Many companies that use Kubernetes are still highly concerned with the security of their systems. However, it is remarkable that these security concerns are not related to the in-built risks of the Kubernetes system itself. Instead, the safety issue is significant because of how complex Kubernetes is, which makes it difficult even for skilled cloud-native developers to navigate the platform.
Security is a priority for any business serious about protecting its data. With Digital Data as your consultant, you can ensure that your Kubernetes implementation is robust and secure – our team of certified professionals will help ensure your organization benefits from all the advantages of a complex system without compromising security. In addition, Digital Data consultants can help you utilize Kubernetes with confidence.
A recent study by StackRox indicated that a steep learning curve, inadequate skills in the labor market, and risks from misconfigurations are the leading causes of Kubernetes security breaches. The study was conducted on more than 540 respondents, where more than 94% of this group had suffered substantial security threats in the past year!
However, they are not the only ones to fall victim; below is a brief list of significant Kubernetes security incidents that have occurred recently.
Capital One
This Kubernetes security incident involving Capital One had significant ramifications and caused many people to wake up and take note of the threat they were dealing with. It occurred precisely a year ago and resulted in the exfiltration of 30GB of credit application information involving about 106 million customers.
The actual cause of the security breach was misconfiguration, an occurrence that we often see in the Kubernetes industry. Specifically, in this case, a misconfigured firewall enabled the attacker to access internal metadata and get credentials of an Amazon Web Services IAM role that did not need to be that “broad” to begin with.
From this incident, we can learn the all-important lesson of being cautious when assigning IAM roles. Many individuals are always in haste, trying to implement Kubernetes and get it to function. As a result, they frequently need to pay more attention to critical steps such as secrets and services management and assigning IAM roles on a per pod basis and not per application.
Another critical step is to change manually and “roll” credentials – if possible, use an automated service that limits the number of times credentials are renewed. In addition, this move also sets an upper limit on the duration a breach can endure.
Docker Hub
It is difficult to anticipate where an attack might come from. This is because, with Kubernetes, distributed environments, containers, and the surface exposed to an attack become increasingly more significant. For example, this is how the attack succeeded in embedding malicious pictures in the Docker hub last year, setting up anyone who uses those pictures to fall victim to “cryptojacking.” This means that users unintentionally set up cryptocurrency miners like Docker containers that illegally use resources to help the attacker mine cryptocurrency. Unfortunately, this is only one of several attacks of similar nature we have witnessed lately.
Just like in the Capital One case, it is important to change passwords and roll credentials regularly to prevent this situation from occurring. In addition, to guarantee security when working with Kubernetes, you must rotate your secrets and audit images to ensure that only authenticated images are being used.
It can be quite challenging to detect malicious images since, most of the time, the containers will function as intended. For this reason, extra checks are necessary to identify any deviations in the application functioning to guarantee that no stowaway processes occur in the background. Unfortunately, this form of attack is quite profitable to the attacker.
Microsoft Azure
Microsoft is another example of a major organization that has become a victim of many cryptojacking incidents. In April this year, it was revealed that there was a wide-scale crypto-mining attack on the Kubernetes cluster in Azure. Later in June, another attack aimed at misconfigured Kubeflow containers to convert them into crypto miners.
Like the compromised image incident with Docker hub, Kubeflow relies on several services that enable users to use images like Jupyter and Katib notebook server. For Jupyter, the selected image doesn’t need to be an authentic notebook image – this is where the attackers gained access. If we study the cause of this misconfiguration, it is extremely similar to the causes of every misconfiguration, including laziness, impatience, and inadequate knowledge.
By default, Kubflow’s UI dashboard is only accessed internally via an Istio ingress gateway. Unfortunately, several users found a shortcut and directly accessed the dashboard without passing via the Kubernetes API server. As a result, the users did not know that they were exposing their dashboard to an attack through the backdoor in their attempts to save time. Essentially, this error had given internet users access to the dashboard through the Istio ingress gateway. Here, we learn that every change in settings or configuration brings profound security concerns to the organization.
Tesla
As the value of cryptocurrencies is rising steeply and more computing resources are being located in the cloud, cases of hijacking resources and data theft have become more profitable for attackers. For example, auto manufacturing company Tesla suffered a cryptojacking incident after a Kubernetes cluster was breached because an administrative console was not password protected.
This attack was revealed to the public through a report written by RedLock Cloud Security Intelligence. In the report released to the public, a misconfiguration enabled the attackers to access Tesla’s AWS S3 bucket credentials. The attackers used the credentials to run a crypto-mining script on a pod.
It is helpful to note that the attackers used numerous “ingenuine” precautionary measures to conceal themselves and avoid being found out. The attackers deliberately avoided using a recognized mining pool and instead used an unlisted one. Also, they relied on a popular CDN service Cloudflare to keep their IP hidden. Finally, they were keen to ensure the mining script did not consume enough CPU resources to raise the alarm or cause detection; they listened on a nonstandard port, meaning detecting any malicious activity based on port traffic was practically impossible. To detect such a security attack, you must monitor configurations to ensure all policies are respected actively.
Jenkins
During this security attack on Jenkins, which occurred around the same time as the Tesla breach, the attackers exploited the company’s system, and crypto mined about $3.5 million, or 10,800 Monero, in 18 months. Monero is similar to the cryptocurrency involving malicious Docker images, as earlier discussed. In the Docker incident, the security audit revealed that six malicious images had been pulled over more than 2 million times. This means that 2 million users were potentially mining Monero for the attackers, which is quite an exploit.
Of the above-highlighted incidences, the Jenkins Kubernetes security breach is perhaps the most daring security breach to be exposed. The Jenkins incident is also notable because it uses susceptible Windows machines and personal computers operating on Jenkins, therefore exposing Jenkins CI servers to the attack.
Also note recently, the malware has shown tremendous ability to pass through several lifecycles, continually updating itself and shifting mining pools to evade detection. In addition, the malware’s ability to reach servers signifies that the attackers, who are initially Chinese, have raised their game. If they can steal more than $3 million from old desktops, they can only cause greater harm using powerful servers.
The Reliability Of Kubernetes Security: The Platform’s Surface Is Constantly Increasing
Kubernetes security breaches are increasing because the attack surface now includes a limitless assemblage of hybrid clouds, on-premises data centers, personal computers, IoT devices, and edge devices, among others.
Closed-minded security has ended – nowadays, you cannot afford to merely focus on your application and use a firewall to protect the rest of the setup. This is because some attacks are likely to originate from a service being run by another service that is in use by a service that you are using. Also, everyone must play their role in security matters, meaning the cloud provider or Kubernetes manager can only help so far, and the rest of the work is yours.
We expect cases of cryptojacking to increase as more attackers find more ways of avoiding detection, apart from when the cloud services bills have risen remarkably high.
Digital Data Can Help
Digital Data is a full-stack, end-to-end consultancy that 100% specializes in architecting, securing, and deploying containerization & orchestration technologies from Mirantis on-premises and in any cloud on any operating system. Some of our deployments are in classified environments serving National Security Missions, where the need for hardening is paramount. We also focus on turning around failed or troubled projects with economics and timelines that make sense. We have a wide customer base and assist in CI/CD pipeline design, security, and optimization. Often times, we are chartered by the C-suite to lead, guide, and direct these modernization initiatives.