Securing Access to Kubernetes Environments with Zero Trust

Modern IT environments are becoming more dynamic by the day. Kubernetes, for example, is pushing the boundaries of what’s possible for many IT organizations.

The benefits of the open source technology to automate deployment, scalability and management of containerized applications are numerous. In particular, IT teams are taking advantage of its power, efficiency and flexibility to develop modern applications quickly and deliver them at scale.

However, the process of ensuring hardened security practices in Kubernetes environments is a growing challenge. As a more significant number of development and production Kubernetes clusters spread across on-premises data centers, multiple public cloud providers and edge locations, this relatively new and dynamic operating model creates major complexity for controlling access.

Since most teams have multiple clusters running in multiple locations — oftentimes with different distributions with management interfaces — enterprise IT needs to account for the teams of developers, operators, contractors and partners who require varying levels of access.

Given the distributed and expansive nature of Kubernetes, IT has to do everything possible to ensure access security to avoid the mistakes that are happening. Below, we’ll look at how to apply Kubernetes zero-trust principles to secure an entire environment, providing zero-trust security for containers.

Zero-Trust Access for Kubernetes Clusters

As a security model that automatically assumes all people, systems and services operating in and between networks cannot be trusted, zero trust is emerging as the best technique to prevent malicious attacks. Based on authentication, authorization and encryption technologies, the purpose of zero trust is to continuously validate security configurations and postures to ensure trust across an environment.

Here’s a basic understanding of how Kubernetes works:

• The core of the Kubernetes control plane for each cluster is the Kubernetes API server.
• API calls are used to query and manipulate the state of all Kubernetes objects.
• Kubernetes objects include namespaces, pods, configuration maps and more.

Controlling access to the use of APIs is the critical function to managing Kubernetes access and accomplishing zero trust. The first step in securing access to Kubernetes clusters is to protect traffic to and from the API server with Transport Layer Security (TLS).

Image Source: kubernetes.io

API server best practices for implementing zero trust:

• Enable TLS everywhere.
• Use a private endpoint for the API server.
• Use third-party authentication for the API server.
• Close firewall inbound rules to the API server, ensuring it is cloaked and not directly accessible from the Internet.

After securing the transport layer, Kubernetes also includes the necessary hooks to implement zero-trust and control API server access for each Kubernetes cluster. These hooks represent four critical areas of a hardened security posture for Kubernetes:

• Authentication
• Authorization
• Admission control
• Logging and auditing

Authentication for Kubernetes

With zero trust, all user-level and service-oriented accounts tied to Kubernetes clusters must be authenticated before executing an API call. Security modules and plugins are widely available for Kubernetes to ensure that the platform will operate effectively with a team’s preferred authentication system:

• HTTP Basic Auth
• Authentication Proxy (to support LDAP, SAML, Kerberos, etc.)
• Client certificates
• Bearer tokens
• OpenID Connect tokens
• Webhook Token authorization

Common best practices for authentication include enabling at least two authentication methods (multifactor authentication or MFA) and the rotation of client certificates regularly.

Authorization for Kubernetes

Allowing every user or service account with authenticated access to carry out any possible action in a Kubernetes cluster must be mitigated. With zero trust, the idea is that a request can only be authorized if an authenticated user has the necessary permission to complete the requested action. For each request made, this model will require specification of the username, action and the objects affected in the Kubernetes cluster.

There are numerous methods that Kubernetes supports for authorization, including:

• Attribute-based access control, or ABAC, authorizes access dynamically based on a combination of user, environment and resource attributes.
• Role-based access control, or RBAC, authorizes access based on the user’s role in the organization, such as developer, admin, security, etc.

Organizations most commonly use RBAC, as its practical nature allows for easier management controls and provides the granularity needed for most use cases. It is common within the industry to enable RBAC with the least privilege.

ABAC can provide additional granularity but requires additional time and resources to define and configure properly. However, troubleshooting an issue can be more challenging with the ABAC method. Therefore, it is common to enable RBAC with the least privilege.

Admission Control for Kubernetes

Admission controllers provide a way to implement business logic to refine a zero-trust approach to Kubernetes. The purpose of admission controllers is to enable the system to automatically act on requests that create, modify, delete or connect to Kubernetes objects. Enabling multiple admission controllers may be necessary to fit your organization’s needs, and if any one of them rejects a particular request, the system automatically rejects it as well.

The variety of built-in admission controllers available today allows teams plenty of options for enforcing policies and implementing various actions. Dynamic controllers enable the rapid modification of requests to adhere to established rule sets. For example, the ResourceQuota admission controller observes incoming requests and ensures they don’t violate the constraints that have been listed in the ResourceQuota object for a namespace. See Using Admission Controllers for more information.

Logging and Auditing for Kubernetes

Essential to a Kubernetes security posture, auditing capabilities provide a track record of the actions performed within a cluster. These capabilities can enable tracking of any action by any user, application and the control plane itself.

There are four different types of audit levels:

• None – Don’t log this event
• Metadata – Log request metadata
• Request – Log event metadata and the request
• RequestResponse – Log event metadata, the request and the response

In addition to specifying audit levels, teams can also control where the audited events are being logged. As the log backend authors events to the cluster’s local filesystem, the webhook backend sends audit events to an external logging system.

Scaling Zero-Trust Architecture

While the different methods and practices described above provide the ability to create a zero-trust environment, configuring and aligning these individual elements properly becomes a more significant challenge when a Kubernetes footprint expands beyond a few clusters. Things get especially complicated when multiple workloads and Kubernetes distributions are involved. This challenge is not new, but is shared by many companies today.

For example, let’s consider a scenario where a company is managing 100 Kubernetes clusters — ranging from development to QA to staging to prod — and the clusters are required to be geographically close to its global customer base for applications to work with real-time streams of video and audio data.

There are three problems this company could encounter with regard to ensuring secure user access to Kubernetes clusters:

1. Assuming this company has a few hundred developers and a few dozen IT operations personnel, the painstaking task of manually adding and removing users from each cluster can create more problems than it solves.
2. If, or more likely when, an incident occurs, the time it takes to remediate is critical. If access methods take those who troubleshoot the problem several minutes just to get logged into an affected cluster, problems could multiply.
3. With log data spread across 100 clusters, the ability to have a holistic view of auditing and compliance reporting might be impossible.

Considerations for the Platform Team

One of the many goals of an enterprise’s platform team is to help enable a globally distributed IT team that manages user access across all its clusters from a central location. The intention is to secure and govern access to a Kubernetes infrastructure effectively while making audit logging and compliance reporting much simpler.

A platform team should consider implementing zero trust for Kubernetes to ensure that the best practices described earlier are applied and enforced to secure an entire Kubernetes environment. By eliminating the need to manually apply best practices on every cluster, the IT organization can operate Kubernetes at scale with far less risk.

Here are three benefits for a platform team to consider when designing zero trust for Kubernetes:

1. Make RBAC ultra-flexible: If a team member changes roles, access permissions should be updated automatically so that no single person ever has too much or too little access.
2. Make accessibility fast and streamlined: Eliminate delayed access to any cluster by providing an authorized user seamless access via secure single sign-on.
3. Credentials for just-in-time scenarios: Service accounts for authorized users should be created on remote clusters with “just-in-time” access and removed automatically after the user logs out, thereby eliminating the chance of out-of-date credentials.

As the number of Kubernetes clusters and containerized applications expands, an organization is increasingly exposed to security risks that are not evident when operating just one or two clusters. As a result, platform teams need to enable a central, enterprise-grade level of security and control for both clusters and applications across their entire Kubernetes infrastructure.

This article was originally published in The New Stack.