This post was authored by Stephan Benny, Bheema “Sarat” Kaki and Avinash Varma.
Rafay delivers a SaaS-first Kubernetes management approach that enables companies to deploy and manage modern applications across data centers, public cloud, and edge environments. Our customers leverage the Rafay platform to manage the lifecycle of both Kubernetes clusters and the applications deployed on said clusters.
With security being a key concern for our enterprise customers, Rafay continues to research the most secure workflows for ongoing cluster configuration and application deployment, while ensuring the best developer experience.
The Kube API (or kubeapi) server is at the center of all operations within Kubernetes. Securing access to the kubeapi server is the first line of defense to protect the cluster. Developers and CI/CD systems interact with the kubeapi server using kubectl, the command-line interface to Kubernetes. Kubectl is essentially the client for a cluster’s kubeapi server. kubectl uses a configuration file called kubeconfig to establish user identity and cluster information. The kubeapi server also provides a role-based authorization mechanism to manage permissions around Kubernetes resources.
To manage a Kubernetes cluster remotely you need to connect to the kubeapi server. Some brave administrators have been known to publicly expose kubeapi endpoints for their clusters, which is a huge security risk. The traditional alternatives to simply opening up kubeapi endpoints on the Internet are VPNs, jump hosts, or direct-connect links. But when you are operating multiple clusters across multiple regions in a public cloud, or in a hybrid (cloud + datacenter) model, VPNs, jump hosts, etc., impose a significant operational burden:
- Onboarding or offboarding of developers with different access roles will quickly become quite complex to manage across multiple clusters. Imagine modifying the RBAC policy of a user across 20 clusters.
- There is no easy way to centrally control who can access which clusters; or capture who did what, on which cluster, and when.
About a year ago, we started engaging with customers to better understand this kubectl access problem. The following are some of the key requirements our customers voiced for a more secure kubectl access solution:
- There must be a centralized platform to configure authentication and authorization policies, providing a seamless model to enforce RBAC across all clusters.
- The platform should work for clusters in the cloud or on-premises.
- The platform should integrate with corporate IdP services for single sign-on (SSO).
- The platform must not require ANY changes to developer behavior or tooling. No client-side software/plugins must be needed, and any tool that behaves like kubectl (e.g. relies on kubeconfig) must continue to work as before.
- Per-cluster service account creation for end-users and CI/CD systems should be automated and abstracted away from developers and DevOps teams. Ideally, service accounts should be created just-in-time once the platform has carried out authentication and authorization of the requestor. Once the access activity has been idle for a configuration amount of time, the service account should be automatically deleted.
- All kubeapi server interactions must be authorized and audited across all clusters. Audit logs should be available centrally.
Zero-Trust Kubectl Access
To address customer needs, we developed a feature in our Kubernetes Management Cloud (KMC) called our Zero-trust Kubectl Access (ZTKA) platform. ZTKA has been shipping for some time now and is one of the most heavily used features of the Rafay platform. ZTKA meets all the requirements highlighted above. Leveraging ZTKA works with Rafay-provisioned clusters as well as customer-provisioned clusters that are simply imported into (or attached to) the Rafay platform through a bootstrap YAML file that can be applied to any cluster (kubectl apply -f bootstap.yaml).
The diagram below highlights the ZTKA global platform architecture:
ZTKA leverages outbound MTLS sessions from clusters to Rafay’s kubeapi proxies to securely establish connectivity. Rafay’s kubeapi proxies are deployed in a highly available model across locations to ensure that there are no single points of failure. Kubeapi proxies are designed to scale horizontally to adapt to changing loads. The following diagram is a closer look at the ZKTA workflow:
With ZTKA, customers are no longer required to manage user-accounts/RBAC on a cluster-by-cluster basis. As new clusters are added, RBAC domains automatically extend to new clusters based on group scope: If a user belongs to a group called DEVELOPERS in Microsoft AD, all clusters configured to map to the DEVELOPERS group will be immediately accessible by the user.
Kubeapi proxies seamlessly integrate with your enterprise IdP to authenticate and perform a dynamic, just-in-time (JIT) RBAC enforcement. Administrators can configure authentication and authorization in one place – the Rafay MKP. Users can download their fleet-wide kubeconfig from the Rafay user console. Users can seamlessly start interacting with any clusters they are authorized to access. Although the connectivity is seamless, all activity is transiting Rafay’s kubeapi proxies where each kubeapi request is authorized based on configured policies. For example, you may choose to give all developers ready-only access to production – with Rafay, you can make this happen in about 30 seconds without needing to configure anything on an individual cluster.
Recall that the MTLS sessions are outbound, meaning customers don’t have to open up inbound ports on their firewalls (or in security group configuration in the cloud). This connectivity model significantly reduces the attack surface for each Kubernetes cluster and enhances cluster security posture.
Because all authentication and authorization activity is centralized and audited, customers facing compliance or regulatory requirements also immediately benefit from using ZKTA.
We strive to ensure that the Rafay MKP continues to add value to everyone. Whether you purely use ArgoCD and git-triggers to automate deployment to clusters, have standardized on helm-client for application deployments, or continue to use raw kubectl to interact with your clusters, Rafay will make your ongoing operations – and your day job – easier.
Stay tuned for more posts regarding Kubernetes management security best practices and the ZTKA. If you want to take ZTKA for a spin, please get in touch.