The Hidden Costs of Building and Managing Kubernetes Clusters with Rancher Open Source

Enterprise IT departments are always on the quest for the best return on investment (ROI) for any technology investment. Understanding the ROI with an open source technology provides its own unique challenges because one important factor in determining the ROI is TCO, or total cost of ownership. After all, the open source software (OSS) employed is free, but that doesn’t tell the whole TCO story. A case in point is Kubernetes technology. The TCO of Kubernetes can be quite hard to quantify given its technical complexity, resource requirements, and the number of available deployment options. And, depending on the approach, enterprises can be surprised with the increasing hidden costs of operating Kubernetes.

This blog sheds light on several hidden costs of running Rancher for Kubernetes management, describes the potential business impact, and why enterprises are choosing to switch to Rafay from Rancher OSS.

What are the hidden costs when creating a K8s environment with Rancher Open Source?

The idiom ‘tip of the iceberg’ means that a much larger situation or problem of a noticeable issue remains mostly hidden. The following are several cost categories that begin hidden under the water but eventually arise to be very apparent when building and supporting a K8s environment with Rancher OSS.

K8s Controller Infrastructure & Support

As mentioned above, Rancher’s K8s distribution requires that each small set of clusters needing its own access control or isolation boundary has its own Controller. For production environments, it is necessary to deploy Controllers with high availability (HA), including the necessary networking infrastructure and load balancers. As a result, the overhead associated with managing these controllers as well as the hardware portion of the TCO quickly increases.

Assuming a typical case of a company starting with 5 Controllers, i.e., one development, one QA and three production clusters across multiple cloud regions, costs can easily be US$60k.

Also note that all of this hardware infrastructure that is required also needs to be hosted, supported, maintained, and upgraded over time by IT personnel.

Governance & Security Risks

Securing network and access to DIY K8s infrastructure is complex, evolving, and most likely one of the highest new-age risks that IT departments have to manage. While building K8s clusters with Rancher is relatively straightforward, it can create security risks to a K8s environment with hidden costs that are difficult to identify initially, such as:

1. Rancher OSS’s lack of native tools for drift detection and consistency enforcement allows for clusters to be built with configurations that can differ and inadvertently generate issues with authentication, outdated K8s versions, exposed Kube API ports to insecure networks, inconsistent RBAC profiles, and audit logging nightmares. These potential Kubernetes security risks can severely compromise applications and IT budgets due to recurring debugging and rework or financial loss due to unscheduled downtimes.
2. Managing Rancher K8s clusters is usually done using kubectl. By default, kubectl doesn’t provide logging of executed commands by user account. As a result, it can be difficult or impossible to understand who made what changes when unless custom coding is done because audits need to be collected from each cluster and then collated centrally for actionable information.

Further, kubectl can be cumbersome to access outside firewalls, and managing more than a handful of clusters becomes complex and error-prone with the use of VPNs, jumphosts, etc.

Time to Value – The Biggest Hidden Cost to Enterprises

Enterprises that have switched to Rafay from Rancher have stated that building a Rancher K8s infrastructure from scratch with approximately 3 engineers can take anywhere between 9 to 12 months just to launch. This is perhaps the hidden cost that has the largest negative impact on an enterprise’s business.

This means the enterprise cannot take advantage of the efficiencies K8s has to offer for about a year. And this directly translates into real negative business impact in the form of slower deployments, higher MTTR, and less IT agility and scalability until a fully functional K8s environment can be leveraged.

K8s Expert Engineers

Conceptually, Kubernetes allows for easier, faster, and disposable “chunks” of infrastructure that can be enabled to internal and external clients cost-effectively. However, to benefit from these fast and flexible “chunks” of infrastructure, companies need to implement new processes to build, integrate, access, maintain, and upgrade K8 clusters. This requires hiring K8s experts that are hard to find and keep because demand is so high and supply is low.

Rancher OSS is an example of a K8s distribution that requires such a team of expert K8s engineers. The Rancher K8s technology is not designed to manage fleets of clusters in a centralized manner and requires a dedicated controller for each small set of clusters that need some type of segmented access control or isolation boundary. The result is a greater level of complexity for K8s teams that have to build clusters, each small set of clusters with its own controller, create access policies, develop backup and restore strategies, implement security requirements, deployment models, etc. From experience working with enterprises that have initially used Rancher, even small Kubernetes installations (e.g., of less than 10 clusters across a dev, QA and production environment) typically require at least 3 full-time expert engineers familiar with Rancher. At a fully loaded cost of $200k/year (fully loaded meaning including healthcare, insurance, training, etc.), that’s $600k per year just to build and maintain a very modest installation.

K8s Support Engineers

Along with the challenges of having a highly experienced K8s team to build and deliver clusters, enterprises also need to have a team of K8s support engineers dedicated to troubleshooting and maintaining Kubernetes environments. In addition, given that multiple Controllers are typically needed each requiring separate monitoring, upgrades, and maintenance, support teams have to be able to manage several groups of clusters independently.

Using the example above of a small Kubernetes installation (e.g., of less than 10 clusters) requires at least 2 full-time support engineers familiar with Rancher. At a fully loaded cost of $150k/year, that’s another $300k per year just for support on an ongoing basis.

How to Eliminate the Hidden Costs of Rancher OSS with Rafay

In contrast to the above costs, Rafay’s SaaS capabilities are designed to reduce the complexities of the implementation and the ongoing challenges of maintaining Kubernetes that impact OPEX/CAPEX and time-to-value.

One SaaS Controller is All Enterprises Need

One major reason why companies are switching to Rafay from Rancher is that they do not need to worry about Controller hardware and costs, scalability, and support, as these are included in Rafay’s SaaS offering regardless of the number and location of K8s clusters in their fleet. Another advantage of Rafay’s SaaS Controller is that only one is needed to deploy and manage any number of clusters with boundary isolation through our multi-tenancy support.

Zero-Trust Security Built-in

To eliminate security risks, Rafay’s Zero-Trust Access Service solves several of the above issues by centralizing and securing access to kubectl from anywhere and governing the use of kubectl by user account. It also enables seamless access via single sign-on (SSO) and role-based access control (RBAC) to kubectl, providing a higher level of security. Furthermore, by acting as a proxy to the Kube API server, Zero-Trust Access Service enables proper network isolation that is considered a fundamental Kubernetes security best practice.

K8s Infrastructure in Weeks Not Months

Finally, with Rafay’s SaaS approach, enterprises obtain all the benefits of SaaS including a fast time-to-value. WIth Rafay, enterprises begin utilizing fully functional K8s infrastructure within the average of 2 to 4 weeks compared to the average 9+ months of a DIY Rancher K8s. This is possible because Rafay’s technology replaces several of the manual steps required to build, integrate, and deploy K8s clusters with a solution that works out of the box – including pre-built, one-click integrations with popular K8s tools such as Hashicorp Vault.

Better Utilization of Expert Resources

Enterprises can also focus their expert resources on higher-value work by leveraging Rafay’s centralized and straightforward interface with features like Cluster Blueprints, Drift Detection, automated K8s upgrades, Workload Templates, and native GitOps. These features can reduce and/or remove the need to develop complex DIY YAML manifests and the ungoverned usage of kubectl for the deployment and maintenance of K8s clusters.

With Rafay’s Cluster Blueprints and templates, clusters and add-on configurations can easily be created, managed, and centrally applied to new and existing clusters. Cluster Blueprints can also be shared across multiple teams for centralized governance of add-ons deployed across the K8s clusters or fleets. This ensures that all the clusters have the same security policies and configurations. If any cluster changes, admins can be notified to optionally allow or deny the changes with Drift Detection that is included in the service. Automated Kubernetes upgrades can be performed on clusters and all supporting components, on-prem, in the cloud, or both with a single click. Preflight checks are conducted to ensure that applications are not impacted during upgrades.

Workload Templates and GitOps ensure a centralized, consistent, secure, and automated process of application lifecycle management that also follows the same straightforward approach of reduction of complexity to deploy and maintain services across Kubernetes clusters.

In conclusion, all Rafay’s SaaS capabilities described above optimize costs and investments by providing features that 1) remove and automate Kubernetes operations that a DIY approach requires custom code to build, implement, and maintain, and 2) reduces infrastructure footprint with a SaaS approach for the K8s Controller and multi-tenancy support that is difficult and costly to achieve with Rancher OSS.

If you are ready to leverage the real power of Kubernetes without the hidden costs of Rancher OSS, sign up for a free trial of Rafay today.