Four Pillars of Kubernetes Fleet Management

With the success of Kubernetes (K8s), many teams have gone from operating just a handful of clusters to operating “fleets” of K8s clusters—both on-premises and in the cloud. As if the growing pains that come with scaling clusters rapidly aren’t bad enough, having a growing number of applications on different clouds, running different K8s distributions and many add-ons has resulted in significant management challenges.

It can be difficult to automate fleet-wide operations or to get a unified, enterprise-wide view of cluster operations and health information. You may find it difficult to maintain configuration consistency at both the cluster and application level across clusters, while also accommodating unique requirements imposed by various internal teams.

This blog explores the concepts behind Kubernetes fleet management and introduces four pillars that are essential for fleet management success.

What is a Kubernetes Fleet?

A simple definition is that a Kubernetes fleet is a logical set of clusters to be managed as a single domain. Your fleet could consist of all the K8s clusters and applications across your company, the set of clusters and apps that you and your team are responsible for, or a set of clusters that all have similar management needs.

Depending on how your company is organized and how it operates, a fleet might include clusters used for development, test, and production. It might also include clusters running on-premises, in one or more public clouds, and in edge locations like retail stores, warehouses, distribution centers, or regional offices.

The truth is, it’s up to you to define what your fleet looks like. Chances are you already have a pretty good idea, but you may be struggling to operationalize your K8s fleet(s) in a way that allows you to monitor and manage them easily.

What is Kubernetes Fleet Management?

We can define fleet management as a process for managing, monitoring, and governing a heterogeneous fleet of K8s clusters and associated apps. In other words, fleet management is how you transition from managing each cluster individually to managing and governing global cluster functions such as security, configuration, and monitoring collectively for a large set of Kubernetes clusters in a centralized manner.

Pillars of Fleet Management

As you plan your fleet management strategy, there are four broad pillars of Kubernetes fleet management that you should be thinking about. Depending on your industry and organizational needs, some of these may be a higher or lower priority, but every organization likely has some requirements in all four areas. These pillars are the foundation of fleet management: Automation, Security, Visibility, Governance. The following sections examine each of these in turn.

Pillar 1: Automation

Managing Kubernetes with kubectl commands and a few scripts when you only have a few clusters might not be too difficult, but this simply doesn’t scale. By automating and standardizing common cluster and application operations, you can manage more clusters with less effort while avoiding misconfigurations due to human errors.

Here are some common Kubernetes and application tasks that you may want to automate:

K8s Fleet Automation Opportunities
Kubernetes-Level	App-Level
New cluster provisioning: Eliminate manual deployments.	CI/CD pipeline integration: Integrate your existing CI/CD pipeline with your fleet.
Cluster templates: Ensure every deployment is standardized and compliant.	App deployments: Deploy and operate apps across multiple clusters. Control app deployments based on policy.
Kubernetes version upgrades: Take the pain out of all-too-frequent version changes.	Ecosystem: Integrate with application-level tools like HashiCorp Vault.
GitOps: Leverage Git as a single source of truth for clusters and apps.
Templates: Ensure clusters and apps are standardized.

Pillar 2: Security

Mission-critical clusters and applications running in production require the highest level of security and control. As your fleet grows, your enterprise is exposed to security risks that aren’t evident when you were only operating a few clusters.

Applying zero-trust principles is the best way to secure your K8s environment. Kubernetes includes all the hooks necessary for zero-trust. Unfortunately, keeping all the individual elements correctly configured and aligned across dozens of clusters is a challenge, especially when multiple workloads and Kubernetes distributions are involved.

You need to be able to integrate with your existing SSO, manage authorization with role-based access control (RBAC) across your fleet, and provide an end-to-end audit trail.

K8s Fleet Security Opportunities
Kubernetes-Level	App-Level
Simple access mgmt, traceability and authorization: Flexible role-based access control, fast and streamlined access, just-in-time credentials, centralized audit logs	Application access: Secrets Management Integration, centralized policy mgmt, RBAC-enabled app access
Zero-trust: centralized control over user access, roles and configurations across clusters and applications

See our recent blogs on Securing Kubernetes: Applying Zero-Trust Principles to Your Kubernetes Environment and Secure Operations for Kubernetes Clusters and Applications for more on this topic.

Pillar 3: Visibility

Your team can’t manage and support what they can’t see. You need a single, fleet-wide view of Kubernetes clusters and applications that includes resources consumed, user and access activity, critical alerts, and overall health.

While there are a number of open source and commercial Kubernetes monitoring tools that can make monitoring easier, implementing these tools in a large fleet creates significant complexity. Many organizations find that monitoring as a service is a better alternative as their fleet grows. The right fleet monitoring tool should give you all the metrics you need in one place, while integrating with whatever monitoring you already use.

K8s Fleet Visibility Opportunities
Kubernetes-Level	App-Level
Multi-Cluster and multi-cloud: ability to monitor any environment across multiple locations, i.e., cloud, multi-cloud, on-prem, hybrid and edge	Application monitoring: view how applications are performing on a node, cluster, multi-clusters or fleet
Alerts: reduce mean time to recovery (MTTR) by up to 60%
Integration: Amazon Prometheus, CloudWatch, Datadog, Grafana, New Relic, Splunk, and the Prometheus Operator (for custom Prometheus)
Single Pane Of Glass (SPOG): reduce mean time to recovery (MTTR) by up to 60%

See the recent blog, Best Practices, Tools, and Approaches for Kubernetes Monitoring for more on fleetwide visibility and monitoring.

Pillar 4: Governance

As the complexity of your K8s fleet grows, it becomes increasingly difficult to ensure you’re complying with security policies and industry regulations. Especially in regulated industries like Financial Services and Healthcare, organizations need fleet management tools that can facilitate governance and enforce policy-based management.

Here are some common Kubernetes and application governance capabilities to consider:

K8s Fleet Governance Opportunities
Kubernetes-Level	App-Level
Policy-based management: Enforce compliance with specified policies.	App catalog: Maintain a list of authorized apps and application-level constraints
Audit trails: Maintain audit logs of all user activity across your entire fleet.	Resource management: Manage and enforce resource quotas across the fleet.
Templates: Ensure clusters and apps are standardized.
Drift detection: Get alerted if clusters and apps deviate from the specified config.
Disaster Recovery: Automate protection of cluster and app data.

Practical Considerations for K8s Fleet Management

While the four pillars just described are the bedrock of fleet management, as you plan your Kubernetes fleet management strategy, there are several more mundane but important considerations as well. Most organizations require some or all of the following:

• Centralized management: Monitor and manage your fleet from a central console using the same core tooling everywhere. While deploying several best-of-breed tools isn’t out of the question, each additional tool adds complexity and increases the learning curve.
• Flexibility to run on any infrastructure: Fleet management should adapt to your requirements instead of forcing you to adapt your operations.
• Integration with a broad range of tooling: Any solution(s) must integrate with the K8s distributions and tools you already rely on.
• Easy to consume: Some tools themselves require a lot of management. If you have to spend a lot of time installing, configuring, and updating tools, agents, etc. that’s time you won’t get back. Many organizations prefer a Software-as-a-Service (SaaS) model that reduces overhead and level of expertise required.

Kubernetes Fleet Management at Rafay

Using Rafay for fleet management, you can control any Kubernetes distribution and immediately gain centralized  automation, security, visibility, and kubernetes governance capabilities for Kubernetes and application lifecycle management—across public clouds such as AWS, Azure, and GCP as well as datacenter and remote/edge locations.

Rafay’s Kubernetes Operations Platform is the ideal solution for Kubernetes fleet management, delivering all the capabilities mentioned in this blog and more. Additional capabilities include:

• Configuration overrides: It can be critical to be able to dynamically customize resources with cluster-specific requirements. Rafay provides overrides for cluster blueprints and workloads so the same artifact can be deployed with customizable configurations on a cluster-by-cluster basis.
• 1-Click upgrades: Seamless, in-place workflows with fine-grained controls enable you to upgrade control planes, worker node groups, critical cluster add-ons, and worker node AMIs in a controlled, phased manner.
• Fleet-wide dashboards: Rafay dashboards ensure reliable operations with a single, organization-wide view. Internal teams are also provided dashboards to visualize sub-fleets tied to Rafay’s native RBAC support.

With Rafay’s SaaS approach, you gain the benefits of complete fleet management with all the convenience of the cloud, including:

• Ease of use
• Get started in minutes
• No administrative clusters to manage
• Automatic scaling to handle hundreds of K8s clusters

Rafay services are designed to work with the tools you already rely on, including out-of-the-box integrations with popular single sign on (SSO) solutions, CNIs, CI/CD tools (including Jenkins), open source and commercial monitoring solutions, secrets management tools, repositories, and many more. APIs ensure integration with homegrown solutions as well as new or obscure tooling.

Rafay delivers the fleet management capabilities you need to ensure the success of your multi-cloud, multi-cluster Kubernetes environment. To discover how Rafay can help you standardize management across your entire fleet of K8s clusters and applications, take a closer look at Rafay’s Kubernetes Operations Platform.

Ready to find out why so many enterprises and platform teams have partnered with Rafay for Kubernetes fleet management? Sign up for a free trial.