Upgrading Amazon EKS Clusters in 2023

Kubernetes is a rapidly evolving open-source project with periodic releases. And organizations embracing Kubernetes must adopt the practice of regular upgrades. This is because Kubernetes doesn’t follow the Long Term Support (LTS) concept. Instead, they have created a versioning and releasing process that follows a quarterly release cycle, with a cadence ranging between 70 and 130 days.

The Kubernetes upgrade process is not just limited to the deployed components. You also need to keep a check on Kubernetes API versioning.

The Kubernetes community has an API versioning scheme with a strict deprecation policy that recommends a deprecated GA APIs support period of 12 months or three releases (whichever is longer).

Therefore, it is necessary to update the cluster and the deployed solutions to leverage the latest security, new features, and bug fixes. Unfortunately, upgrading Kubernetes is difficult as it consists of a collection of components. These components provide various features ranging from application deployments and upgrades to logging and monitoring to persistent data storage.

Kubernetes upgrades are significantly simplified with Amazon EKS. Running your Amazon EKS cluster with the latest Kubernetes version is essential to ensure optimal cluster performance and functionality. Amazon EKS follows the Kubernetes release philosophy. Amazon EKS provides three production-ready versions of Kubernetes at any given time. Support for the new Kubernetes GA version is provided sometime after its release. Amazon EKS also recommends updating your cluster to the latest Kubernetes version when it becomes available.

How to Update Amazon EKS Clusters Using the AWS Management Console:

The Amazon EKS upgrade process involves launching new API server nodes with the updated Kubernetes version to replace the existing ones. The Kubernetes upgrade process is an incremental approach that does not allow you to jump to the latest version directly.

So, you have to adopt all intermediate versions until you reach your targeted version. To prepare for an effective Kubernetes upgrade, you must determine all the components, along with the control plane, which will require an update. As a best practice, make sure you test the behavior of your apps against new versions of Kubernetes before you update your production clusters. This article covers the steps involved in incrementally upgrading EKS components.

Prerequisites

As a first step, validate the version of your Kubernetes cluster and worker nodes. The Kubernetes minor version of worker nodes in your cluster must be the same as the version of your control plane’s current version.

Therefore, if your worker nodes run an older version than your current Kubernetes Control Plane version, you must update your worker nodes before upgrading the control plane.

You can get the Kubernetes Control Plane using the following command:

$ kubectl version --short Client Version: v1.19.7 Server Version: v1.18.16-eks-7737de

Next, determine the worker nodes version using the following command:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-125-37.us-east-2.compute.internal Ready 87m v1.18.9-eks-d1db3c
ip-192-168-150-29.us-east-2.compute.internal Ready 87m v1.18.9-eks-d1db3c

Amazon EKS clusters come with a pod security policy admission controller enabled by default. Before updating, ensure that all pod security policies are in place. Validate the default policy with the following command:

$ kubectl get PSP eks.privileged
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
eks.privileged true *RunAsAny RunAsAny RunAsAny RunAsAny false*

How to Update the Amazon EKS Control Plane

1. Amazon EKS allows administrators to upgrade the control plane from the AWS management console. The AWS console shows an information banner with the Update Now button. Click it to start the upgrade process.

It takes time to update the cluster control plane. During this time, the cluster update-history console shows an in-progress task regarding cluster upgrade.

Conduct this activity during non-peak or off-peak hours as you are not permitted to make any changes to your cluster or schedule nodes while the upgrade is in progress. Alternatively, you can also follow the following CLI commands to perform an Amazon EKS upgrade:
Initiate the upgrade task using update-cluster-version command

$ docker run --rm -it -v ~/.aws:/root/.aws -v ~/.kube:/root/.kube amazon/aws-cli eks --region us-east-2 update-cluster-version --name dev --kubernetes-version 1.19 update: createdAt: '2021-07-19T11:15:20.980000+00:00' errors: [] id: 24938002-69b4-4fbf-a32f-0c1040233144 params: - type: Version value: '1.19' - type: PlatformVersion value: eks.5 status: InProgress type: VersionUpdate

2. The update task returns back a task Id. You can check the task details and status using the provided task Id

$ docker run --rm -it -v ~/.aws:/root/.aws -v ~/.kube:/root/.kube amazon/aws-cli eks --region us-east-2 describe-update --name dev --update-id 24938002-69b4-4fbf-a32f-0c1040233144 update: createdAt: '2021-07-19T11:15:20.980000+00:00' errors: [] id: 24938002-69b4-4fbf-a32f-0c1040233144 params: - type: Version value: '1.19' - type: PlatformVersion value: eks.5 status: InProgress type: VersionUpdate

Once the cluster update is complete, update your worker nodes to the same Kubernetes minor version.

How to Upgrade Amazon EKS Nodes

Scale-down Kubernetes AutoScaler

Kubernetes has a component called “Autoscaler,” which makes sure that every pod has a place to run and no nodes are left idle. Thus it is imperative to scale down the AutoScaler application to prevent AutoScaling operations from interfering with the worker nodes upgrade. You can determine the autoscaler using the following Kubectl command :

$ kubectl get deployment cluster-autoscaler -n kube-system

Update Nodes

The Amazon EKS worker nodes should provide an upgrade message if it determines a possibility of an auto-upgrade. The auto-upgrade feature is not available in the case of custom AMIs. Click the update-now button to perform the upgrade.

Kubernetes nodes are responsible for executing workloads. Thus Amazon EKS provides a “RollingUpdate” strategy to perform a non-disruptive upgrade. The rolling upgrade will deploy a new version of a component and then scale down the old version. The process will continue until all of the old version components have been replaced with new version components. Alternatively, users can also select force update, which will stop the worker nodes and disrupt executing processes. Specify either of the above strategies for performing upgrades.

It takes time to update the cluster worker nodes. During this time, the update-history worker node console shows an in-progress task regarding cluster upgrade.

Update EKS Addons nodes

EKS also provides managed add-ons for Kube-proxy, CNI VPC, and CoreDNS. The following table lists EKS recommended add-on versions for each supported Kubernetes cluster version.

Kubernetes version	1.20	1.19	1.18	1.17
kube-proxy	1.20.4-eksbuild.2	1.19.6-eksbuild.2	1.18.8-eksbuild.1	1.17.9-eksbuild.1
CoreDNS	1.8.3	1.8.0	1.7.0	1.6.6
VPC CNI	1.8.x	1.8.x	1.8.x	1.8.x

All addons are listed on the EKS cluster addons console. Whenever a new addon version is available, the respective addon will show an update now button.

Upgrade to the newer version by specifying the version of an addon you want to upgrade. Amazon EKS also provides an override of the existing configuration checkbox which mitigates issues by discarding older add-on configuration in favor of the new add-on configuration.

Update Kubeproxy

Specify the version of the addon to run.

Update the Cluster DNS Provider

Upgrade the cluster DNS provider to the latest supported version. Specify the version of the addon to run.

Update VPC CNI

An important feature of Amazon EKS add-ons is support for AWS IAM roles for service accounts. Add-ons that require IAM permissions (like VPC CNI plugin) can inherit permissions from the Amazon EC2 nodes in which they are installed. Check and upgrade the version of the cluster’s Amazon VPC CNI Plugin for Kubernetes.

Rafay’s Automated Amazon EKS Upgrade Process

At this point, you have worked your way through the Amazon EKS upgrade, realizing the different steps one needs to take. Typically organizations have many EKS clusters across different environments and different teams. Consequently, the manual steps are repeated many times, which can be a time-consuming process. Teams adopting Amazon EKS need an upgrade process that is consistent, repeatable, and automated.

Rafay provides an automated, standardized, repeatable process for Amazon EKS upgrades. The console allows you to select components for the upgrade, Amazon EKS control planes, worker node groups, cluster add-ons, and then take care of the complete process. It also performs pre and post-upgrade checks to ensure cluster correctness and stability.

Rafay’s dashboard shows Upgrade Available message for clusters that have an upgrade available.

The cog icon for cluster actions also provides an “upgrade cluster” action item. Click either the action menu or the info message to start a Kubernetes Version upgrade.

You can select the Control Place + Node groups option to start the upgrade process. Rafay also keeps a detailed record of all applied upgrades for auditing and governance needs. After an upgrade job is submitted, Rafay performs pre-checks, then the upgrade, and then a number of post-upgrade validation checks.

Streamlining the Amazon EKS Upgrade Process with Rafay

Kubernetes adopters need an effective upgrade strategy to keep updated with the latest fixes and take advantage of new features. The article presents the steps to upgrade Amazon EKS clusters using the AWS Management console. Amazon EKS addresses many operations challenges for cluster upgrades and allows you to focus on workloads and business value instead of control plane high availability or data plane compatibility.

That said, upgrading each Amazon cluster can take time and includes the manual steps above. To help streamline and automate the process, Rafay has built a number of automation tools that upgrade Amazon EKS clusters super easily. You can use the Rafay Kubernetes Operations Platform for a repeatable, consistent, and automated EKS upgrade process, thereby improving teams’ productivity and leveraging the latest Kubernetes features.

Kubernetes Best Practices Checklist

Get started with K8s using this checklist of best practices

View Now!