Choosing Between Amazon ECS and EKS

We frequently get asked by users that are currently on AWS whether they should be using Amazon ECS or EKS to deploy and operate their containerized applications. Since this is such a common question and the answers are somewhat nuanced, we wanted to share our thoughts and recommendations for the benefit of all users.  Here is a table summarizing all the important criteria you need to factor in as you evaluate between Amazon ECS and EKS.

Amazon ECS	Amazon EKS
You have a simple app with not many micro services	Your app is a large, complex and distributed with many micro services
Your app has minimal distributed dependencies	Your app has complex inter-app dependencies. You may need to operate on-demand, parallel workloads
App team’s part time job is Infra/Ops on AWS	You have an existing IT/Ops team that can support you
You have selected AWS as your ONLY infra provider for the long term	You need to design for consistency and portability to support multi-cloud or hybrid
Budget is not a major concern	Cost controls are critical

Why Amazon ECS?

If you are new to containers and cloud and if the app team is being forced to make infrastructure decisions for their containerized application, they will find it extremely easy to get started with Amazon Elastic Container Service (ECS). They will typically experience the following benefits right away:

• Do not have to deal with a moving target with Kubernetes versions
• Zero cost for the managed ECS control plane
• A pay-as-you-go model for the Fargate based data plane with full support for Spot etc

We always caution customers to consider and evaluate carefully avoid going down a one-way door. This is particularly critical for mid-to-large size organizations to consider carefully so that they can avoid painful and expensive migrations to Amazon EKS down the line. Note that ECS offers two launch types: Fargate and EC2. Fargate is the serverless option and users do not have to provision or manage the underlying servers. If you cannot use ECS with Fargate, you will not see most of the benefits of ECS.

Why Amazon EKS?

To help understand this better, let’s also look at this from the lens of Kubernetes esp. Amazon EKS and why it became so popular.

Low Control Plane Cost

The monthly cost for the SLA backed, managed control plane for Amazon EKS is only ~$70 per month (rounding error for most budgets). Amazon EKS integrates natively with AWS services esp. such as Spot, IAM and Fargate etc. With the use of market leading add-ons such as Karpenter, operational effort associated with EKS cluster sizing and scaling can be completely automated as well.

Forced Update Cadence

A forced update cadence due to enhancements via new upstream Kubernetes releases is actually a good thing! There is enormous momentum behind Kubernetes that organizations are benefiting from. By standardizing on Kubernetes, organizations ensure they do not run the risk of stagnation. Note that the cost/burden of periodic updates can be extremely low if organizations leverage platforms with fleet, API deprecation checks and Fleet-wide updates of add-on type capabilities to keep their clusters standardized, consistent and on the latest versions. This should be a critical requirement in your evaluation criteria for Kubernetes Management.

Access to IT/Ops Talent

Organizations know that personnel costs can get prohibitive and not having the qualified personnel can significantly impact their corporate initiatives. With CNCF backing Kubernetes with a comprehensive certification program, there is an extremely large and fast growing pool of talent that organizations are able to tap into for Operations and DevOps talent.

Note that it is significantly easier and more cost effective to find Kubernetes expertise in the market relative to Amazon ECS.

Standardized API

One of the biggest misconceptions in the market is that the value of Kubernetes is about the distribution and the managed/hosted control plane. This is grossly incorrect! The biggest benefit of Kubernetes is that your applications get to use a standardized, portable, version controlled API (i.e. the Kubernetes API). Because of this, we have seen our customers deploy the exact same application on multiple clouds with zero changes. It would be terrible for an organization to squander this superpower. Just think of the leverage this provides you in your engagement with your provider and how it empowers you to support your business requirements. An application using the Kubernetes API can be deployed on ANY Kubernetes distribution on ANY infrastructure provider as long as the distribution itself is compliant with upstream Kubernetes.

No Vendor Lock-in

When we refer to “vendor lock-in”, we are not just referring to the infrastructure or the managed service. We are also referring to the tooling supporting the infrastructure and applications. In fact, this is typically the long pole in the tent for most organizations. Being a closed, proprietary offering, Amazon ECS does not have the kind of support for deploying and configuring plugins to manage security, monitoring, and more importantly automation. Since EKS is essentially standard upstream Kubernetes, there is an endless range of 3rd party tooling available for this eco-system. For example, instead of:

• Being limited to AWS CodePipeline, with EKS, you can use ArgoCD or FluxCD or Rafay or a long list of CD tools to deploy and operate your applications
• Being limited to AWS IAM, you have Kubernetes RBAC that can be seamlessly integrated with your corporate Identity Provider (IdP) such as Okta, Microsoft EntraID, Ping etc.
• Being limited to AWS based autoscaling configs, you can use Kubernetes HPA/VPA/KEDA etc.
• Being limited to AWS CloudWatch, you have Prometheus, Grafana and a plethora or commercial offerings
• Being limited to AWS Step Functions, you have a number of 3rd party workflow tools

Identical Developer Environments

With Amazon EKS, you can have almost identical setups so what you test locally is the same and you can use the same network plugins, security pod restrictions, etc. Since you cannot run ECS locally, you have to resort to a “dual stack” approach i.e. running production apps in a black box while leaving your developers to fend for themselves with things like docker compose etc. What you actively develop and test on can mirror the production environment.

Best of Breed Patterns

With Amazon EKS, you can leverage and use battle tested patterns such as GitOps from a range of providers like ArgoCD, FluxCD and Rafay. This reduces business risk significantly.

Power of Community

There is an enormous and active community for Kubernetes. For example, it is trivial to find a solution for a problem packaged as Helm charts vs hoping someone has written/published an ECS task for your problem.

Customization

Doing anything custom in Amazon ECS can be extremely painful or impossible. With Kubernetes, you have access to well documented and established patterns such as CRDs and Operator patterns. As always, our recommendation would be to pursue the customization route only if the use case warrants it. In a nutshell, Kubernetes is the unchallenged king of customization.

Segregation of Access

With ECS, task separation vs host access grants “too much” or “too little” wrt access. Trying to manage that granularity through AWS IAM is extremely brittle or downright impossible in most cases. With EKS (Kubernetes), organizations can segregate access by service, deployment, and enable developers to do more without risk of breaking things or accessing things they should not. They can combine it with things like AWS Secrets Manager or HashiCorp Vault for secrets management. In a nutshell, you have elegant and simple ways of enabling secure secrets in your services without making it available for developers who can touch the ECS host itself.

Summary

If you are a mid-to-large organization, even if your application is the first containerized application, you need to skate to where the puck is going. The IT/Ops team will eventually have to support other containerized applications and the business will likely be required to support other infrastructure providers as well. As a result, it may make sense for organizations currently on AWS to stick with the logical, practical and sensible option i.e. Amazon EKS.

Whether you are an organization just starting out with your first containerized application or have a mature practice already, you can leverage the Rafay Platform to help streamline and optimize your operations. Learn how Rafay’s customers have benefited significantly.