Amazon Managed Service for Prometheus (AMP) is a new managed service by Amazon launching in general availability and providing an easy, scalable equivalent to the CNCF open-source Prometheus project. With AMP, enterprises get all the container monitoring benefits of open-source Prometheus without the hardware, storage, or software reliability and maintenance issues. Additionally, Prometheus can easily collect and send application metrics from multiple AWS and third-party services running inside of clusters without having to manually instrument the application to Amazon Managed Service for Prometheus. AMP auto-scales ingestion, storage, alerting, and querying of metrics based on your monitoring needs and the elastic nature of metrics generated from your dynamic workloads. AMP is perfect for AWS customers looking to focus on developing, deploying, and monitoring business-critical applications without having to use critical resources on deploying and managing a monitoring tool across cloud and on-premises container environments.

Rafay is excited to be a launch partner with AMP enabling AWS customers looking to standardize on AMP. Organizations can use Rafay’s add-on and cluster blueprinting capability ensuring AMP is deployed for all or a specific set of clusters. Rafay’s cluster blueprints give control to DevOps teams enabling them to automate the installation, management, and enforcement of standardized tools or software add-ons across their Kubernetes clusters providing governance and compliance across any or all Kubernetes environments.

How to Install and Configure Amazon Managed Service for Prometheus with Rafay

For this blog post we will walk you through configuring the Prometheus service with Rafay to monitor the performance of an EKS cluster.  We will assume you have an EKS cluster up and running and have a visualization tool to view the AMP data.  If you’d like to see how to set up an EKS cluster and/or Grafana, an open-source analytics and visualization tool, you can see the full Rafay AMP recipe here.

Configuring and Deploying AMP with Rafay:

Step 1: AMP Workspace

• Login into the AWS Console.
• Select the Amazon Prometheus service and create an AMP workspace.

In the example below, we have created our AMP workspace called “amp-demo” in the “us-east-1” AWS region.

Step 2: IRSA

In this step, you will create an IRSA so that the Prometheus pods on the EKS cluster will have the necessary permissions to perform “remote writes” to the AMP remote write endpoint. In AWS, it is a recommended best practice to use AWS Identity and Access Management (IAM) roles for service accounts (IRSA) to access AWS services outside the EKS cluster because of the following benefits:

Create Namespace

We will deploy Prometheus to a namespace called “monitoring”:

• Click on Infrastructure -> Namespace.
• Create a new namespace with the name “monitoring”.

Create IRSA

In the example below, the EKS cluster’s name is “amp-demo” and the IRSA name is “amp-irsa”.

./rctl create iam-service-account amp-demo --name amp-irsa --namespace monitoring --policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess

Request submitted successfully to create IAM service account(s).

Verify IRSA

Creation of the IRSA can take a few seconds. You can verify the status of the IRSA by using RCTL.

./rctl get iam-service-account amp-demo

[{"metadata":{"name":"amp-irsa","namespace":"monitoring"},"attachPolicyARNs":["arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess"]}]

You can also verify that the k8s service account was created in the EKS cluster in the “monitoring” namespace.

• Click on the Zero Trust Kubectl Shell on the web console.

kubectl get sa -n monitoring


NAME       SECRETS   AGE

amp-irsa   1         2m33s

default    1         18m

As you can see in this example, the “amp-irsa” service account was successfully created in the “monitoring” namespace.

Step 3: Cluster Blueprint

In this step, you will

• Add the Prometheus Repo to Rafay.
• Create a Prometheus add-on inside of Rafay.
• Provide the Prometheus add-on with custom values so that it has permissions to perform remote writes to the newly created AMP workspace in the previous step.
• Create a new custom cluster blueprint with the Prometheus add-on created previously.
• Apply the new blueprint across your clusters and verify setup.

Add the Prometheus Repo

Configure the Prometheus repo endpoint so that the controller can automatically retrieve the required Helm chart directly from the Internet facing repository.

• Click on Integrations -> Repository.
• Create New Repository -> Provide a name such as “prometheus” and Select “Helm” for Type.
• Enter “https://prometheus-community.github.io/helm-charts” for Endpoint and Select “Internet Facing” for Reachability.

• Optionally, you can validate the correct configuration of the repository by clicking on the validate option.

 

Create Rafay Add-on

• Click on Infrastructure -> Add-ons.
• Click on Create New Add-on with the name “prometheus”.
• Select “Helm3” for add-on type.
• Select “Pull files from repository” for Artifact Sync.
• Select the repository type as “helm”.
• Select the “monitoring” namespace from the dropdown.

Custom Values

We need to customize the Prometheus add-on with an override “values file” comprising the following:

1. The IRSA for Prometheus to perform remote writes to the AMP endpoint.
2. The AMP remote write endpoint URL details (from AWS Console).
3. The AWS region for the AMP workspace.
4. Save the YAML provided below to a file and update the fields appropriately.
5. Add the values.yaml file to the add-on as an override.

 

New Add-on Version

• Click on “New Version” to create a new version of the prometheus add-on
• Provide a version (e.g. v1)
• Select the “prometheus” repository
• Enter “prometheus” for the Chart Name
• Enter “14.1.0” for the version number
• Upload the custom values file from the previous step

New Blueprint

• Select blueprints and create a new blueprint (e.g. amp)
• Click on “New version” (e.g. v1)
• Select the “prometheus” add-on and “version” from the list of custom add-ons.
• Save blueprint

Step 4: Apply Blueprint

Now, we are ready to apply the newly created, custom blueprint to our EKS cluster.

• Select Infrastructure -> Clusters
• Click on the gear icon on the far right of the EKS Cluster
• Update blueprint and select the new blueprint and version

In a few minutes, all the k8s resources matching the custom cluster blueprint will become operational on the cluster. Notice that the cluster’s blueprint name and version match what you created in the prior step.

Step 5: Verify Setup

Optionally, to verify if Prometheus is able to remote write to the AMP workspace

• Click on the EKS cluster
• Select Resources to view the integrated k8s dashboard
• Select “pods” from the resource selector and filter by the “monitoring” namespace
• Under actions for the “prometheus” pod, click on “logs” for the “prometheus server” container

Congratulations! You have successfully set up and deployed AMP with Rafay. With Rafay cluster blueprinting, you can automate the deployment of AMP across your Kubernetes clusters making it easier to install, maintain, update, and provide governance across your EKS or on-premise Kubernetes clusters.

Ready to find out why so many enterprise platform teams have partnered with Rafay to streamline Amazon EKS and EKS Anywhere operations? Sign up for a free trial today.