Kubernetes Secrets Management with HashiCorp Vault and Rafay

Thanks to Lan Nguyen and Mohan Atreya for their contributions to this blog.

Kubernetes and containers enable applications to be instantaneously deployed across public clouds, data centers and the edge. While this enables an organization to be nimble and fast, it presents unique challenges for DevOps and DevSecOps personnel needing to secure their applications and data across their infrastructure.

Ensuring security principles such as authentication, privacy, and authorization are properly in place across a complex infrastructure is a daunting task. In order to secure access to various applications and their dependent components, companies typically manually manage hundreds of application usernames and passwords, additional credentials such as those for databases, digital certificates and API tokens, just to name a few. These credentials are often stored (many times in plain text) in numerous places such as within source code, configuration files, version control systems and even CI/CD tools — over time leading to secrets sprawl. This typical scenario makes it impossible to keep secrets secret, to reduce the risk of breaches, audit who did what, and effectively rotate secrets over time (a security best practice).

Kubernetes deployments face the exact same challenges and are often exacerbated due to the dynamic and distributed nature of microservices. Consider a simple Kubernetes deployment of three clusters globally distributed close to the end users. While the end user benefits from the proximity of the application, it is a nightmare for DevOps to manually manage and distribute secrets on an on-going basis. If the secrets are statically managed by a select few individuals then the keys can be easily compromised due to a rogue admin or due to theft. When the secrets need to be rotated as a part of a periodical audit or due to a potential leak or compromise, manually managing them creates dependencies on individuals and leaves the deployment in a compromised state during the transition. There is also the case of dangling secrets in Kubernetes when secrets are left behind in clusters long after the workload has been removed. These limitations contradict the dynamic and on-demand nature of Kubernetes and microservices and leaves the deployment compromised.

HashiCorp Vault offers an elegant and scalable solution to centrally manage and distribute secrets for Kubernetes deployments. In this post, we’ll detail the key features and use cases of HashiCorp Vault and how Rafay’s integration with HashiCorp Vault simplifies the installation and operational overhead associated with secrets management in your Kubernetes infrastructure.

Rafay is a proud member of Hashicorp’s partner program, which ensures both companies collaborate for the benefit of all joint customers. Stay tuned for more collaboration, features and educational content from this partnership.

What is HashiCorp Vault?

HashiCorp Vault secures applications running on Kubernetes by centrally managing and controlling access to secrets and encryption keys. Vault can store and dynamically distribute secrets such as tokens, passwords, certificates and encryption keys for secure access to infrastructure and applications. Vault also provides encryption-as-a-service with centralized key management and enables encryption of data in transit and at rest across the cloud, data center and the edge. Here are some of the key features of Vault:

 

• Secret Storage: Vault supports a plethora of application and service choices as a storage backend for secret storage. Examples include MySQL, Cassandra, Consul, MSSQL, Azure and Google Cloud Storage. Vault offers a set of plugins to integrate with these storage backends.
• Audit Logs: Vault offers detailed audit logs to track client interactions for authentication, token creation, secret access and revocation. This provides a detailed audit trail and can be used to detect security anomalies and breaches. IT admins can further use this data to fine tune the security access policies and prove compliance with industry regulations.
• Dynamic Secrets: Secrets can be programmatically generated on-demand for users and applications with an expiry timer (TTL) and an option to renew. Hence, secrets do not lurk around and they are generated as and when needed. If needed, secrets can also be revoked immediately after their use.
• API-Driven Encryption: Vault offers Encryption-as-a-Service by providing HTTPS based APIs to encrypt and decrypt application data. This relieves the application developers from the burden of encrypting and decrypting application data and enables IT admins to centrally manage it through Vault.
• Policy-Driven Key Rotation: Encryption keys can be rotated periodically or on-demand with Vault. As new keys are distributed throughout the environment, previously encrypted data can still be decrypted since both the old and new keys are saved in a keyring.

Vault offers a comprehensive set of capabilities to manage and distribute secrets in a Kubernetes deployment. However, the creation and deployment of an application securely as a set of microservices using Kubernetes touches application developers and DevOps personnel equally. While security is a top concern for both of these groups, integrating Vault with Kubernetes requires the understanding of Kubernetes init containers, side cars and more which introduces a learning curve. Rafay offers an easy and seamless integration with Vault so that application developers and DevOps personnel can jointly and swiftly adopt Vault into their development, packaging, deployment and operational processes.

Integrating Vault with the Rafay Kubernetes Management Cloud

The Rafay Kubernetes Management Cloud (KMC) is a SaaS-first Kubernetes management solution with enterprise-class scalability, zero-trust security (of the infrastructure) and interoperability for managing apps across public clouds, data centers & edge. To enable security rapidly within the infrastructure and between applications, Rafay has a seamless and robust integration with Vault. As the number of Kubernetes clusters expand from proof-of-concept to development to test to production across multiple use cases and applications, managing secrets across all these clusters becomes cumbersome and challenging. By integrating with Vault, Rafay removes the learning curve and enables application developers, deployers, and operators to all be on the same page by leveraging Vault for their respective functions. Here are the key highlights of Rafay’s integration with Vault:

• Establishing Two Way Trust Between Vault and Kubernetes clusters: is a manual task. Every Kubernetes cluster has to be individually configured for Vault and vice versa. Rafay makes it very easy and intuitive for DevOps to complete this task.
• System of Record for Vault Configuration: Rafay provides centralized access and visibility of Vault configuration parameters spanning across multiple clusters.
• Simple Annotations to Dynamically Retrieve Secrets from Vault Server: Rafay enables an application developer to secure their application secrets by simply adding a few annotations into the Kubernetes YAML file or helm chart to integrate with an existing Vault instance. Rafay supports annotations for both file and environment variable secrets.
• Consistent Secrets Management Across Development Streams: With Rafay’s integration of Vault, application developers don’t need to go through the process of setting up Vault on their own. In a typical scenario, an application is made of microservices delivered by multiple developers. With the use of the same annotations across these developers, consistency is maintained across applications during deployment.

How to Integrate Vault with the Rafay Kubernetes Management Cloud

Once Hashicorp Vault is installed on your infrastructure, you can follow these steps to easily establish trust between your new, centralized HashiCorp Vault server and your Rafay Managed clusters.

Step 1: Create a Secret Store by logging into the Rafay KMC, clicking on Integrations > Secret Stores, then “New Secret Store”. Provide a name for the secret store, select “Vault” from the Provider drop-down and click “CREATE” as shown here:

 

Step 2: Edit the Secret Store by providing the Vault Host, select the cluster name from the drop-down list and provide an “Auth Path” for the selected cluster. Then select save changes.

Once the Vault integration configuration is deployed to the clusters, copy the related Vault settings for each of the clusters shown below and complete the configuration in Vault (see Step 3).

Step 3: Create an Auth Path for Clusters. In order for Vault to grant access to the clusters to retrieve secrets, the Kubernetes Auth Method for each of the clusters will need to be created in Vault from the information retrieved at the end of Step 2 above.

This step completes the establishment of “Trust” between the clusters and the central Vault Server. The settings in Vault can be updated using the Vault CLI, UI or API. Detailed instructions on this part of the process can be found in our product documentation.

Application workloads can use Rafay’s Secret Store annotations to dynamically retrieve secrets from the Vault server. For application developers and DevOps personnel new to containers, microservices, Kubernetes, YAML files and Helm charts, the Rafay workload wizard offers an intuitive and click-to-configure experience so that an application developer can configure their workloads to leverage Vault in just a few clicks. The secrets are made available to the containers as part of their “Startup Configuration” and can be in the form of environment variables or files. Below is an example of Rafay workload wizard enabling file based Vault integration. See here for more details about this feature.

Application workloads based on Kubernetes YAML or Helm can use Rafay supported annotations for Secret Store to dynamically retrieve secrets from Vault server to the pod’s environment variables. Below is an example snippet of pod annotations to inject Vault secrets as environment variables for K8s YAML. See here for more details about this feature.

 annotations:
    rafay.dev/secretstore: vault
    vault.secretstore.rafay.dev/role: <vault_role>
  ...
  spec:
    serviceAccountName: <service_acount>
    containers:
      env:
      - name: <environment_name>
        value: secretstore:vault:<path_to_secrets>

Application workloads based on Kubernetes YAML or Helm can use Rafay supported annotations to inject Vault secrets as file mounts as well. Below is an example of injecting Vault secrets as file mounts using K8s YAML. See here for more details about this feature.

  annotations:
    rafay.dev/secretstore: vault
    vault.secretstore.rafay.dev/role: <vault_role>
    vault.secretstore.rafay.dev/filesecret-config-1: |
        {
            "vaultSecretRef": "<path_to_vault_secrets1>",
            "secretFileName": "<filename1>",
            "volumeMountPath": "<mount_path1>",
            "containers": ["<container_name1>"]
        }
    vault.secretstore.rafay.dev/filesecret-config-2: |
        {
            "vaultSecretRef": "<path_to_vault_secrets2>",
            "secretFileName": "<filename2>",
            "volumeMountPath": "<mount_path2>",
            "containers": ["<container_name2>"]
        }
  ...
  spec:
    serviceAccountName: <service_acount>
    containers:
  ...

Kubernetes Secrets Management Made Easy with Vault and Rafay

HashiCorp Vault is a scalable and powerful tool for centrally managing secrets, encryption keys, tokens, passwords and certificates. It can help DevOps and DevSecOps personnel to implement security best practices in a continual manner while significantly reducing the attack surface. Rafay’s Kubernetes Management Cloud provides an easy and seamless integration with Vault, to enable Vault services quickly, and to ensure cross-functional teams are working hand in hand operationally. The combination of Hashicorp Vault and Rafay enables organizations to rapidly secure their Kubernetes infrastructure and the applications running within it and keep it secure no matter how quickly the number of clusters under management grows across public clouds, data centers & the edge.