This article describes how to integrate Fortanix Self-Defending KMS with Kubernetes for Sidecar Injection of Application Secrets. It also contains the information that a user requires to:
- Perform sidecar injection for auto-provisioning
- Perform the above integration
Cloud-native and hybrid applications are increasingly deployed within containers. Kubernetes is transforming not only the application architecture but also the development and operations pipelines for application delivery.
Every day, teams come to rely on numerous secrets in their development and operational DevOps processes. Secrets ranging from passwords, tokens, certificates, SSH keys, and database credentials simply cannot be hardcoded or configured statically anymore.
Fortanix offers a powerful yet straightforward solution for enterprises and DevOps alike to centralize the storage and management of security artifacts. Secrets, cryptographic keys, and certificates can be extended to widely used applications like databases, PKI systems, HTTP servers/proxies, and custom applications. Most importantly it does so without sacrificing the security of an HSM and the agility of a massively scalable KMS built on a modern technology stack.
Kubernetes operates two realms commonly known as the control plane and data plane. Various components within the control plane are responsible for the orchestration of backbone services necessary to run a distributed application.
In addition to serving as an external KMS provider to Kubernetes, Fortanix Self-Defending KMS can also own and manage application-level secrets for Kubernetes. This document explains a high-level overview of a specific Kubernetes Authentication scheme as well as illustrate its integration with Self-Defending KMS for managing application secrets.
- Kubernetes Cluster:
- Version 1.13 allows for Kubernetes Admissions Controller to inject secrets
- Self-Defending KMS Account:
- Either on the Cloud or locally within your enterprise HSM
- Container Registry and Runtimes:
- Docker and Containerd have been successfully tested
An enterprise application deployed within a Kubernetes cluster can rely on the built-in Service Account attestation from the API Server and authenticate to Fortanix Self-Defending KMS. Once authenticated, the application can perform desired cryptographic and key management functions inside Self-Defending KMS without duplicating secrets natively. The roles and permissions for the application for access control to the secrets/keys can be mapped directly to Kubernetes Service Accounts as well as controlled independently inside Self-Defending KMS.
Sidecar Injection for Auto-Provisioning
To further ease the adoption of secrets and cryptographic functions, Fortanix has proven that Kubernetes and Self-Defending KMS can be tightly integrated through Admissions Controllers. Developers and operators then only need to specify the application secrets that will be pulled in automatically through environment variables in the destination pods/containers.
Admission Controllers were introduced in Kubernetes version 1.9 and made publicly available (beta) since version 1.13. They are commonly implemented as Validation or Mutation Webhooks that inspect, verify, and/or alter the pod specification during entry into or update within the cluster. When implemented with a dedicated sidecar for reading and refreshing secrets from Self-Defending KMS, secrets are securely injected into the application.
The illustration above demonstrates the setup and workflow of authentication and the injection of secrets. Cluster Admin updates the API Server to accept Admission Controllers. A Mutating Webhook Configuration is applied, which runs a Kubernetes Service to inspect pod specifications and can be limited to either the default or a specific namespace. This mutation service is a simple web server that leverages the Kubernetes API to read the config maps, volumes and, containers of the pod seeking entry into or update within the cluster. Pod specifications need to have certain predefined annotations relevant to the mutation service. Optionally, the mutation service can be programmed to perform dynamic changes to the pod based on a flexible set of annotations.
Now, every time an application deployment or replica set attempts to create or update pods in the intended or target namespace, the Mutating Webhook will intercept and inject the Kubernetes artifacts (config maps, volumes and, most importantly the sidecar containers). Since Self-Defending KMS is a vault-less and agent-less KMS and Secrets repository, the init and refresh containers are simple commands instead of elaborate service daemons or processes. Either of these containers will authenticate with Self-Defending KMS based on the trustworthy JWT generated by Kubernetes for the pod (or simply an API KEY delivered through a pod annotation). Once authenticated, the sidecar containers pull secrets based on RBAC within Self-Defending KMS identified by the API KEY or Kubernetes Service Account. Secrets are then injected directly into the pod and provisioned as environment variables. Ultimately, this allows applications to remain unaltered and their dependency on secrets seamlessly switches to Fortanix Self-Defending KMS from legacy repositories.
The source code for this project is available at https://bitbucket.org/fortanix/kubernetes-integration/ under secrets_injection.
The architecture of this automation is illustrated below. Fundamentally, the Secret Controller is the Kubernetes Admission Controller that serves as the Mutating Webhook to intercept pod entry and update in the cluster. This service daemon is a simple Web Server that is implemented in any programming language (Go, Java, C++, Python, and so on) which is natively supported by the Kubernetes API. This web server typically does the following –
- The main method to keep the webserver daemon running
- HTTP endpoint at which requests from API Seitrver are received, for example: /mutate
- TLS certificates for mutual authentication and secure exchange of payload
The /mutate request typically invokes another package or function that performs a Kubernetes API Admission Review to:
- Validate if the request is compatible with the configuration
- pod namespace
- pod annotations
- pod associated service account
- Mutate the request and dispatch the response such that pod specification includes:
- Config maps for any variables needed by the sidecar containers
- Environment variables
- Projected Service Account token
- Secrets, if needed
- Sidecar Containers
- Init Container
- Refresh Container (optional)
- Update pod annotations to reflect that:
- Mutation or secrets injection is applied
- Additional parameters passed to the sidecar or main app containers
- Config maps for any variables needed by the sidecar containers
NOTE: Code samples tested with generic Kubernetes Admission Controllers available on bitbucket.org. These open-source projects abstract the underlying logic of Kubernetes API server updates through simple configuration templates that inject init and/or sidecar containers along with volumes and config-maps.
More details on https://bitbucket.org/fortanix/kubernetes-integration/
DevOps is fast transitioning into “DevSecOps”. Security teams are tasked during application development to model network topology as well as protect the credentials and secrets used to provision infrastructure, deploy applications and/or databases. Passwords and API tokens are constantly spread out within the enterprise and often stored on disk using configuration files or data bags. Security typically falls to dedicated roles but involves all other teams that are delivering an application.
As a reference, the diagram below illustrates the use of the Kubernetes Service Account Token Projection in AWS. It is used to tailor a fine-grained RBAC (role-based-access-control) mechanism in conjunction with the AWS IAM (identify-and-authorization-management) for its EKS offering.
More details are available at kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection
AWS guide on the provisioning of fine-grained IAM roles for Kubernetes service accounts: aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/