This article describes how to integrate Fortanix Self-Defending Key Management Service (KMS) with Trustworthy Tokens for bounded RBAC in Kubernetes for managing application secrets. It also contains the information that a user requires to:
- Perform auto-auth integration using trustworthy tokens
Cloud-native and hybrid applications are increasingly deployed within containers. Kubernetes is transforming not only the application architecture but also the development and operations pipelines for application delivery.
Every day, teams come to rely on numerous secrets in their development and operational DevOps processes. Secrets ranging from passwords, tokens, certificates, SSH keys, and database credentials simply cannot be hardcoded or configured statically anymore.
Fortanix offers a powerful yet straightforward solution for enterprises and DevOps alike to centralize the storage and management of security artifacts. Secrets, cryptographic keys, and certificates can be extended to widely used applications like databases, PKI systems, HTTP servers/proxies, and custom applications. Most importantly it does so without sacrificing the security of an HSM and the agility of a massively scalable KMS built on a modern technology stack.
Kubernetes operates two realms commonly known as the control plane and data plane. Various components within the control plane are responsible for the orchestration of backbone services necessary to run a distributed application.
In addition to serving as an external KMS provider to Kubernetes, Fortanix Self-Defending KMS can also own and manage application-level secrets for Kubernetes. This document explains a high-level overview of a specific Kubernetes Authentication scheme as well as illustrate its integration with Fortanix Self-Defending KMS for managing application secrets.
Fortanix Self-Defending KMS supports standard authentication mechanisms. For Kubernetes and Google Cloud Service Accounts, a trustworthy dynamic token to identify and authorize applications access to secrets stored within the HSM is critical. Such applications could previously authenticate to Fortanix Self-Defending KMS with passwords or API Keys and now can authenticate with a Kubernetes generated trustworthy signed JSON Web Token (JWT). Thereby managing the credentials independently from the application simply by relying on the PKI established within the orchestration control plane ensures better security.
- Kubernetes Cluster:
- Version 1.12 allows Service Account Token Volume projection for signed JWTs
- Fortanix Self-Defending KMS Account:
- Either on the Cloud or locally within your enterprise HSM
- Container Registry and Runtimes:
- Docker and Containerd have been successfully tested
An enterprise application deployed within a Kubernetes cluster can rely on the built-in Service Account attestation from the API Server and authenticate to Fortanix Self-Defending KMS. Once authenticated, the application can perform desired cryptographic and key management functions inside Fortanix Self-Defending KMS without duplicating secrets natively. The roles and permissions for the application for access control to the secrets/keys can be mapped directly to Kubernetes Service Accounts as well as controlled independently inside Fortanix Self-Defending KMS.
Auto-Auth Using Trustworthy Tokens
Kubernetes 1.12 released a new feature called Service Account Token Volume Projection. This allows for applications to rely on named service accounts and an additional layer of security for authentication and authorization.
Upon deployment, the API Server dynamically injects the service account identity into the pod(s) during runtime to provide an attestation. The identity provides a verifiable way for external applications or third parties to inspect a signed token and extend to it additional permissions to resources. Applications running within separate containers inside the same pod will have access to the signed token. The signing of identity tokens is configurable by the Cluster Admin by patching the API Server. This diagram illustrates the flow of the signed identity token from the initial assignment to re-issuance or revocation.
When Self-Defending KMS receives and verifies the token using the public key of the Kubernetes API Server, it can then validate additional roles and access controls for its native resources. Since these signed tokens have a finite expiration period and are rotated automatically by Kubernetes based on the Pod spec, capturing of a single token has limited use, thereby drastically reducing the threat surface.
Figure 1 also shows the authentication scheme established within Kubernetes for the API Server, the Kubelet, the deployment pods, and the external application. Since the signed token is generated from the service account, which itself is encrypted at-rest, there is no storage of the JWT during injection by Kubelet into the pod.
Each service account in Kubernetes gets a default token. These tokens are signed by the API Server, but not rotated and have unlimited validity during their existence. Using these tokens is perhaps equivalent to a legacy password. Kubernetes took note and created the token volume projection to address the security gaps. These signed tokens are rotated automatically based on pod specifications and their validity is limited in scope based on attributes outlined below –
- Associated with a single Kubernetes service account
- Permissioned by the cluster role and role bindings
- Identified by an issuer name or URL
- Distributed to a specific audience for whom the token is generated and targeted
- Invalidated outside of its time window through a Not-Before or Valid-After TTL
- Signed by the API Server private key and verifiable by its public key
Based on these tenets, the signed tokens (JWT) are more trustworthy than standard service account tokens and when automatically authorized by Fortanix Self-Defending KMS, renders applications access to their secrets, keys, or certificates in a seamless manner. This tremendously improves the authentication of applications, services, and/or machines by removing the need to manage individual identities. Applications can be provisioned within Kubernetes based on codified specifications and scaled out or migrated with great ease. Secrets can be mounted as environment variables within deployments through simple API calls or by tightly integrating the SDK into the application logic. Any application code changes will be dependent on the mode of deployment of Self-Defending KMS and reliance on etcd for sinking secrets outside of the HSM. Chapter 3 provides the configuration changes and some simple scripts that show
Please consult your Fortanix Account Executive for any questions or scripts related to this.
The Fortanix Self-Defending KMS has first-class support for trustworthy tokens, which are signed by Kubernetes and injected into pods with ingrained controls to cap the use of the tokens and rotate them periodically. The source code for this setup is provided under jwt_demo at https://bitbucket.org/fortanix/kubernetes-integration/.
Copy the KEY ID (optional) and the PUBLIC KEY material to the Clipboard.
Depending upon the Kubernetes cluster, this step can require a primitive configuration change in the API Server (jwt_demo/etc/kubernetes/manifest/kube-apiserver.yaml).
- Minikube command to automatically start the cluster:
“minikube start \
--extra-config=apiserver.service-account-api- audiences=https://sdkms.fortanix.com \
- Docker Desktop requires editing kube-apiserver.yaml manifest by accessing the master or host node:
Here is a Docker Desktop command to get remote access to the master node:
Once the API server is configured, verify that the kube-system pods are running.
docker run --rm -it --privileged --pid=host walkerlee/nsenter \
-t 1 -m -- vi /etc/kubernetes/manifests/kube-apiserver.yaml
- If the Kubernetes controller-manager is in CrashbackLoopOff, then it is likely failing to authenticate during bootstrap. Verify that the service-account-signing-key matches the service-account-key-file specified in the kube-controller-manager.yaml
- If the service-account-issuer has extra trailing characters such as a forward slash (/) it will fail authentication with Fortanix Self-Defending KMS.
Signing Key Extraction for Fortanix Self-Defending KMS
The public key corresponding to the signing key of the Kubernetes API Server is needed so Fortanix Self-Defending KMS can verify the JWT upon receipt of an authentication request.
Also, if the pub file is not present within the Kubernetes PKI folder, then –
- Run the extract_signing_key.sh to get the public component of the key.
- Note the Public Key material and the Key ID (optional, may not be present)
Completing Setup for Auto-Auth
Fortanix provides a sandbox for evaluation at https://sdkms.fortanix.com
Creating an Application in Fortanix Self-Defending KMS for secrets access
- Log in to Fortanix Self-Defending KMS as an Account Administrator.
- Note the Fortanix Self-Defending KMS Account ID (UUID) from the Account Settings screen
- Choose the Group where the Application will reside. All secrets and other security objects from that Group will be visible and accessible to the Application
- Create a new Application with the following parameters:
- Name – Sample Kubernetes App
- Description – Optional
- Authentication Method – JSON Web Token
- Valid Issuer(s): Should match the specification in 3.3.1
- Specify the Public Key material (and Key ID if exists) obtained from Step 3.3.2
- Verify the Group specified for the Application is correct
Once the Fortanix Self-Defending KMS Application is created, you can verify and update the settings later. This is necessary should the Kubernetes PKI rotate or update the key used by the API SERVER for signing the Service Account Tokens.
The above illustration shows a Fortanix Self-Defending KMS App corresponding to the Kubernetes Service Account. Details of the Fortanix Self-Defending KMS App attributes are shown on the next page. The public key extracted in the previous step can also be dynamically fetched for verification. This supports auto-TLS if its setup in Kubernetes, wherein the keys and certificates are refreshed more frequently.
Kubernetes Pod Provisioning
With the underlying infrastructure setup, we are ready to deploy our test application in Kubernetes. Instructions below are for the sample app, but artifacts such as the Kubernetes Service Account and the POD can be reconfigured for the target application.
- Build a sample containerized application or service that will access secrets from Fortanix Self-Defending KMS.
- Alternatively, a Dockerfile is provided in bitbucket repo, so run:
docker build -t jwtdemo:1.0
- Alternatively, a Dockerfile is provided in bitbucket repo, so run:
- Create the ConfigMap for the 3 environment variables:
- SDKMS Endpoint URL = https://sdkms.fortanix.com
- SDKMS Account ID (mandatory for authentication of the JWT)
- SDKMS Secret or Key name for demonstration
kubectl create configmap jwtdemo-config --from-env-file=env
- Create the Service Account:
Pod spec now allows Kubernetes to project the Service Account Token as a configurable volume, which can have a shorter or longer expiration based on the target app.
kubectl apply -f pod.yaml
- name: jwtdemo-token
For the sample application, there are three test scripts provided along with this guide. Each test script authenticates against Fortanix Self-Defending KMS using the JWT obtained in the Kubernetes POD and performs a basic function.
- /home/root/jwtdemo App simply encrypts some plain text using a named AES key in Fortanix Self-Defending KMS.
- /home/root/set_sec App prompts the user to input a secret name and saves it to Fortanix Self-Defending KMS.
- /home/root/get_sec.sh App prompts the user to input a secret name and fetches it from Fortanix Self-Defending KMS.
- /home/root/kvp_test App prompts the user to input key-value pairs from Fortanix Self-Defending KMS.
These scripts are also provided inside the jwt_demo/pod_scripts folder in https://bitbucket.org/fortanix/kubernetes-integration/.
DevOps is fast transitioning into “DevSecOps”. Security teams are tasked during application development to model network topology as well as protect the credentials and secrets used to provision infrastructure, deploy applications and/or databases. Passwords and API tokens are constantly spread out within the enterprise and often stored on disk using configuration files or data bags. Security typically falls to dedicated roles but involves all other teams that are delivering an application.
As a reference, the diagram below illustrates the use of the Kubernetes Service Account Token Projection in AWS. It is used to tailor a fine-grained RBAC (role-based-access-control) mechanism in conjunction with the AWS IAM (identify-and-authorization-management) for its EKS offering.
More details are available at kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection
AWS guide on the provisioning of fine-grained IAM roles for Kubernetes service accounts: aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/